strong detectability and observers conditions are given for the existence of observers that estimate unmeasured outputs on the basis of partial information on the input and the state the concepts of strong detectability and strong observability introduced before in the literature for discrete time systems only are defined and studied for continuous time systems it is shown that there are two different concepts of strong detectability which coincide for discrete time systems algebraic conditions for either concept are given it is shown that these concepts are intimately related to the existence of strong observers i e observers that only use the output of the plant

robustness analysis of network controllability abstract structural controllability which is an interesting property of complex networks attracts many researchers from various fields the maximum matching algorithm was recently applied to explore the minimum number of driver nodes where control signals are injected for controlling the whole network here we study the controllability of directed erdos renyi and scale free networks under attacks and cascading failures results show that degree based attacks are more efficient than random attacks on network structural controllability cascade failures also do great harm to network controllability even if they are triggered by a local node failure

ensemble pruning via semi definite programming an ensemble is a group of learning models that jointly solve a problem however the ensembles generated by existing techniques are sometimes unnecessarily large which can lead to extra memory usage computational costs and occasional decreases in effectiveness the purpose of ensemble pruning is to search for a good subset of ensemble members that performs as well as or better than the original ensemble this subset selection problem is a combinatorial optimization problem and thus finding the exact optimal solution is computationally prohibitive various heuristic methods have been developed to obtain an approximate solution however most of the existing heuristics use simple greedy search as the optimization method which lacks either theoretical or empirical quality guarantees in this paper the ensemble subset selection problem is formulated as a quadratic integer programming problem by applying semi definite programming sdp as a solution technique we are able to get better approximate solutions computational experiments show that this sdp based pruning algorithm outperforms other heuristics in the literature its application in a classifier sharing study also demonstrates the effectiveness of the method

nearly defective complex eigenvalues in bessel matrices abstract it is known that an n n bessel matrix t n exhibits simple well separated complex eigenvalues which are getting more and more ill conditioned for increasing n the aim of this paper is to study the behavior of such eigenvalues which are simple in theory but in practice become defective using finite precision to this end we provide a sharp estimate of the distance of t n to the nearest defective matrix when n increases

iterative boolean combination of classifiers in the roc space an application to anomaly detection with hmms hidden markov models hmms have been shown to provide a high level performance for detecting anomalies in sequences of system calls to the operating system kernel using boolean conjunction and disjunction functions to combine the responses of multiple hmms in the roc space may significantly improve performance over a single best hmm however these techniques assume that the classifiers are conditional independent and their of roc curves are convex these assumptions are violated in most real world applications especially when classifiers are designed using limited and imbalanced training data in this paper the iterative boolean combination ibc technique is proposed for efficient fusion of the responses from multiple classifiers in the roc space it applies all boolean functions to combine the roc curves corresponding to multiple classifiers requires no prior assumptions and its time complexity is linear with the number of classifiers the results of computer simulations conducted on both synthetic and real world host based intrusion detection data indicate that the ibc of responses from multiple hmms can achieve a significantly higher level of performance than the boolean conjunction and disjunction combinations especially when training data are limited and imbalanced the proposed ibc is general in that it can be employed to combine diverse responses of any crisp or soft one or two class classifiers and for wide range of application domains

learn mf a random subspace approach for the missing feature problem we introduce learn mf an ensemble of classifiers based algorithm that employs random subspace selection to address the missing feature problem in supervised classification unlike most established approaches learn mf does not replace missing values with estimated ones and hence does not need specific assumptions on the underlying data distribution instead it trains an ensemble of classifiers each on a random subset of the available features instances with missing values are classified by the majority voting of those classifiers whose training data did not include the missing features we show that learn mf can accommodate substantial amount of missing data and with only gradual decline in performance as the amount of missing data increases we also analyze the effect of the cardinality of the random feature subsets and the ensemble size on algorithm performance finally we discuss the conditions under which the proposed approach is most effective

mixture densities maximum likelihood and the em algorithm the problem of estimating the parameters which determine a mixture density has been the subject of a large diverse body of literature spanning nearly ninety years during the last two decades the

multivariate adaptive regression splines a new method is presented for flexible regression modeling of high dimensional data the model takes the form of an expansion in product spline basis functions where the number of basis functions as well as the parameters associated with each one product degree and knot locations are automatically determined by the data this procedure is motivated by the recursive partitioning approach to regression and shares its attractive properties unlike recursive partitioning however this method produces continuous models with continuous derivatives it has more power and flexibility to model relationships that are nearly additive or involve interactions in at most a few variables in addition the model can be represented in a form that separately identifies the additive contributions and those associated with the different multivariable interactions

nonparametric e mixture estimation this study considers the common situation in data analysis when there are few observations of the distribution of interest or the target distribution while abundant observations are available from auxiliary distributions in this situation it is natural to compensate for the lack of data from the target distribution by using data sets from these auxiliary distributions in other words approximating the target distribution in a subspace spanned by a set of auxiliary distributions mixture modeling is one of the simplest ways to integrate information from the target and auxiliary distributions in order to express the target distribution as accurately as possible there are two typical mixtures in the context of information geometry the and mixtures the mixture is applied in a variety of research fields because of the presence of the well known expectation maximazation algorithm for parameter estimation whereas the mixture is rarely used because of its difficulty of estimation particularly for nonparametric models the mixture however is a well tempered distribution that satisfies the principle of maximum entropy to model a target distribution with scarce observations accurately this letter proposes a novel framework for a nonparametric modeling of the mixture and a geometrically inspired estimation algorithm as numerical examples of the proposed framework a transfer learning setup is considered the experimental results show that this framework works well for three types of synthetic data sets as well as an eeg real world data set

feature extraction for classification problems and its application to face recognition this study investigates a new method of feature extraction for classification problems the method is based on the independent component analysis ica however unlike the original ica one of the unsupervised learning methods it is developed for classification problems by utilizing class information the proposed method is an extension of our previous work on binary class problems to multi class problems it treats the class labels as input features in order to produce two sets of new features one that carries much information on the class labels and the other that is irrelevant to the class the learning rule for this method is obtained using the stochastic gradient method to maximize the likelihood of the observed data among the new features using only class relevant ones the dimension of the feature space can be greatly reduced in line with the principle of parsimony resulting better generalization this method was applied to recognize face identities and facial expressions using various databases such as the yale at t former orl color feret face databases and so on the performance of the proposed method was compared with those of conventional methods such as the principal component analysis pca fisher s linear discriminant fld etc the experimental results show that the proposed method performs well for face recognition problems

mixture of regression models with varying mixing proportions a semiparametric approach in this article we study a class of semiparametric mixtures of regression models in which the regression functions are linear functions of the predictors but the mixing proportions are smoothing functions of a covariate we propose a one step backfitting estimation procedure to achieve the optimal convergence rates for both regression parameters and the nonparametric functions of mixing proportions we derive the asymptotic bias and variance of the one step estimate and further establish its asymptotic normality a modified expectation maximization type em type estimation procedure is investigated we show that the modified em algorithms preserve the asymptotic ascent property numerical simulations are conducted to examine the finite sample performance of the estimation procedures the proposed methodology is further illustrated via an analysis of a real dataset

combining predictions in pairwise classification an optimal adaptive voting strategy and its relation to weighted voting weighted voting is the commonly used strategy for combining predictions in pairwise classification even though it shows good classification performance in practice it is often criticized for lacking a sound theoretical justification in this paper we study the problem of combining predictions within a formal framework of label ranking and under some model assumptions derive a generalized voting strategy in which predictions are properly adapted according to the strengths of the corresponding base classifiers we call this strategy adaptive voting and show that it is optimal in the sense of yielding a map prediction of the class label of a test instance moreover we offer a theoretical justification for weighted voting by showing that it yields a good approximation of the optimal adaptive voting prediction this result is further corroborated by empirical evidence from experiments with real and synthetic data sets showing that even though adaptive voting is sometimes able to achieve consistent improvements weighted voting is in general quite competitive all the more in cases where the aforementioned model assumptions underlying adaptive voting are not met in this sense weighted voting appears to be a more robust aggregation strategy

hierarchical mixtures of experts and the em algorithm we present a tree structured architecture for supervised learning the statistical model underlying the architecture is a hierarchical mixture model in which both the mixture coefficients and the mixture components are generalized linear models glim s learning is treated as a maximum likelihood problem in particular we present an expectation maximization em algorithm for adjusting the parameters of the architecture we also develop an on line learning algorithm in which the parameters are updated incrementally comparative simulation results are presented in the robot dynamics domain

accurate computation of the zeros of the generalized bessel polynomials a general method for approximating polynomial solutions of second order linear homogeneous differential equations with polynomial coefficients is applied to the case of the families of differential equations defining the generalized bessel polynomials and an algorithm is derived for simultaneously finding their zeros then a comparison with several alternative algorithms is carried out it shows that the computational problem of approximating the zeros of the generalized bessel polynomials is not an easy matter at all and that the only algorithm able to give an accurate solution seems to be the one presented in this paper

a theoretical study on six classifier fusion strategies we look at a single point in feature space two classes and l classifiers estimating the posterior probability for class spl omega sub 1 assuming that the estimates are independent and identically distributed normal or uniform we give formulas for the classification error for the following fusion methods average minimum maximum median majority vote and oracle

estimating the state probability distribution for epidemic spreading in complex networks the problem of state estimation of spreading phenomena in complex networks is considered on the basis of a detectability based approach using a simple reduced model based state distribution estimator where the monitored nodes are driven directly by the measured data asymptotic convergence conditions are provided in terms of the number and location of the required sensors on the basis of the network topology the convergence of the estimator is established in terms of the largest eigenvalue of a reduced connectivity matrix which stems from removing the monitored nodes and their connections from the original graph in the case of unit weights this condition corresponds to measuring the nodes with highest degree numerical simulations for a complete and a scale free network each of 500 nodes and randomly distributed and unit weights respectively illustrate the estimator functioning with 20 sensors for the complete and 38 sensors for the scale free network

a parallel algorithm for the nonsymmetric eigenvalue problem this paper describes a parallel algorithm for computing the eigenvalues and eigenvectors of a nonsymmetric matrix the algorithm is based on a divide and conquer procedure and uses an iterative refinement technique

variance constrained state estimation for complex networks with randomly varying topologies this paper investigates the variance constrained h infty state estimation problem for a class of nonlinear time varying complex networks with randomly varying topologies stochastic inner coupling and measurement quantization a kronecker delta function and markovian jumping parameters are utilized to describe the random changes of network topologies a gaussian random variable is introduced to model the stochastic disturbances in the inner coupling of complex networks as a kind of incomplete measurements measurement quantization is taken into consideration so as to account for the signal distortion phenomenon in the transmission process stochastic nonlinearities with known statistical characteristics are utilized to describe the stochastic evolution of the complex networks we aim to design a finite horizon estimator such that in the simultaneous presence of quantized measurements and stochastic inner coupling the prescribed variance constraints on the estimation error and the desired h infty performance requirements are guaranteed over a finite horizon sufficient conditions are established by means of a series of recursive linear matrix inequalities and subsequently the estimator gain parameters are derived a simulation example is presented to illustrate the effectiveness and applicability of the proposed estimator design algorithm

the concave convex procedure the concave convex procedure cccp is a way to construct discrete time iterative dynamical systems that are guaranteed to decrease global optimization and energy functions monotonically this procedure can be applied to almost any optimization problem and many existing algorithms can be interpreted in terms of it in particular we prove that all expectation maximization algorithms and classes of legendre minimization and variational bounding algorithms can be reexpressed in terms of cccp we show that many existing neural network and mean field theory algorithms are also examples of cccp the generalized iterative scaling algorithm and sinkhorn s algorithm can also be expressed as cccp by changing variables cccp can be used both as a new way to understand and prove the convergence of existing optimization algorithms and as a procedure for generating new algorithms

the cal d q r algorithm basic theory convergence and conditional stability we describe a fast matrix eigenvalue algorithm that uses a matrix factorization and reverse order multiply technique involving three factors and that is based on the symmetric matrix factorization as well as on d orthogonal reduction techniques where d mbox diag pm1 is computed from the given matrix a it operates on a similarity reduction of a real matrix a nn to general tridiagonal form t and computes all of a s eigenvalues in frac 4 3 n 3 o n 2 operations where the o n 2 part of the operations is possibly performed over bbb c instead of the 7 8 n 3 real flops required by the qr eigenvalue algorithm potential breakdo wn of the dqr algorithm can occur in the reduction to tridiagonal form and in the d orthogonal dqr reductions both however can be monitored during the computations the former occurs rather rarely for dimensions n leq 300 and can essentially be bypassed while the latter is extremely rare and can be bypassed as well in our conditionally stable implementation of the dqr steps we prove an implicit dq theorem which allows implicit shifts give a convergence proof for the dqr algorithm and show that dqr is conditionally stable for general balanced tridiagonal matrices t rm b

bayesian density regression summary the paper considers bayesian methods for density regression allowing a random probability distribution to change flexibly with multiple predictors the conditional response distribution is expressed as a non parametric mixture of regression models with the mixture distribution changing with predictors a class of weighted mixture of dirichlet process priors is proposed for the uncountable collection of mixture distributions it is shown that this specification results in a generalized polya urn scheme which incorporates weights that are dependent on the distance between subjects predictor values to allow local dependence in the mixture distributions we propose a kernel based weighting scheme a gibbs sampling algorithm is developed for posterior computation the methods are illustrated by using simulated data examples and an epidemiologic application

an analysis of ensemble pruning techniques based on ordered aggregation several pruning strategies that can be used to reduce the size and increase the accuracy of bagging ensembles are analyzed these heuristics select subsets of complementary classifiers that when combined can perform better than the whole ensemble the pruning methods investigated are based on modifying the order of aggregation of classifiers in the ensemble in the original bagging algorithm the order of aggregation is left unspecified when this order is random the generalization error typically decreases as the number of classifiers in the ensemble increases if an appropriate ordering for the aggregation process is devised the generalization error reaches a minimum at intermediate numbers of classifiers this minimum lies below the asymptotic error of bagging pruned ensembles are obtained by retaining a fraction of the classifiers in the ordered ensemble the performance of these pruned ensembles is evaluated in several benchmark classification tasks under different training conditions the results of this empirical investigation show that ordered aggregation can be used for the efficient generation of pruned ensembles that are competitive in terms of performance and robustness of classification with computationally more costly methods that directly select optimal or near optimal subensembles

bayesian nonparametric vector autoregressive models abstract vector autoregressive var models are the main work horse models for macroeconomic forecasting and provide a framework for the analysis of complex dynamics that are present between macroeconomic variables whether a classical or a bayesian approach is adopted most var models are linear with gaussian innovations this can limit the model s ability to explain the relationships in macroeconomic series we propose a nonparametric var model that allows for nonlinearity in the conditional mean heteroscedasticity in the conditional variance and non gaussian innovations our approach differs from that of previous studies by modelling the stationary and transition densities using bayesian nonparametric methods our bayesian nonparametric var bayesnp var model is applied to us and uk macroeconomic time series and compared to other bayesian var models we show that bayesnp var is a flexible model that is able to account for nonlinear relationships as well as heteroscedasticity in the data in terms of short run out of sample forecasts we show that bayesnp var predictively outperforms competing models

density estimation by mixture models with smoothing priors in the statistical approach for self organizing maps soms learning is regarded as an estimation algorithm for a gaussian mixture model with a gaussian smoothing prior on the centroid parameters the values of the hyperparameters and the topological structure are selected on the basis of a statistical principle however since the component selection probabilities are fixed to a common value the centroids concentrate on areas with high data density this deforms a coordinate system on an extracted manifold and makes smoothness evaluation for the manifold inaccurate in this article we study an extended som model whose component selection probabilities are variable to stabilize the estimation a smoothing prior on the component selection probabilities is introduced an estimation algorithm for the parameters and the hyperparameters based on empirical bayesian inference is obtained the performance of density estimation by the new model and the som model is compared via simulation experiments

epidemic dynamics and endemic states in complex networks we study by analytical methods and large scale simulations a dynamical model for the spreading of epidemics in complex networks in networks with exponentially bounded connectivity we recover the usual epidemic behavior with a threshold defining a critical point below that the infection prevalence is null on the contrary on a wide range of scale free networks we observe the absence of an epidemic threshold and its associated critical behavior this implies that scale free networks are prone to the spreading and the persistence of infections whatever spreading rate the epidemic agents might possess these results can help understanding computer virus epidemics and other spreading phenomena on communication and social networks

mixtures of marginal models summary in this paper we adapt a mixture model originally developed for regression models with independent data for the more general case of correlated outcome data which includes longitudinal data as a special case the estimation is performed by a generalisation of the em algorithm which we call the expectation solution es algorithm in this es algorithm the m step of the em algorithm is replaced by a step requiring the solution of a series of generalised estimating equations the es algorithm a general algorithm for solving generalised estimating equations with incomplete data is then applied to the present problem of mixtures of marginal models in addition to allowing for correlation inherent in correlated outcome data the systematic component of this mixture of marginal models is more flexible than the conventional linear function the methodology is applied in the contexts of normal and poisson response data some theory regarding the es algorithm is presented

a probabilistic model of classifier competence for dynamic ensemble selection the concept of a classifier competence is fundamental to multiple classifier systems mcss in this study a method for calculating the classifier competence is developed using a probabilistic model in the method first a randomised reference classifier rrc whose class supports are realisations of the random variables with beta probability distributions is constructed the parameters of the distributions are chosen in such a way that for each feature vector in a validation set the expected values of the class supports produced by the rrc and the class supports produced by a modelled classifier are equal this allows for using the probability of correct classification of the rrc as the competence of the modelled classifier the competences calculated for a validation set are then generalised to an entire feature space by constructing a competence function based on a potential function model or regression three systems based on a dynamic classifier selection and a dynamic ensemble selection des were constructed using the method developed the des based system had statistically significant higher average rank than the ones of eight benchmark mcss for 22 data sets and a heterogeneous ensemble the results obtained indicate that the full vector of class supports should be used for evaluating the classifier competence as this potentially improves performance of mcss

on the convergence properties of the em algorithm two convergence aspects of the em algorithm are studied i does the em algorithm find a local maximum or a stationary value of the incomplete data likelihood function ii does the sequence of parameter estimates generated by em converge several convergence results are obtained under conditions that are applicable to many practical situations two useful special cases are a if the unobserved complete data specification can be described by a curved exponential family with compact parameter space all the limit points of any em sequence are stationary points of the likelihood function b if the likelihood function is unimodal and a certain differentiability condition is satisfied then any em sequence converges to the unique maximum likelihood estimate a list of key properties of the algorithm is included

epidemic spreading driven by biased random walks abstract random walk is one of the basic mechanisms of many network related applications in this paper we study the dynamics of epidemic spreading driven by biased random walks in complex networks in our epidemic model infected nodes send out infection packets by biased random walks to their neighbor nodes and this causes the infection of susceptible nodes that receive the packets infected nodes recover from the infection at a constant rate and will not be infected again after recovery we obtain the largest instantaneous number of infected nodes and the largest number of ever infected nodes respectively by tuning the parameter of the biased random walks simulation results on model and real world networks show that spread of the epidemic becomes intense and widespread with increase of either delivery capacity of infected nodes average node degree or homogeneity of node degree distribution

a tree structured adaptive network for function approximation in high dimensional spaces nonlinear function approximation is often solved by finding a set of coefficients for a finite number of fixed nonlinear basis functions however if the input data are drawn from a high dimensional space the number of required basis functions grows exponentially with dimension leading many to suggest the use of adaptive nonlinear basis functions whose parameters can be determined by iterative methods the author proposes a technique based on the idea that for most of the data only a few dimensions of the input may be necessary to compute the desired output function additional input dimensions are incorporated only where needed the learning procedure grows a tree whose structure depends upon the input data and the function to be approximated this technique has a fast learning algorithm with no local minima once the network shape is fixed and it can be used to reduce the number of required measurements in situations where there is a cost associated with sensing three examples are given controlling the dynamics of a simulated planar two joint robot arm predicting the dynamics of the chaotic mackey glass equation and predicting pixel values in real images from pixel values above and to the left

a globally convergent method for simultaneously finding polynomial roots a new method for the simultaneous approximation of all the roots of a polynomial is given the method converges for almost every initial approximation the set of the exceptional starting points being a closed set of measure zero at least if all the polynomial roots are real and simple the method exhibits quadratic convergence not only to simple but also to multiple roots

on neighbouring matrices with quadratic elementary divisors this paper presents new results on matrices with quadratic elementary divisors neighbouring a given matrixa and relates them to those given by kakan it shows that all results of this kind may give severe overestimates of the distance to the nearest defective matrix and that matrices with elementary divisors of degrees higher than two are often at a much greater distance than those with quadratic divisors

independent factor analysis we introduce the independent factor analysis ifa method for recovering independent hidden sources from their observed mixtures ifa generalizes and unifies ordinary factor analysis fa principal component analysis pca and independent component analysis ica and can handle not only square noiseless mixing but also the general case where the number of mixtures differs from the number of sources and the data are noisy ifa is a two step procedure in the first step the source densities mixing matrix and noise covariance are estimated from the observed data by maximum likelihood for this purpose we present an expectation maximization em algorithm which performs unsupervised learning of an associated probabilistic model of the mixing situation each source in our model is described by a mixture of gaussians thus all the probabilistic calculations can be performed analytically in the second step the sources are reconstructed from the observed data by an optimal nonlinear estimator a variational approximation of this algorithm is derived for cases with a large number of sources where the exact algorithm becomes intractable our ifa algorithm reduces to the one for ordinary fa when the sources become gaussian and to an em algorithm for pca in the zero noise limit we derive an additional em algorithm specifically for noiseless ifa this algorithm is shown to be superior to ica since it can learn arbitrary source densities from the data beyond blind separation ifa can be used for modeling multidimensional data by a highly constrained mixture of gaussians and as a tool for nonlinear signal encoding

inferring decision trees using the minimum description length principle abstract we explore the use of rissanen s minimum description length principle for the construction of decision trees empirical results comparing this approach to other methods are given

hierarchical unsupervised learning with growing via phase transitions we address unsupervised learning subject to structural constraints with particular emphasis placed on clustering with an imposed decision tree structure most known methods are greedy optimizing one node of the tree at a time to minimize a local cost by constrast we develop a joint optimization method derived based on information theoretic principles and closely related to known methods in statistical physics the approach is inspired by the deterministic annealing algorithm for unstructured data clustering which was based on maximum entropy inference the new approach is founded on the principle of minimum cross entropy using informative priors to approximate the unstructured clustering solution while imposing the structural constraint the resulting method incorporates supervised learning principles applied in an unsupervised problem setting in our approach the tree grows by a sequence of bifurcations that occur while optimizing an effective free energy cost at decreasing temperature scales thus estimates of the tree size and structure are naturally obtained at each temperature in the process examples demonstrate considerable improvement over known methods

asymptotic convergence rate of the em algorithm for gaussian mixtures it is well known that the convergence rate of the expectation maximization em algorithm can be faster than those of convention first order iterative algorithms when the overlap in the given mixture is small but this argument has not been mathematically proved yet this article studies this problem asymptotically in the setting of gaussian mixtures under the theoretical framework of xu and jordan 1996 it has been proved that the asymptotic convergence rate of the em algorithm for gaussian mixtures locally around the true solution is o e0 5 e where e 0 is an arbitrarily small number o x means that it is a higher order infinitesimal as x 0 and e is a measure of the average overlap of gaussians in the mixture in other words the large sample local convergence rate for the em algorithm tends to be asymptotically superlinear when e tends to zero

the random subspace method for constructing decision forests much of previous attention on decision trees focuses on the splitting criteria and optimization of tree sizes the dilemma between overfitting and achieving maximum accuracy is seldom resolved a method to construct a decision tree based classifier is proposed that maintains highest accuracy on training data and improves on generalization accuracy as it grows in complexity the classifier consists of multiple trees constructed systematically by pseudorandomly selecting subsets of components of the feature vector that is trees constructed in randomly chosen subspaces the subspace method is compared to single tree classifiers and other forest construction methods by experiments on publicly available datasets where the method s superiority is demonstrated we also discuss independence between trees in a forest and relate that to the combined classification accuracy

integration of stochastic models by minimizing α divergence when there are a number of stochastic models in the form of probability distributions one needs to integrate them mixtures of distributions are frequently used but exponential mixtures also provide a good means of integration this letter proposes a one parameter family of integration called integration which includes all of these well known integrations these are generalizations of various averages of numbers such as arithmetic geometric and harmonic averages there are psychophysical experiments that suggest that integrations are used in the brain the divergence between two distributions is defined which is a natural generalization of kullback leibler divergence and hellinger distance and it is proved that integration is optimal in the sense of minimizing divergence the theory is applied to generalize the mixture of experts and the product of experts to the mixture of experts the predictive distribution is also stated in the bayesian framework