a bootstrap method for structure detection of narmax models many systems may be described by narmax models using only a few terms however depending on the order of the system the number of candidate terms can become very large selection of a subset of these candidate terms is necessary for an efficient system description this is an unresolved issue in system identification for over parameterized models therefore in this paper we develop a bootstrap structure detection bsd algorithm as a means of determining the structure of highly over parameterized models the performance of this bsd technique was evaluated by using it to estimate the structure of a 1 simple narmax model 2 moderately over parameterized narmax model and 3 highly over parameterized narmax model the results demonstrate that the bsd algorithm is a robust method for detecting the structure of narmax models this method provides accurate estimates of parameter statistics without relying on assumptions made by traditional procedures and yields a parsimonious description of the system

online model selection based on the variational bayes the bayesian framework provides a principled way of model selection this framework estimates a probability distribution over an ensemble of models and the prediction is done by averaging over the ensemble of models accordingly the uncertainty of the models is taken into account and complex models with more degrees of freedom are penalized however integration over model parameters is often intractable and some approximation scheme is needed recently a powerful approximation scheme called the variational bayes vb method has been proposed this approach defines the free energy for a trial probability distribution which approximates a joint posterior probability distribution over model parameters and hidden variables the exact maximization of the free energy gives the true posterior distribution the vb method uses factorized trial distributions the integration over model parameters can be done analytically and an iterative expectation maximization like algorithm whose convergence is guaranteed is derived in this article we derive an online version of the vb algorithm and prove its convergence by showing that it is a stochastic approximation for finding the maximum of the free energy by combining sequential model selection procedures the online vb method provides a fully online learning method with a model selection mechanism in preliminary experiments using synthetic data the online vb method was able to adapt the model structure to dynamic environments

bootstrapping regression models the regression and correlation models are considered it is shown that the bootstrap approximation to the distribution of the least squares estimates is valid and some error bounds are given

belief condensation filtering inferring a sequence of variables from observations is prevalent in a multitude of applications traditional techniques such as kalman filters kfs and particle filters pfs are widely used for such inference problems however these techniques fail to provide satisfactory performance in many important nonlinear or non gaussian scenarios in addition there is a lack of a unified methodology for the design and analysis of different filtering techniques to address these problems in this paper we propose a new filtering methodology called belief condensation bc filtering first we establish a general framework for filtering techniques and propose an optimality criterion that leads to bc filtering we then propose efficient bc algorithms that can best represent the complex distributions arising in the filtering process the performance of the proposed techniques is evaluated for two representative nonlinear non gaussian problems showing that the bc filtering can provide accuracy approaching the theoretical bounds and outperform existing techniques in terms of the accuracy versus complexity tradeoff

the euclid algorithm and the fast computation of cross covariance and autocovariance sequences a simple linear procedure is given to compute the cross covariance sequence associated with the outputs of two rational digital transfer functions driven by the same white noise sequence such a computation often appears in the study of digital filters in wiener filtering in noise variance estimation in the study of low order approximations and in the study of multichannel systems a fast algorithm based on the euclid algorithm is introduced to solve the linear system of equations involved in the computation and a detailed analysis of the matrix is given the special case of the autocovariance computation is reviewed and the same study is performed alternate polynomial presentations are given and are shown to involve the same matrices and similar fast algorithms

some asymptotic theory for the bootstrap efron s bootstrap method of distribution approximation is shown to be asymptotically valid in a large number of situations including t statistics the empirical and quantile processes and von mises functionals some counter examples are also given to show that the approximation does not always succeed

cramer rao bounds for ar parameter and reflection coefficient estimators the finite examples cramer rao bound for the autoregressive parameters of a gaussian autoregressive model is given in closed form quick algorithms for computing this bound for the autoregressive parameters and also for the reflection coefficients are presented the exact bounds are of interest since they can differ appreciably from the asymptotic bounds if replicated independent records are available as in some spatial array processing problems the exact bounds will be approached when the number of replications goes to infinity but the asymptotic bound will not since the record length remains finite

variational message passing bayesian inference is now widely established as one of the principal foundations for machine learning in practice exact inference is rarely possible and so a variety of approximation techniques have been developed one of the most widely used being a deterministic framework called variational inference in this paper we introduce variational message passing vmp a general purpose algorithm for applying variational inference to bayesian networks like belief propagation vmp proceeds by sending messages between nodes in the network and updating posterior beliefs using local operations at each node each such update increases a lower bound on the log evidence unless already at a local maximum in contrast to belief propagation vmp can be applied to a very general class of conjugate exponential models because it uses a factorised variational approximation furthermore by introducing additional variational parameters vmp can be applied to models containing non conjugate distributions the vmp framework also allows the lower bound to be evaluated and this can be used both for model comparison and for detection of convergence variational message passing has been implemented in the form of a general purpose inference engine called vibes variational inference for bayesian networks which allows models to be specified graphically and then solved variationally without recourse to coding

online identification of the bilinear model expansion on laguerre orthonormal bases this paper proposes a new representation of discrete bilinear model by developing its coefficients associated to the input to the output and to the crossed product on three independent laguerre orthonormal bases compared to classical bilinear model the resulting model entitled bilinear laguerre model ensures a significant parameter number reduction as well as simple recursive representation however this reduction is still subject to an optimal choice of the laguerre poles defining the three laguerre bases therefore we propose an analytical solution to optimise the laguerre poles which depend on fourier coefficients defining the bilinear laguerre model and that are identified using the regularised square error the identification procedures of the laguerre poles and fourier coefficients are combined and carried out on a sliding window to provide an online identification algorithm of the bilinear laguerre model the bilinear laguerre model as well as the proposed algorithm are illustrated and tested

identification of non linear stochastic systems by state dependent parameter estimation the paper outlines how improved estimates of time variable parameters in models of stochastic dynamic systems can be obtained using recursive filtering and fixed interval smoothing techniques with the associated hyper parameters optimized by maximum likelihood based on prediction error decomposition it then shows how by exploiting special data re ordering and back fitting procedures similar recursive parameter estimation techniques can be utilized to estimate much more rapid state dependent parameter sdp variations in this manner it is possible to identify and estimate a widely applicable class of nonlinear stochastic systems as illustrated by several examples that include simulated and real data from chaotic processes finally the paper points out that such sdp models can form the basis for new methods of signal processing automatic control and state estimation for nonlinear stochastic systems

application of a least absolute shrinkage and selection operator to aeroelastic flight test data identification of non linear systems involves estimating unknown parameters and model regressor selection selection of a subset of candidate terms that best describes the observed output model selection is an important step in black box modelling of any observed process this procedure is concerned with selecting a subset of parameters to give a parsimonious description of the system which may afford greater insight into the functionality of the system or a simpler controller design in this study a least absolute shrinkage and selection operator lasso technique is investigated for computing efficient model descriptions of non linear aeroelastic systems the lasso minimises the residual sum of squares by the addition of an l1 penalty term on the parameter vector of the traditional l2 minimisation problem its use for model selection is a natural extension of this constrained minimisation approach to pseudolinear regression problems which produces some model parameters that are exactly zero and the

corrections to construction of the exact fisher information matrix of gaussian time series models by means of matrix differential rules this note contains some corrections and amplifications that were deduced from the implementation and testing of the method

on a fast algorithm for the exact information matrix of a gaussian arma time series the paper is devoted to a new algorithm for the computation of the exact fisher information matrix of a gaussian autoregressive moving average time series the number of operations is an order of magnitude smaller than the fastest existing procedure the algorithm is based on a set of new recursions for the covariance matrix of the derivatives of the state vector with respect to the parameters combined with the chandrasekhar recursions used in the evaluation of the likelihood function

marginalized particle filters for mixed linear nonlinear state space models the particle filter offers a general numerical tool to approximate the posterior density function for the state in nonlinear and non gaussian filtering problems while the particle filter is fairly easy to implement and tune its main drawback is that it is quite computer intensive with the computational complexity increasing quickly with the state dimension one remedy to this problem is to marginalize out the states appearing linearly in the dynamics the result is that one kalman filter is associated with each particle the main contribution in this paper is the derivation of the details for the marginalized particle filter for a general nonlinear state space model several important special cases occurring in typical signal processing applications will also be discussed the marginalized particle filter is applied to an integrated navigation system for aircraft it is demonstrated that the complete high dimensional system can be based on a particle filter using marginalization for all but three states excellent performance on real flight data is reported

fast kalman like filtering for large dimensional linear and gaussian state space models this article considers the filtering problem for linear and gaussian state space models with large dimensions a setup in which the optimal kalman filter kf might not be applicable owing to the excessive cost of manipulating huge covariance matrices among the most popular alternatives that enable cheaper and reasonable computation is the ensemble kf enkf a monte carlo based approximation in this article we consider a class of a posteriori distributions with diagonal covariance matrices and propose fast approximate deterministic based algorithms based on the variational bayesian vb approach more specifically we derive two iterative kf like algorithms that differ in the way they operate between two successive filtering estimates one involves a smoothing estimate and the other involves a prediction estimate despite its iterative nature the prediction based algorithm provides a computational cost that is on the one hand independent of the number of iterations in the limit of very large state dimensions and on the other hand always much smaller than the cost of the enkf the cost of the smoothing based algorithm depends on the number of iterations that may in some situations make this algorithm slower than the enkf the performances of the proposed filters are studied and compared to those of the kf and enkf through a numerical example

construction of the exact fisher information matrix of gaussian time series models by means of matrix differential rules the fisher information matrix is of fundamental importance for the analysis of parameter estimation of time series models in this paper the exact information matrix of a multivariate gaussian time series model expressed in state space form is derived a computationally efficient procedure is used by applying matrix differential rules for the derivatives of a matrix function j j with respect to its vector argument an algorithm is given it is sketched for the general state space structure without specifying a parametrization it is then detailed for the vector autoregressive moving average varma model with a given parametrization where explicit recurrent relations are developed

computation of the fisher information matrix for time series models the fisher information matrix is useful in time series modeling mainly because the significance of estimated parameters can also be derived from it it can also be used in iterative procedures of parameter estimation the paper is mainly concerned with algorithmic aspects related to the computation of that matrix either asymptotically or exactly after a review of the literature on the subject several recent methods are described and compared from the point of view of a complexity b accuracy and c the class of models for which they can be used

computation of the fisher information matrix for siso models closed form expressions and an algorithm for obtaining the fisher information matrix of gaussian single input single output siso time series models are presented it enables the computation of the asymptotic covariance matrix of maximum likelihood estimators of the parameters the procedure makes use of the autocovariance function of one or more autoregressive processes under certain conditions the siso model can be a special case of a vector autoregressive moving average arma model for which there is a method to evaluate the fisher information matrix that method is compared with the procedure described in the paper

adaptive volterra filters with evolutionary quadratic kernels using a combination scheme for memory control this paper proposes a new paradigm for adaptive volterra filtering using a time variant size of the quadratic kernel memory in order to optimally identify any unknown transversal second order nonlinear system to this end competing volterra structures of different sizes are employed in a hierarchical combination scheme so as to find the best configuration of the second order kernel memory using the already known diagonal coordinate representation the length and number of required quadratic kernel diagonals can be concurrently estimated by monitoring the combination performance subsequently the memory size of the involved models is dynamically increased or decreased following a set of intuitive rules since this automatic memory adaptation is performed along with the coefficient updates an efficient volterra filter is realized offering great flexibility and minimizing the risk of under or overmodeling any given quadratic nonlinearity besides the straightforward scheme a simplified version is presented greatly reducing the algorithmic demands this efficient version is based on a virtualization of the competing volterra filters by jointly using common coefficients and hence exhibits a computational complexity suitable for practical implementations the robust estimation performance of the approach is demonstrated by various examples for a nonlinear acoustic echo cancellation scenario involving stationary noise real speech signals and realistic volterra kernels

variational bayes for continuous hidden markov models and its application to active learning in this paper we present a variational bayes vb framework for learning continuous hidden markov models chmms and we examine the vb framework within active learning unlike a maximum likelihood or maximum a posteriori training procedure which yield a point estimate of the chmm parameters vb based training yields an estimate of the full posterior of the model parameters this is particularly important for small training sets since it gives a measure of confidence in the accuracy of the learned model this is utilized within the context of active learning for which we acquire labels for those feature vectors for which knowledge of the associated label would be most informative for reducing model parameter uncertainty three active learning algorithms are considered in this paper 1 query by committee qbc with the goal of selecting data for labeling that minimize the classification variance 2 a maximum expected information gain method that seeks to label data with the goal of reducing the entropy of the model parameters and 3 an error reduction based procedure that attempts to minimize classification error over the test data the experimental results are presented for synthetic and measured data we demonstrate that all of these active learning methods can significantly reduce the amount of required labeling compared to random selection of samples for labeling

variational bayesian filtering the use of the variational bayes vb approximation in bayesian filtering is studied both as a means to accelerate marginalized particle filtering and as a deterministic local one step approximation the vb method of approximation is reviewed together with restrictions that allow various computational savings to be achieved these variants provide a range of algorithms that can be used in a principled tradeoff between quality of approximation and computational cost in combination with marginalized particle filtering they generalize previously published work on variational filtering and extend currently available methods for speeding up stochastic approximations in bayesian filtering in particular the free form nature of the vb approximation allows optimal selection of moments which summarize the particles other bayesian filtering schemes are developed by replacing the marginalization operator in bayesian filtering with vb marginals this leads to further computational savings at the cost of quality of approximation the performance of the various vb filtering schemes is illustrated in the context of a gaussian model with a nonlinear substate and a hidden markov model

advanced point mass method for nonlinear state estimation state estimation of discrete time nonlinear non gaussian stochastic systems by point mass approach which is based on discretization of state space by a regular grid and numerical solution of bayesian recursive relations is treated the stress is laid to grid design which is crucial for estimator quality and significantly affects the computational demands of the estimator boundary based grid design thrifty convolution and multigrid design with grid splitting and merging are proposed the main advantages of these techniques are nonnegligible support delimitation time saving computation of convolution and effective processing of multimodal probability density functions respectively the techniques are involved into the basic point mass approach and a new general purpose more sophisticated point mass algorithm is designed computational demands and estimation quality of the designed algorithm are presented and compared with the particle filter in a numerical example

orthogonal least squares methods and their application to non linear system identification abstract identification algorithms based on the well known linear least squares methods of gaussian elimination cholesky decomposition classical gram schmidt modified gram schmidt householder transformation givens method and singular value decomposition are reviewed the classical gram schmidt modified gram schmidt and householder transformation algorithms are then extended to combine structure determination or which terms to include in the model and parameter estimation in a very simple and efficient manner for a class of multivariate discrete time non linear stochastic systems which are linear in the parameters

multi parameter estimation in compound gaussian clutter by variational bayesian in this paper we consider the problem of multi parameter estimation in the presence of compound gaussian clutter for cognitive radar by the variational bayesian method the advantage of variational bayesian is that the estimation of multi variate parameters is decomposed to problems of estimation of univariate parameters by variational approximation thus enabling analytically tractable approximate posterior densities in complex statistical models consisting of observed data unknown parameters and hidden variables we derive the asymptotic bayesian cramer rao bounds and demonstrate by numerical simulations that the proposed approach leads to improved estimation accuracy than the expectation maximization method and the exact bayesian method in the case of non gaussian nonlinear signal models and small data sample size

an algorithm for the exact fisher information matrix of vector armax time series abstract in this paper an algorithm is developed for the exact fisher information matrix of a gaussian vector armax or varmax process the algorithm proposed in this paper is composed by chandrasekhar recursion equations at a vector matrix level and some of these recursions consist of derivatives based on appropriate differential rules applied to a state space model for a vector process the chosen representation is such that the recursions extracted from the state space model are given in terms of expectations of derivatives of innovations and not the process and observation disturbances the algorithm will be illustrated by an example on that example a comparison is made with results from e4 a toolbox for matlab and with the asymptotic information matrix

bayesian techniques for blind deconvolution this paper introduces extended bayesian filters ebfs a new family of blind deconvolution filters for digital communications the blind deconvolution problem is formulated as a nonlinear and non gaussian fixed lag minimum mean square error filtering problem and the ebf is derived as a suboptimal recursive estimator the model based setting makes extensive use of the transmitted symbol and noise distributions a key feature of the ebf is that the filter lag can be chosen to be larger than the channel length while the complexity is exponential in a parameter which is typically chosen to be smaller than both the channel length and the filter lag extensive simulations characterizing the performance of ebfs in severe intersymbol interference channels are presented the fast convergence and robust equalization of the ebfs are demonstrated for uncoded linearly modulated signals e g differentially encoded quaternary phase shift keying qpsk transmitted over unknown channels comparisons are made to other blind symbol by symbol demodulation algorithms the results show that the ebf provides much better performance at increased complexity compared to the constant modulus algorithm and the extended kalman filter and achieves a better performance complexity trade off than other bayesian demodulation algorithms the simulations also show that the ebf is applicable with large constellations and shaped modulations

variational bayesian estimation of statistical properties of composite gamma log normal distribution an iterative variational bayesian method is proposed for estimation of the statistical properties of the composite gamma log normal distribution specifically the nakagami parameter of the gamma component and the mean and variance parameters of the log normal component moreover the random hidden signal in the model plays a key role in the parameter estimation hence its density is also estimated the parameter estimation performance is analyzed by evaluation of the variational bayesian cramer rao bound from the variational posterior densities the numerical simulations show a good agreement between the bounds and the estimator variances in order to benchmark the proposed estimators the mse and relative bias of the parameter estimates are compared with those of the expectation maximization algorithm the proposed estimator has generally outperformed the benchmark algorithm the performance improvement is significant for the nakagami parameter varying from 9 db for a larger sample size to 24 db for a smaller sample size similarly the performance for the variance parameter varies from 1 5 db to 5 2 db with increasing the sample size the proposed algorithm is also tested on a simulated data set based on correlated latent variables and the estimator is found to be effective for weakly correlated data finally for the hidden signal the estimated density from the proposed method is found to be having lower kullback leibler divergence in comparison to that of the expectation maximization algorithm

mixture based extension of the ar model and its recursive bayesian identification an extension of the autoregressive ar model is studied which allows transformations and distortions on the regressor to be handled many important signal processing problems are amenable to this extended ar i e ear model it is shown that bayesian identification and prediction of the ear model can be performed recursively in common with the ar model itself the ear model does however require that the transformation be known when it is unknown the associated transformation space is represented by a finite set of candidates what follows is a mixture based ear model i e the mear model an approximate identification algorithm for mear is developed using a restricted variational bayes vb method this restores the elegant recursive update of sufficient statistics the mear model is applied to the robust identification of ar processes corrupted by outliers and burst noise respectively and to click removal for speech

music analysis using hidden markov mixture models we develop a hidden markov mixture model based on a dirichlet process dp prior for representation of the statistics of sequential data for which a single hidden markov model hmm may not be sufficient the dp prior has an intrinsic clustering property that encourages parameter sharing and this naturally reveals the proper number of mixture components the evaluation of posterior distributions for all model parameters is achieved in two ways 1 via a rigorous markov chain monte carlo method and 2 approximately and efficiently via a variational bayes formulation using dp hmm mixture models in a bayesian setting we propose a novel scheme for music analysis highlighting the effectiveness of the dp hmm mixture model music is treated as a time series data sequence and each music piece is represented as a mixture of hmms we approximate the similarity of two music pieces by computing the distance between the associated hmm mixtures experimental results are presented for synthesized sequential data and from classical music clips music similarities computed using dp hmm mixture modeling are compared to those computed from gaussian mixture modeling for which the mixture modeling is also performed using dp the results show that the performance of dp hmm mixture modeling exceeds that of the dp gaussian mixture modeling

computation of the exact information matrix of gaussian time series with stationary random components the paper presents an algorithm for efficient recursive computation of the fisher information matrix of gaussian time series whose random components are stationary and whose means and covariances are functions of a parameter vector the algorithm is first developed in a general framework and then specialized to the case of autoregressive moving average processes with possible additive white noise the asymptotic behavior of the algorithm is explored and a termination criterion is derived finally the algorithm is used to demonstrate the behavior of the exact cramer rao bound for unbiased estimates for some arma processes as a function of the number of data points it is shown that for processes with zeros near the unit circle and short data records the exact cramer rao bound differs dramatically from its common approximation based on asymptotic theory

parameter orthogonality and approximate conditional inference on propose une statistique du rapport de vraisemblance construite a partir de la distribution conditionnelle des observations etant donne les estimateurs du maximum de vraisemblance pour les parametres de nuisance

computation of the exact information matrix of gaussian dynamic regression time series models in this paper the computation of the exact fisher information matrix of a large class of gaussian time series models is considered this class which is often called the single input single output siso model includes dynamic regression with autocorrelated errors and the transfer function model with autoregressive moving average errors the method is based on a combination of two computational procedures recursions for the covariance matrix of the derivatives of the state vector with respect to the parameters and the fast kalman filter recursions used in the evaluation of the likelihood function it is much faster than existing procedures an expression for the asymptotic information matrix is also given

a suboptimal bootstrap method for structure detection of non linear output error models with application to human ankle dynamics identification of non linear systems involves estimating unknown parameters and structure detection selection of a subset of candidate terms that best describe the observed output this is a necessary procedure to compute an efficient system description which may afford greater insight into the functionality of the system or a simpler controller design for non linear systems simple output additive noise can generate multiplicative terms between the input output and noise the terms associated with noise need to be modelled to obtain unbiased parameter estimates significantly increasing the number of candidate terms to be estimated and considered in special cases it may be possible to use an output error oe model structure and the instrumental variable iv estimator to obtain unbiased parameters without modelling the noise this significantly reduces the dimensionality of the structure computation problem therefore in this paper we propose a suboptimal bootstrap structure detection sobsd algo

structure identification of nonlinear dynamic systems a survey on input output approaches abstract in the past several methods have been elaborated for the identification of nonlinear dynamic systems most of the methods assume that the structure of the system is given a priori therefore they are in reality parameter estimation algorithms and structure identification is thus usually performed by repeated parameter estimation however in nonlinear system theory several methods are known to determine the structure of a system in this paper structure identification of block oriented especially cascade models of semi linear dynamic models with signal dependent parameters and of nonlinear dynamic models being linear in the parameters will be considered different structure selection methods are summarized based on step and impulse tests frequency response measurements correlation analysis repeated reproducible tests and normal operating data

orthogonal parameter estimation algorithm for non linear stochastic systems an orthogonal parameter estimation algorithm is derived which allows each parameter in a linear in the parameters non linear difference equation model to be estimated sequentially and quite independently of the other parameters in the model the algorithm can be applied for any persistently exciting input and provides both unbiased estimates in the presence of correlated noise and an indication of which terms to include in the model several simulated examples are included to demonstrate the effectiveness of the algorithm

nonlinear black box modeling in system identification a unified overview a nonlinear black box structure for a dynamical system is a model structure that is prepared to describe virtually any nonlinear dynamics there has been considerable recent interest in this area with structures based on neural networks radial basis networks wavelet networks and hinging hyperplanes as well as wavelet transform based methods and models based on fuzzy sets and fuzzy rules this paper describes all these approaches in a common framework from a user s perspective it focuses on what are the common features in the different approaches the choices that have to be made and what considerations are relevant for a successful system identification application of these techniques it is pointed out that the nonlinear structures can be seen as a concatenation of a mapping form observed data to a regression vector and a nonlinear mapping from the regressor space to the output space these mappings are discussed separately the latter mapping is usually formed as a basis function expansion the basis functions are typically formed from one simple scalar function which is modified in terms of scale and location the expansion from the scalar argument to the regressor space is achieved by a radial or a ridge type approach basic techniques for estimating the parameters in the structures are criterion minimization as well as two step procedures where first the relevant basis functions are determined using data and then a linear least squares step to determine the coordinates of the function approximation a particular problem is to deal with the large number of potentially necessary parameters this is handled by making the number of used parameters considerably less than the number of offered parameters by regularization shrinking pruning or regressor selection

variational bayes for generalized autoregressive models we describe a variational bayes vb learning algorithm for generalized autoregressive gar models the noise is modeled as a mixture of gaussians rather than the usual single gaussian this allows different data points to be associated with different noise levels and effectively provides robust estimation of ar coefficients the vb framework is used to prevent overfitting and provides model order selection criteria both for ar order and noise model order we show that for the special case of gaussian noise and uninformative priors on the noise and weight precisions the vb framework reduces to the bayesian evidence framework the algorithm is applied to synthetic and real data with encouraging results