balanced incomplete factorization in this paper we present a new incomplete factorization of a square matrix into triangular factors in which we get standard lu or ldl t factors direct factors and their inverses inverse factors at the same time algorithmically we derive this method from the approach based on the sherman morrison formula r bru j cerdan j marin and j mas siam j sci comput 25 2003 pp 701 715 in contrast to the robust incomplete decomposition rif algorithm m benzi and m t ma numer linear algebra appl 10 2003 pp 385 400 the direct and inverse factors here directly influence each other throughout the computation consequently the algorithm to compute the approximate factors may mutually balance dropping in the factors and control their conditioning in this way for the symmetric positive definite case we derive the theory and present an algorithm for computing the incomplete ldl t factorization and we discuss experimental results we call this new approximate ldl t factorization the balanced incomplete factorization bif our experimental results confirm that this factorization is very robust and may be useful in solving difficult ill conditioned problems by preconditioned iterative methods moreover the internal coupling of the computation of direct and inverse factors results in much shorter setup times times to compute approximate decomposition than rif a method of a similar and very high level of robustness we also derive and present the theory for the general nonsymmetric case but do not discuss its implementation

analysis of subspace fitting and ml techniques for parameter estimation from sensor array data it is shown that the multidimensional signal subspace method termed weighted subspace fitting wsf is asymptotically efficient this results in a novel compact matrix expression for the cramer rao bound crb on the estimation error variance the asymptotic analysis of the maximum likelihood ml and wsf methods is extended to deterministic emitter signals the asymptotic properties of the estimates for this case are shown to be identical to the gaussian emitter signal case i e independent of the actual signal waveforms conclusions concerning the modeling aspect of the sensor array problem are drawn

subspace fitting approaches for frequency estimation using real valued data a novel data covariance model has recently been proposed for the subspace based estimation of multiple real valued sine wave frequencies in this paper we develop weighted subspace fitting approaches using this new data model a new parameterization of the noise subspace is proposed this parameterization is used to solve the subspace fitting problem analytically an expression for the residual covariance matrix is derived this covariance matrix is further used to obtain an optimally weighted gauss markov estimator a computationally efficient suboptimal weighting is also proposed and the associated estimator is close to the gauss markov estimator in performance the suboptimal weighting strategy is quite general and can be used in other related applications the performance of the algorithms are illustrated using numerical simulations the proposed subspace fitting approach shows improved resolution performance it is also robust to additive noise

multiple emitter location and signal parameter estimation processing the signals received on an array of sensors for the location of the emitter is of great enough interest to have been treated under many special case assumptions the general problem considers sensors with arbitrary locations and arbitrary directional characteristics gain phase polarization in a noise interference environment of arbitrary covariance matrix this report is concerned first with the multiple emitter aspect of this problem and second with the generality of solution a description is given of the multiple signal classification music algorithm which provides asymptotically unbiased estimates of 1 number of incident wavefronts present 2 directions of arrival doa or emitter locations 3 strengths and cross correlations among the incident waveforms 4 noise interference strength examples and comparisons with methods based on maximum likelihood ml and maximum entropy me as well as conventional beamforming are included an example of its use as a multiple frequency estimator operating on time series is included

a general framework for nonlinear multigrid inversion a variety of new imaging modalities such as optical diffusion tomography require the inversion of a forward problem that is modeled by the solution to a three dimensional 3d partial differential equation for these applications image reconstruction is particularly difficult because the forward problem is both nonlinear and computationally expensive to evaluate we propose a general framework for nonlinear multigrid inversion that is applicable to a wide variety of inverse problems the multigrid inversion algorithm results from the application of recursive multigrid techniques to the solution of optimization problems arising from inverse problems the method works by dynamically adjusting the cost functionals at different scales so that they are consistent with and ultimately reduce the finest scale cost functional in this way the multigrid inversion algorithm efficiently computes the solution to the desired fine scale inversion problem importantly the new algorithm can greatly reduce computation because both the forward and inverse problems are more coarsely discretized at lower resolutions an application of our method to bayesian optical diffusion tomography with a generalized gaussian markov random field image prior model shows the potential for very large computational savings numerical data also indicates robust convergence with a range of initialization conditions for this nonconvex optimization problem

detection of coherent signals using weighted subspace smoothing a new approach to detecting the number of coherent signals incident upon a uniformly spaced linear array is presented the approach combines a modified spatial smoothing scheme and a modified mdl criterion the modified spatial smoothing scheme referred to as weighted subspace smoothing is actually a generalization of the post smoothing approach proposed by krim and proakis in 1994 it is shown that the noise eigenvalues obtained with weighted subspace smoothing are more accurate than those obtained with the original spatial smoothing we thus attempt to improve the detection performance using these more accurate eigenvalues a novel modification of the minimum description length mdl criterion is proposed to accomplish this computer simulations are presented to study the performance improvement of this new approach

a generalised instrumental variable estimator for multivariable errors in variables identification problems a new and very general estimation method the generalised instrumental variable estimator give has been introduced to deal with the errors in variables identification the give contains several well known estimation methods as special cases that can be obtained by various user choices in give these methods include the bias eliminating least squares method a number of different versions of the frisch scheme and the extended compensated least squares method this article shows how previous analysis for single input single output systems can be extended in a non trivial way to also cope with multi input multi output systems various computational aspects of give are presented and it is shown how the parameter estimates can be computed in an efficient way the asymptotic distribution of the parameter estimates is also derived and analysed

maximum likelihood estimation of the parameters of multiple sinusoids from noisy measurements the problem of estimating the frequencies phases and amplitudes of sinusoidal signals is considered a simplified maximum likelihood gauss newton algorithm which provides asymptotically efficient estimates of these parameters is proposed initial estimates for this algorithm are obtained by a variation of the overdetermined yule walker method and periodogram based procedure use of the maximum likelihood gauss newton algorithm is not however limited to this particular initialization method some other possibilities to get suitable initial estimates are briefly discussed an analytical and numerical study of the shape of the likelihood function associated with the sinusoids in noise process reveals its multimodal structure and clearly sets the importance of the initialization procedure some numerical examples are presented to illustrate the performance of the proposed estimation procedure comparison to the performance corresponding to the cramer rao lower bound is also presented using a simple expression for the asymptotic cramer rao bound covariance matrix derived in the paper

technical communique modified aic rule for model selection in combination with prior estimated noise models in this paper we adapt the aic model selection rule to be used in combination with prior estimated noise models in the presence of model errors it is shown that under these conditions a slightly modified cost function extension is needed resulting in a multiplicative complexity term instead of an additive one as would be found for fixed noise models and no model errors the equivalency of this result with the classical prediction error framework is discussed

maximum likelihood estimation of the non parametric frf for pulse like excitations this technical note introduces the closed form maximum likelihood estimator for estimating the coefficients of the non parametric frequency response function from system identification experiments it is assumed that the experiments consist of repeated pulse excitations and that both the excitation and system response are measured which leads to an error in variables setting monte carlo simulations indicate that the estimator achieves efficiency at low signal to noise ratios with only few measurements comparison with the least squares estimator shows that better unbiased results are obtained

an improved bias compensation approach for errors in variables model identification parametric estimation of the dynamic errors in variables models is considered in this paper in particular a bias compensation approach is examined in a generalized framework sufficient conditions for uniqueness of the identified model are presented subsequently a statistical accuracy analysis of the estimation algorithm is carried out the asymptotic covariance matrix of the system parameter estimates depends on a user chosen filter and a certain weighting matrix it is shown how these can be tuned to boost the estimation performance the numerical simulation results suggest that the covariance matrix of the estimated parameter vector is very close to the cramer rao lower bound for the estimation problem

time delay and spatial signature estimation using known asynchronous signals this paper considers the problem of estimating the parameters of known signals received asynchronously by an array of antennas the parameters of primary interest are the time delays of the signals and their spatial signatures at the array estimates of the signal directions of arrival are also considered but are of secondary importance in this work maximum likelihood algorithms and more computationally efficient approximations are developed for both the case where all received signals are identical the channel estimation overlapping echo problem and where they are all distinct conditions are also derived under which the standard matched filter approach yields consistent and statistically efficient parameter estimates the issue of solution uniqueness is addressed and in particular an upper bound on the number of signals whose parameters may be uniquely identified is derived for a number of different cases typically the bound is far greater than the number of sensors and is limited only by the number of data samples collected some representative simulation examples are also included to illustrate the algorithms performance relative to the cramer rao bound

estimation of the parameters of sinusoidal signals in non gaussian noise accurate estimation of the amplitude and frequency parameters of sinusoidal signals from noisy observations is an important problem in many signal processing applications in this paper the problem is investigated under the assumption of non gaussian noise in general and laplace noise in particular it is proven mathematically that the maximum likelihood estimator derived under the condition of laplace white noise is able to attain an asymptotic cramer rao lower bound which is one half of that achieved by periodogram maximization and nonlinear least squares it is also proven that when applied to non laplace situations the laplace maximum likelihood estimator which may also be referred to as the nonlinear least absolute deviations estimator can achieve an even higher statistical efficiency especially when the noise distribution has heavy tails a computational procedure is proposed to overcome the difficulty of local extrema in the likelihood function simulation results are provided to validate the analytical findings

worst case identification of errors in variables models in closed loop a worst case identification method in frequency domain is proposed to cope with the identification of errors in variables models eivms in closed loop with a priori bound for the disturbing noises of an eivm in closed loop a frequency domain normalized coprime factor model ncfm with perturbation is derived and thus the identification of the eivm becomes that of the ncfm by employing the v gap metric as an optimization criterion the worst case error for an identified nominal ncfm is easily quantified and the parameter optimization can be effectively solved by linear matrix inequalities lmis during the parameter optimization the derivative of the nominal ncfm is constrained to some degree to reduce the effect of overfitting phenomenon different from other eivm identification methods we use v gap metric to characterize the disturbing noises and quantify the worst case error for the nominal ncfm as a result the identification result is not a deterministic model but a model set moreover this model set can be perfectly combined with the robust controller design finally a numerical simulation is presented to verify the proposed method

brief box jenkins continuous time modeling this paper treats the identification of continuous time models using arbitrary band limited excitation signals a modeling approach is presented that has the following two advantages 1 asymptotically the amount of data tends to infinity there is no approximation error over the complete frequency band from dc to nyquist 2 it allows to identify general parametric noise models the key idea is to combine a continuous time plant model with a discrete time noise model hybrid box jenkins model structure

a robust ilu with pivoting based on monitoring the growth of the inverse factors abstract an incomplete lu decomposition with pivoting is presented that progressively monitors the growth of the inverse factors of l u the information on the growth of the inverse factors is used as feedback for dropping entries in l and u this method often yields a robust and effective preconditioner especially when the system is highly indefinite numerical examples demonstrate the effectiveness of this approach

maximum likelihood methods for direction of arrival estimation five methods of direction of arrival doa estimation which can be derived from the maximum likelihood ml principle are considered the ml method mlm results from the application of the ml principle to the statistics of the observed raw data the standard multiple signal classification music procedure called music 1 is obtained as a brute force approximation of the mlm an improved music procedure named music 2 is obtained by applying the ml principle to the statistics of certain linear combinations of the sample noise space eigenvectors a procedure which compromises between the good performance of the mlm and the computational simplicity of music is a method of direction estimation mode 1 which is derived as a large sample realization of the mlm a fifth method called mode 2 is obtained by using the ml principle on the statistics of certain linear combinations of the sample eigenvectors mode 2 is computationally less demanding than the mlm it is of the same complexity as mode 1 and statistically more efficient a numerical comparison of these five doa estimation methods is presented it confirms the analytic results on their theoretical performance levels

a multilevel proximal gradient algorithm for a class of composite optimization problems composite optimization models consist of the minimization of the sum of a smooth not necessarily convex function and a nonsmooth convex function such models arise in many applications where in addition to the composite nature of the objective function a hierarchy of models is readily available it is common to take advantage of this hierarchy of models by first solving a low fidelity model and then using the solution as a starting point to a high fidelity model we adopt an optimization point of view and show how to take advantage of the availability of a hierarchy of models in a consistent manner we do not use the low fidelity model just for the computation of promising starting points but also for the computation of search directions we establish the convergence and convergence rate of the proposed algorithm our numerical experiments on large scale image restoration problems and the transition path problem suggest that for certain classes of problems the proposed algorithm is significantly fas

ilu preconditioners for nonsymmetric saddle point matrices with application to the incompressible navier stokes equations motivated by the numerical solution of the linearized incompressible navier stokes equations we study threshold incomplete lu factorizations for nonsymmetric saddle point matrices the resulting preconditioners are used to accelerate the convergence of a krylov subspace method applied to finite element discretizations of fluid dynamics problems in three space dimensions the paper presents and examines an extension for nonsymmetric matrices of the tismenetsky kaporin incomplete factorization it is shown that in numerically challenging cases of higher reynolds number flows one benefits from using this two parameter modification of a standard threshold ilu preconditioner the performance of the ilu preconditioners is studied numerically for a wide range of flow and discretization parameters and the efficiency of the approach is shown if threshold parameters are chosen suitably the practical utility of the method is further demonstrated for the haemodynamic problem of simulating blood flow in a right c

statistical analysis of music and subspace rotation estimates of sinusoidal frequencies consideration is given to the analysis of the large sample second order properties of multiple signal classification music and subspace rotation sur methods such as esprit for sinusoidal frequency estimation explicit expressions for the covariance elements of the estimation errors associated with either method are derived these expressions of covariances are then used to analyze and compare the statistical performances of the music and sur estimation sure methods both music and sure are based on the eigendecomposition of a sample data covariance matrix the expressions for the estimation error variances derived are used to study the dependence of music and sure performances on the dimension of the data covariance matrix used

a robust and efficient ilu that incorporates the growth of the inverse triangular factors in this paper we present a new ilu decomposition which is based on an existing sparse direct solver in contrast to many incomplete lu decompositions this ilu incorporates information about the inverse factors l 1 and u 1 which have direct influence on the dropping strategy we demonstrate in several large scale examples that this implementation constructs a robust preconditioner

multigrid methods for pde optimization research on multigrid methods for optimization problems is reviewed optimization problems considered include shape design parameter optimization and optimal control problems governed by partial differential equations of elliptic parabolic and hyperbolic type

orderings for factorized sparse approximate inverse preconditioners the influence of reorderings on the performance of factorized sparse approximate inverse preconditioners is considered some theoretical results on the effect of orderings on the fill in and decay behavior of the inverse factors of a sparse matrix are presented it is shown experimentally that certain reorderings like minimum degree and nested dissection can be very beneficial the benefit consists of a reduction in the storage and time required for constructing the preconditioner and of faster convergence of the preconditioned iteration in many cases of practical interest

doa estimation by exploiting the symmetric configuration of uniform linear array a new direction of arrival doa estimation method is proposed for uniform linear array in this paper when both uncorrelated and coherent sources are present the uncorrelated and coherent sources are estimated at two different stages the uncorrelated sources are first estimated using conventional subspace methods and then they are eliminated by exploiting the symmetric configuration of the array finally the remaining coherent sources are estimated by reconstructing a non toeplitz matrix the number of sources resolved by our method can exceed the number of array elements simulation results demonstrate the effectiveness and efficiency of our proposed method

esprit estimation of signal parameters via rotational invariance techniques an approach to the general problem of signal parameter estimation is described the algorithm differs from its predecessor in that a total least squares rather than a standard least squares criterion is used although discussed in the context of direction of arrival estimation esprit can be applied to a wide variety of problems including accurate detection and estimation of sinusoids in noise it exploits an underlying rotational invariance among signal subspaces induced by an array of sensors with a translational invariance structure the technique when applicable manifests significant performance and computational advantages over previous algorithms such as mem capon s mlm and music

bootstrap amg we develop an algebraic multigrid amg setup scheme based on the bootstrap framework for multiscale scientific computation our approach uses a weighted least squares definition of interpolation based on a set of test vectors that are computed by a bootstrap setup cycle and then improved by a multigrid eigensolver and a local residual based adaptive relaxation process to emphasize the robustness efficiency and flexibility of the individual components of the proposed approach we include extensive numerical results of the method applied to scalar elliptic partial differential equations discretized on structured meshes as a first test problem we consider the laplace equation discretized on a uniform quadrilateral mesh a problem for which multigrid is well understood then we consider various more challenging variable coefficient systems coming from covariant finite difference approximations of the two dimensional gauge laplacian system a commonly used model problem in amg algorithm development for linear systems arising in lattice field theory computations

a sparse approximate inverse preconditioner for the conjugate gradient method a method for computing a sparse incomplete factorization of the inverse of a symmetric positive definite matrix a is developed and the resulting factorized sparse approximate inverse is used as an explicit preconditioner for conjugate gradient calculations it is proved that in exact arithmetic the preconditioner is well defined if a is an h matrix the results of numerical experiments are presented

frequency domain sample maximum likelihood estimation for spatially dependent parameter estimation in pdes the identification of the spatially dependent parameters in partial differential equations pdes is important in both physics and control problems a methodology is presented to identify spatially dependent parameters from spatio temporal measurements local non rational transfer functions are derived based on three local measurements allowing for a local estimate of the parameters a sample maximum likelihood estimator smle in the frequency domain is used because it takes noise properties into account and allows for high accuracy consistent parameter estimation confidence bounds on the parameters are estimated based on the noise properties of the measurements this method is successfully applied to the simulations of a finite difference model of a parabolic pde with piecewise constant parameters

preconditioning techniques for large linear systems a survey this article surveys preconditioning techniques for the iterative solution of large linear systems with a focus on algebraic methods suitable for general sparse matrices covered topics include progress in incomplete factorization methods sparse approximate inverses reorderings parallelization issues and block and multilevel extensions some of the challenges ahead are also discussed an extensive bibliography completes the paper

numerically robust transfer function modeling from noisy frequency domain data using vector orthogonal polynomials as basis functions for the representation of the rational form of a linear time invariant system in frequency domain identification problems it is shown that the notorious numerical ill conditioning of these maximum likelihood problems can be overcome completely for the identification of high order 100 100 systems operating over a wide frequency band or even in the situation of over or undermodeling condition numbers less than ten are reported for real measurements

optimal utilization of signal free samples for array processing in unknown colored noise fields many algorithms for direction of arrival doa estimation require the noise covariance matrix to be known or to possess a known structure in many cases the noise covariance is in fact estimated from separate measurements this paper addresses the combined effects of finite sample sizes both in the estimated noise covariance matrix and in the data with signals present it is assumed that a batch of signal free samples is available in addition to the signal containing samples no assumption is made on the structure of the noise covariance in this paper the asymptotic covariance of the weighted subspace fitting wsf algorithm is derived for the case in which the data are whitened using an estimated noise covariance the expression obtained suggests an optimal weighting that improves performance compared to the standard choice in addition a new method based on covariance matching is proposed both methods are asymptotically statistically efficient the cramer rao lower bound crb on the covariance of the estimate for the data model is also derived monte carlo simulations show promising small sample performance for the two new methods and confirm the asymptotic results

a line search multigrid method for large scale nonlinear optimization we present a line search multigrid method for solving discretized versions of general unconstrained infinite dimensional optimization problems at each iteration on each level the algorithm computes either a direct search direction on the current level or a recursive search direction from coarser level models introducing a new condition that must be satisfied by a backtracking line search procedure the recursive search direction is guaranteed to be a descent direction global convergence is proved under fairly minimal requirements on the minimization method used at all grid levels using a limited memory bfgs quasi newton method to produce the direct search direction preliminary numerical experiments show that our line search multigrid approach is promising

brief paper study of conditional ml estimators in time and frequency domain system identification this paper shows that the full covariance matrix can be replaced by its main diagonal in maximum likelihood estimation mle of linear dynamic systems in the time or frequency domain without affecting the asymptotic properties this is a very important result for frequency domain identification since it allows to use nonparametric noise models it also provides a theoretical justification for the common practice in time domain identification to neglect the initial condition effects of the noise model

modification and compensation strategies for threshold based incomplete factorizations standard single level incomplete factorization preconditioners are known to successfully accelerate krylov subspace iterations for many linear systems the classical modified incomplete lu milu factorization approach improves the acceleration given by standard ilu approaches by modifying the nonunit diagonal in the factorization to match the action of the system matrix on a given vector typically the constant vector here we examine the role of similar modifications within the dual threshold ilut algorithm we introduce column and row variants of the modified ilut algorithm and discuss optimal ways of modifying the columns or rows of the computed factors to improve their accuracy and stability modifications are considered for both the diagonal and off diagonal entries of the factors based on one or many vectors chosen a priori or through an arnoldi iteration numerical results are presented to support our findings

music maximum likelihood and cramer rao bound the performance of the music and ml methods is studied and their statistical efficiency is analyzed the cramer rao bound crb for the estimation problems is derived and some useful properties of the crb covariance matrix are established the relationship between the music and ml estimators is investigated as well a numerical study is reported of the statistical efficiency of the music estimator for the problem of finding the directions of two plane waves using a uniform linear array an exact description of the results is included

frequency domain system identification using non parametric noise models estimated from a small number of data sets abstract this paper discusses the problem of identifying a linear system from the frequency data when the measurements of the input and the output signals are both disturbed with noise a typical example of such a problem is the identification of a system in a feedback loop it is known that this problem can be solved using errors in varaibles methods if the covariance matrices of the disturbing noise on input and output measurements are a priori known it is shown that the exact covariance matrices can be replaced by the sample covariance matrices the system can be identified from the sample means and sample covariance matrices calculated from a small number m of independently repeated experiments it is shown that under these conditions the estimates are still strongly consistent for an increasing number of data points n in each experiment n if m 4 the loss in efficiency is quantified m 6 and the expected value of the cost function m 4 and its variance m 6 are calculated

uncertainty of transfer function modelling using prior estimated noise models assuming small model errors unmodelled dynamics and or nonlinear distortions and large signal to noise ratios we derive in this paper explicit expressions for the covariance matrix of a frequency domain estimator using prior estimated noise models these analytic expressions i give a clear insight in the behaviour of the covariance matrix as a function of the signal to noise ratio the unmodelled dynamics and the nonlinear distortions and ii allow to predict accurately the order of magnitude of the actual uncertainty of the estimates the link with the classical prediction error approach is also established

on some nonparametric methods for detection of the number of signals a simpler more elegant and more flexible method for estimating the number of signals is proposed strong consistency is proved bounds for error probabilities are given this method depends on choosing a function with a parameter r the previous methods correspond to the case r 1 it seems the larger the r the faster the convergence we can choose r as any positive integer even applications are discussed including the coherent cases

detection of signals by information theoretic criteria a new approach is presented to the problem of detecting the number of signals in a multichannel time series based on the application of the information theoretic criteria for model selection introduced by akaike aic and by schwartz and rissanen mdl unlike the conventional hypothesis testing based approach the new approach does not requite any subjective threshold settings the number of signals is obtained merely by minimizing the aic or the mdl criteria simulation results that illustrate the performance of the new method for the detection of the number of signals received by a sensor array are presented

an optimal method for stochastic composite optimization this paper considers an important class of convex programming cp problems namely the stochastic composite optimization sco whose objective function is given by the summation of general nonsmooth and smooth stochastic components since sco covers non smooth smooth and stochastic cp as certain special cases a valid lower bound on the rate of convergence for solving these problems is known from the classic complexity theory of convex programming note however that the optimization algorithms that can achieve this lower bound had never been developed in this paper we show that the simple mirror descent stochastic approximation method exhibits the best known rate of convergence for solving these problems our major contribution is to introduce the accelerated stochastic approximation ac sa algorithm based on nesterov s optimal method for smooth cp nesterov in doklady an sssr 269 543 547 1983 nesterov in math program 103 127 152 2005 and show that the ac sa algorithm can achieve the aforementioned lower bound on the rate of convergence for sco to the best of our knowledge it is also the first universally optimal algorithm in the literature for solving non smooth smooth and stochastic cp problems we illustrate the significant advantages of the ac sa algorithm over existing methods in the context of solving a special but broad class of stochastic programming problems

technical communique generating robust starting values for frequency domain transfer function estimation this paper proposes a frequency dependent weighting function that when used together with the classical starting value algorithms linear least squares total least squares improves starting values for frequency domain identification of parametric rational transfer function models the idea behind the method is to approximate the weighting of the maximum likelihood estimator without prior knowledge of the model parameters the proposed weighting scheme is applied to simulation and measurement data enhanced parameter estimates are obtained even for wide band high order systems with large transfer function dynamics

on parameter estimation using nonparametric noise models fitting multidimensional parametric models in frequency domain using nonparametric noise models is considered in this paper a nonparametric estimate of the noise statistics is obtained from a finite number of independent data sets the estimated noise model is then substituted for the the true noise covariance matrix in the maximum likelihood loss function to obtain suboptimal parameter estimates the goal here is to present an analysis of the resulting estimates sufficient conditions for consistency are derived and an asymptotic accuracy analysis is carried out the first and second order statistics of the cost function at the global minimum point are also explored which can be used for model validation the analytical findings are validated using numerical simulation results

robust approximate inverse preconditioning for the conjugate gradient method we present a variant of the ainv factorized sparse approximate inverse algorithm which is applicable to any symmetric positive definite matrix the new preconditioner is breakdown free and when used in conjunction with the conjugate gradient method results in a reliable solver for highly ill conditioned linear systems we also investigate an alternative approach to a stable approximate inverse algorithm based on the idea of diagonally compensated reduction of matrix entries the results of numerical tests on challenging linear systems arising from finite element modeling of elasticity and diffusion problems are presented

identification of linear systems with nonlinear distortions this paper studies the impact of nonlinear distortions on linear system identification it collects a number of previously published methods in a fully integrated approach to measure and model these systems from experimental data first a theoretical framework is proposed that extends the linear system description to include the impact of nonlinear distortions the nonlinear system is replaced by a linear model plus a nonlinear noise source the class of nonlinear systems covered by this approach is described and the properties of the extended linear representation are studied these results are used to design the experiments to detect the level of the nonlinear distortions to measure efficiently the best linear approximation to reveal the even or odd nature of the nonlinearity to identify a parametric linear model and to improve the model selection procedures in the presence of nonlinear distortions

blind maximum likelihood identification of wiener systems this paper is about the identification of discrete time wiener systems from output measurements only blind identification assuming that the unobserved input is white gaussian noise that the static nonlinearity is invertible and that the output is observed without errors a gaussian maximum likelihood estimator is constructed its asymptotic properties are analyzed and the cramer rao lower bound is calculated a two step procedure for generating high quality initial estimates is presented as well the paper includes the illustration of the method on a simulation example

an algorithm for pole zero modeling and spectral analysis an explicit connection between fitting exponential models and pole zero models to observed data is made the fitting problem is formulated as a constrained nonlinear minimization problem this problem is then solved using a simplified iterative algorithm the algorithm is applied to simulated data and the performance of the algorithm is compared to previous results

modified pisarenko harmonic decomposition for single tone frequency estimation in this correspondence based on an alternative derivation of the pisarenko harmonic decomposition phd method a new asymptotically unbiased estimator for the frequency of a single real tone in white noise is devised with the use of novel sample covariance expressions furthermore extension to sample covariances with higher lags for performance enhancement is investigated while a simple and effective scheme is suggested to resolve the corresponding frequency ambiguity problem the variance of the modified pisarenko s method is also derived which is then utilized to find the best estimate among all admissible solutions from various sets of sample covariances computer simulations are included to corroborate the theoretical development and to demonstrate that the proposed approach outperforms several existing low complexity frequency estimators in terms of nearly uniform performance and estimation accuracy

frequency domain identification of linear systems using arbitrary excitations and a nonparametric noise model presents a generalized frequency domain identification method to identify single input single output siso systems combining two previously published extensions in one method arbitrary but persistent excitations are allowed and a nonparametric noise model is extracted from the same data that are used to identify the system the method is directly applicable to identification in feedback if an external persistently exciting reference signal is available

the use of nonparametric noise models extracted from overlapping subrecords for system identification in this paper we study the asymptotic properties consistency and asymptotic efficiency of a frequency domain errors in variables estimator using data extracted from overlapping subrecords while the classical approach without overlap needs six consecutive periods we show in this paper that using overlapping subrecords consistent estimators can be found with only two periods of the steady state response to a periodic excitation this is done using overlapping subrecords moreover the system identification procedure developed for data extracted from independent periods is shown to be valid for data extracted from overlapping subrecords this allows the user to reduce the measurement time considerably without changing the identification procedure

on closed loop system identification using polyspectral analysis given noisy input output time domain data parametric models of linear systems operating in closed loop can be consistently identified from time domain noisy input output measurements using cumulant spectra without requiring parametric modeling of noise two algorithms are proposed and their asymptotic performance is analyzed

factorized approximate inverses with adaptive dropping this paper presents a new approach to constructing factorized approximate inverses for a symmetric and positive definite matrix a the proposed strategy is based on adaptive dropping that reflects the quality of preserving the relation uz i between the direct factor u and the inverse factor z satisfying a u tu and a 1 zz t an important part of the approach is column pivoting used to minimize the growth of the condition number of leading principal submatrices of u that occurs explicitly in the dropping criterion numerical experiments demonstrate that the resulting approximate inverse factorization is robust as a preconditioner for solving large and sparse systems of linear equations

multilevel preconditioners constructed from inverse based ilus this paper analyzes dropping strategies in a multilevel incomplete lu decomposition context and presents a few strategies for obtaining related ilus with enhanced robustness the analysis shows that the incomplete lu factorization resulting from dropping small entries in gaussian elimination produces a good preconditioner when the inverses of these factors have norms that are not too large as a consequence a few strategies are developed whose goal is to achieve this feature a number of templates for enabling implementations of these factorizations are presented numerical experiments show that the resulting ilus offer a good compromise between robustness and efficiency

estimation for the multivariate errors in variables model with estimated error covariance matrix the errors in variables model in which the unobserved true values satisfy multiple linear restrictions is considered under the assumptions that the unobservable true values are normally distributed and that an estimator of the covariance matrix of the measurement error is available the maximum likelihood estimators are derived the limiting properties of the estimators are obtained for a wide range of assumptions including the assumption of fixed true values

frequency domain subspace system identification using non parametric noise models in the general case of non uniformly spaced frequency domain data and or arbitrarily coloured disturbing noise the frequency domain subspace identification algorithms described in mckelvey akcay and ljung ieee trans automatic control 41 7 1996 960 and van overschee and de moor signal processing 52 2 1996 179 are consistent only if the covariance matrix of the disturbing noise is known this paper studies the asymptotic properties strong convergence convergence rate asymptotic normality strong consistency and loss in efficiency of these algorithms when the true noise covariance matrix is replaced by the sample noise covariance matrix obtained from a small number of independent repeated experiments as an additional result the strong convergence in case of model errors the convergence rate and the asymptotic normality of the subspace algorithms with known noise covariance matrix follows

estimating linear statistical relationships this paper on estimating linear statistical relationships includes three lectures on linear functional and structural relationships factor analysis and simultaneous equations models the emphasis is on relating the several models by a general approach and on the similarity of maximum likelihood estimators under normality in the different models in the first two lectures the observable vector is decomposed into a systematic part and a random error the systematic part satisfies the linear relationships estimators are derived for several cases and some of their properties given estimation of the coefficients of a single equation in a simultaneous equations model is shown to be equivalent to estimation of linear functional relationships

markov based eigenanalysis method for frequency estimation this paper proposes an eigenanalysis based method for estimating the frequencies of complex valued sine waves the basic idea behind this method consists of using a set of linearly independent vectors that are orthogonal to the signal subspace spanned by the principal eigenvectors of the data covariance matrix exploiting that orthogonality condition gives an overdetermined system of linear equations the unknown parameters of which are uniquely related to the frequencies analytical expressions are derived for the covariances of the equation errors in the sample version of the aforementioned linear system of equations based on these expressions a markov like estimate of the unknown parameters is introduced which asymptotically with respect to either the number of data samples or the signal to noise ratio provides the minimum variance frequency estimates in a fairly large class of consistent estimators the paper includes monte carlo simulations that support the theoretical analysis results and show that those results may apply to scenarios with rather low values of the number of data samples and the signal to noise ratio

performance study of conditional and unconditional direction of arrival estimation a numerical and analytical study of conditional and unconditional direction of arrival doa estimation is presented explicit expressions for the unconditional cramer rao bounds on the doa estimation accuracy and the covariance matrix of the conditional maximum likelihood method are given it is shown that many doa estimation methods have the same asymptotic statistical properties under conditional and unconditional models the situation of two narrowband plane signals impinging on a uniformly spaced linear array is discussed

brief identification of static errors in variables models the rank reducibility problem the problem of identifying linear static relations from noisy data is investigated under the assumptions of the frisch scheme the work presents an attempt to simplify the search for solutions of the related rank reducibility problem the proposed approach is based on a relaxation of the original problem by means of algebraic manipulations of the non linear equations which characterize the solution set it is shown that this method can be advantageously used when particular conditions regarding the number of variables and the number of linear relations are satisfied

recursive trust region methods for multiscale nonlinear optimization a class of trust region methods is presented for solving unconstrained nonlinear and possibly nonconvex discretized optimization problems like those arising in systems governed by partial differential equations the algorithms in this class make use of the discretization level as a means of speeding up the computation of the step this use is recursive leading to true multilevel multiscale optimization methods reminiscent of multigrid methods in linear algebra and the solution of partial differential equations a simple algorithm of the class is then described and its numerical performance is shown to be numerically promising this observation then motivates a proof of global convergence to first order stationary points on the fine grid that is valid for all algorithms in the class

esprit like estimation of real valued sinusoidal frequencies subspace based estimation of multiple real valued sine wave frequencies is considered in this paper a novel data covariance model is proposed in the proposed model the dimension of the signal subspace equals the number of frequencies present in the data which is half of the signal subspace dimension for the conventional model consequently an esprit like algorithm using the proposed data model is presented the proposed algorithm is then extended for the case of complex valued sine waves performance analysis of the proposed algorithms are also carried out the algorithms are tested in numerical simulations when compared with esprit the newly proposed algorithm results in a significant reduction in computational burden without any compromise in the accuracy

errors in variables identification using maximum likelihood estimation in the frequency domain this paper deals with the identification of errors in variables eiv models corrupted by additive and uncorrelated white gaussian noises when the noise free input is an arbitrary signal not required to be periodic in particular a frequency domain maximum likelihood ml estimator is proposed and analyzed in some detail as some other eiv estimators this method assumes that the ratio of the noise variances is known the estimation problem is formulated in the frequency domain it is shown that the parameter estimates are consistent an explicit algorithm for computing the asymptotic covariance matrix of the parameter estimates is derived the possibility to effectively use lowpass filtered data by using only part of the frequency domain is discussed analyzed and illustrated

parametric and nonparametric identification of linear systems in the presence of nonlinear distortions a frequency domain approach this paper studies the asymptotic behavior of nonparametric and parametric frequency domain identification methods to model linear dynamic systems in the presence of nonlinear distortions under some general conditions for random multisine excitations in the first part a related linear dynamic system rlds approximation to the nonlinear system nls is defined and it is shown that the differences between the nls and the rlds can be modeled as stochastic variables with known properties in the second part a parametric model for the rlds is identified convergence in probability of this model to the rlds is proven a function of dependency is defined to detect and separate the presence of unmodeled dynamics and nonlinear distortions and to bound the bias error on the transfer function estimate

doa estimation of temporally and spatially correlated narrowband noncircular sources in spatially correlated white noise the main contribution of our work consists in developing for the first time a method of estimating the direction of arrival doa parameters assuming noncircular and temporally and spatially correlated signals this new approach based on a significant enhancement of the two sided instrumental variable signal subspace fitting iv ssf method outperforms its classical version in terms of lower bias and error variance moreover it will be shown that our new method is statistically more efficient than the mode method especially in the case of partly and fully coherent signals where only the extended and the classical two sided iv ssf methods are applicable we also derive an explicit expression for the stochastic cramer rao bound crb of the doa estimates from temporally and spatially correlated signals generated from noncircular sources the new crb is compared to those of circular temporally correlated and noncircular independent and identically distributed signals to show that the crb obtained assuming both noncircular sources and temporally correlated signals is lower than the crbs derived considering only one of these two assumptions this illustrates the potential gain that both noncircularity and temporal correlation provide when considered together it will also be proven that the difference between the three crbs increases with the number of snapshots however as the signal to noise ratio snr increases the crbs merge together and decrease linearly moreover at low snr values it will be shown that temporal correlation is more informative about the unknown doa parameters than noncircularity finally the crb derived assuming noncircular and temporally correlated signals depends on the noncircularity rate the circularity phase separation and the doa separation

a note on minimizing the prediction error when the zeros are restricted to the unit circle it is shown that the problem of minimizing the prediction error under the constraint that the zeros of the predictor polynomial be on the unit circle can be solved by using a modification of the levinson algorithm consequences of this result are discussed

exact maximum likelihood parameter estimation of superimposed exponential signals in noise a unified framework for the exact maximum likelihood estimation of the parameters of superimposed exponential signals in noise encompassing both the time series and the array problems is presented an exact expression for the ml criterion is derived in terms of the linear prediction polynomial of the signal and an iterative algorithm for the maximization of this criterion is presented the algorithm is equally applicable in the case of signal coherence in the array problem simulation shows the estimator to be capable of providing more accurate frequency estimates than currently existing techniques the algorithm is similar to those independently derived by kumaresan et al in addition to its practical value the present formulation is used to interpret previous methods such as prony s pisarenko s and modifications thereof

direction of arrival estimation for constant modulus signals in many cases where direction finding is of interest the signals impinging on an antenna array are known to be phase modulated and hence to have a constant modulus cm this is a strong property by itself it is already sufficient for source separation and can be used to construct improved direction finding algorithms we first derive the relevant cramer rao bounds crbs for arbitrary array configurations and specialize to uniform linear arrays we then propose a simple suboptimal direction estimation algorithm in which the signals are separated using the cm property followed by direction finding on the decoupled signals compared with the esprit algorithm and the crb for arbitrary signals the algorithm shows good results

a proximal method for composite minimization we consider minimization of functions that are compositions of convex or prox regular functions possibly extended valued with smooth vector functions a wide variety of important optimization problems fall into this framework we describe an algorithmic framework based on a subproblem constructed from a linearized approximation to the objective and a regularization term properties of local solutions of this subproblem underlie both a global convergence result and an identification property of the active manifold containing the solution of the original problem preliminary computational results on both convex and nonconvex examples are promising

errors in variables identification in dynamic networks consistency results for an instrumental variable approach in this paper we consider the identification of a linear module that is embedded in a dynamic network using noisy measurements of the internal variables of the network this is an extension of the errors in variables eiv identification framework to the case of dynamic networks the consequence of measuring the variables with sensor noise is that some prediction error identification methods no longer result in consistent estimates the method developed in this paper is based on a combination of the instrumental variable philosophy and closed loop prediction error identification methods and leads to consistent estimates of modules in a dynamic network we consider a flexible choice of which internal variables need to be measured in order to identify the module of interest this allows for a flexible sensor placement scheme we also present a method that can be used to validate the identified model

brief an estimator of the inverse covariance matrix and its application to ml parameter estimation in dynamical systems an exact formula of the inverse covariance matrix of an autoregressive stochastic process is obtained using the gohberg semencul explicit inverse of the toeplitz matrix this formula is used to build an estimator of the inverse covariance matrix of a stochastic process based on a single realization in this paper we show that this estimator can be conveniently applied to maximum likelihood parameter estimation in nonlinear dynamical system with correlated measurement noise the efficiency of the estimation scheme is illustrated via monte carlo simulations it is shown that the statistical properties of the estimated parameters are largely improved using the proposed inverse covariance matrix estimator in comparison to the classical variance estimator

simple effective computation of principal eigenvectors and their eigenvalues and application to high resolution estimation of frequencies we present the results of an investigation of the prony lanczos p l method 14 38 and the power method 39 for simple computation of approximations to a few eigenvectors and eigenvalues of a hermitian matrix we are motivated by realization of high resolution signal processing in an integrated circuit the computational speeds of the above methods are analyzed they are completely dependent on the speed of a matrix vector product operation if only a few eigenvalues or eigenvectors are needed the suggested methods can substitute for methods of the linpack or eispack subroutine libraries the accuracies of the suggested methods are evaluated using matrices formed from simulated data consisting of two sinusoids plus gaussian noise comparisons are made to the corresponding eigenvalues and eigenvectors obtained using linpack as a reference also the accuracies of frequency estimates obtained from the eigenvectors are compared

measurement and identification of nonlinear systems consisting of linear dynamic blocks and one static nonlinearity this paper considers the measurement and the identification of nonlinear time invariant single input single output siso systems consisting of a multivariable linear dynamic system and one static nonlinear siso system this includes wiener hammerstein systems in a linear feedback loop the nonparametric identification of the frequency response functions of the linear parts are obtained without measuring the signals over the static nonlinearity measurements on an electronic circuit demonstrate the usability of this identification scheme

the statistical performance of the music and the minimum norm algorithms in resolving plane waves in noise this paper presents an asymptotic statistical analysis of the null spectra of two eigen assisted methods music 1 and minimum norm 2 for resolving independent closely spaced plane waves in noise particular attention is paid to the average deviation of the null spectra from zero at the true angles of arrival for the plane waves these deviations are expressed as functions of signal to noise ratios number of array elements angular separation of emitters and the number of snapshots in the case of music an approximate expression is derived for the resolution threshold of two plane waves with equal power in noise this result is validated by monte carlo simulations

further results and insights on subspace based sinusoidal frequency estimation subspace based methods for parameter identification have received considerable attention in the literature starting with a scalar valued process it is well known that subspace based identification of sinusoidal frequencies is possible if the scalar valued data is windowed to form a low rank vector valued process music and esprit like estimators have for some time been applied to this vector model in addition a statistically attractive markov like procedure for this class of methods has been proposed herein the markov like procedure is reinvestigated several results regarding rank performance and structure are given in a compact manner the large sample equivalence with the approximate maximum likelihood method by stoica et al 1988 is also established

brief paper accuracy analysis of time domain maximum likelihood method and sample maximum likelihood method for errors in variables and output error identification for identifying errors in variables models the time domain maximum likelihood tml method and the sample maximum likelihood sml method are two approaches both methods give optimal estimation accuracy but under different assumptions in the tml method an important assumption is that the noise free input signal is modelled as a stationary process with rational spectrum for sml the noise free input needs to be periodic it is interesting to know which of these assumptions contain more information to boost the estimation performance in this paper the estimation accuracy of the two methods is analyzed statistically for both errors in variables eiv and output error models oem numerical comparisons between these two estimates are also done under different signal to noise ratios snrs the results suggest that tml and sml have similar estimation accuracy at moderate or high snr for eiv for oem identification these two methods have the same accuracy at any snr

maximum likelihood localization of multiple sources by alternating projection an algorithm referred to as apm for computing the maximum likelihood estimator of the locations of simple sources in passive sensor arrays is presented the algorithm is equally applicable to the case of coherent signals and to the case of a single snapshot the algorithm is iterative the maximum of the likelihood function is computed by successive approximations the convergence of the algorithm to the global maximum is demonstrated for a variety of scenarios the key to this global convergence is the initialization scheme

on the performance of signal subspace processing part i narrow band systems this paper presents an analytical evaluation of detection determination of the number of sources and estimation performances of narrow band signal subspace processing for multiple source direction finding the probabilities of underestimating and overestimating the number of sources are derived under asymptotic conditions and around the threshold regions in terms of the choice of a penalty function and signal noise and array parameters for the cases of at most two closely spaced sources in the spatially white noise a scalar measure is introduced for the evaluation of the quality of the estimated signal subspace based on the statistics of this measure performance thresholds are demonstrated for the signal to noise ratio angle separation and correlation between two equipowered sources

sensor array processing based on subspace fitting algorithms for estimating unknown signal parameters from the measured output of a sensor array are considered in connection with the subspace fitting problem the methods considered are the deterministic maximum likelihood method ml esprit and a recently proposed multidimensional signal subspace method these methods are formulated in a subspace fitting based framework which provides insight into their algebraic and asymptotic relations it is shown that by introducing a specific weighting matrix the multidimensional signal subspace method can achieve the same asymptotic properties as the ml method the asymptotic distribution of the estimation error is derived for a general subspace weighting and the weighting that provides minimum variance estimates is identified the resulting optimal technique is termed the weighted subspace fitting wsf method numerical examples indicate that the asymptotic variance of the wsf estimates coincides with the cramer rao bound the performance improvement compared to the other techniques is found to be most prominent for highly correlated signals

reduced complexity maximum likelihood direction of arrival estimation based on spatial aliasing spatial aliasing is an undesirable phenomenon that prevents the unique determination of the direction of arrival doa of impinging signals in array signal processing however the characteristics of spatial aliasing that generates ambiguous doas can be also used to reduce the computational complexity in maximum likelihood ml doa estimation this paper proposes a structural method to dramatically reduce the computational complexity of the ml doa estimation using the spatial aliasing generated by a nested array structure with a doubly scaled aper ture an ml full grid search is computationally simplified by the highly compressed searching range and the small number of candidate values to be searched which are derived based on spatial aliasing performance analyses based on the theoretical bounds and computational complexity with computer simulations show that the proposed method requires an extremely reduced computational load compared to the conventional ml doa estimation with a uniform linear array ula while achieving the performance enhancement in the estimation accuracy due to the enlarged array aperture

welch method revisited nonparametric power spectrum estimation via circular overlap the objective of this paper is twofold the first part provides further insight in the statistical properties of the welch power spectrum estimator a major drawback of the welch method reported in the literature is that the variance is not a monotonic decreasing function of the fraction of overlap selecting the optimal fraction of overlap which minimizes the variance is in general difficult since it depends on the window used we show that the explanation for the nonmonotonic behavior of the variance as reported in the literature does not hold in the second part this extra insight allows one to eliminate the nonmonotonic behavior of the variance for the welch power spectrum estimator pse by introducing a small modification to the welch method the main contributions of this paper are providing extra insight in the statistical properties of the welch pse modifying the welch pse to circular overlap the variance is a monotonically decreasing function of the fraction of overlap making the method more user friendly and an extra reduction of variance with respect to the welch pse without introducing systematic errors this reduction in variance is significant for a small number of data records only

a stochastic multiscale model for electricity generation capacity expansion abstract long term planning for electric power systems or capacity expansion has traditionally been modeled using simplified models or heuristics to approximate the short term dynamics however current trends such as increasing penetration of intermittent renewable generation and increased demand response requires a coupling of both the long and short term dynamics we present an efficient method for coupling multiple temporal scales using the framework of singular perturbation theory for the control of markov processes in continuous time we show that the uncertainties that exist in many energy planning problems in particular load demand uncertainty and uncertainties in generation availability can be captured with a multiscale model we then use a dimensionality reduction technique which is valid if the scale separation present in the model is large enough to derive a computationally tractable model we show that both wind data and electricity demand data do exhibit sufficient scale separation a numerical example using real data and a finite difference approximation of the hamilton jacobi bellman equation is used to illustrate the proposed method we compare the results of our approximate model with those of the exact model we also show that the proposed approximation outperforms a commonly used heuristic used in capacity expansion models