performance limitations of non minimum phase systems in the servomechanism problem abstract this paper studies the cheap regulator problem and the cheap servomechanism problem for systems which may be non minimum phase the study extends some well known properties of perfect regulation and the perfect tracking and disturbance rejection of minimum phase systems to non minimum phase systems it is shown that perfect rejection to disturbances applied to the plant input can be achieved no matter whether the system is minimum phase or non minimum phase whereas a fundamental limitation exists in the achievable transient performance of tracking and rejection to disturbances applied to the plant output for a non minimum phase system and that this limitation can be simply and completely characterized by the number and locations of those zeros of the system which lie in the right half of the complex plane furthermore this limitation provides a quantitative measure of the degree of difficulty which is inherent in the control of such non minimum phase systems

online blind source separation using incremental nonnegative matrix factorization with volume constraint online blind source separation bss is proposed to overcome the high computational cost problem which limits the practical applications of traditional batch bss algorithms however the existing online bss methods are mainly used to separate independent or uncorrelated sources recently nonnegative matrix factorization nmf shows great potential to separate the correlative sources where some constraints are often imposed to overcome the non uniqueness of the factorization in this paper an incremental nmf with volume constraint is derived and utilized for solving online bss the volume constraint to the mixing matrix enhances the identifiability of the sources while the incremental learning mode reduces the computational cost the proposed method takes advantage of the natural gradient based multiplication updating rule and it performs especially well in the recovery of dependent sources simulations in bss for dual energy x ray images online encrypted speech signals and high correlative face images show the validity of the proposed method

perfect and subperfect regulation in linear multivariable control systems this paper is concerned with the synthesis of a high gain feedback control called perfect regulation p r for linear multivariable systems with external signals the existence condition for p r is derived in which the minimum phase property plays an essential role the use of an observer is discussed for achieving p r under restricted state observation some asymptotically ideal feedback properties are demonstrated such as the complete desensitization the complete servo performance with decoupling and the complete disturbance rejection which formulate the loop tightness for multivariable systems a design method for applying p r to non minimum phase systems called subperfect regulation s p r is proposed based on the factorization of the plant transfer function matrix into the minimum phase part and the totally non minimum phase part it is a multivariable extension of a well known design technique for scalar systems to overcome the difficulty of phase non minimality computational procedures for p r and s p r are discussed an illustrative example is shown

online nonnegative matrix factorization with robust stochastic approximation nonnegative matrix factorization nmf has become a popular dimension reduction method and has been widely applied to image processing and pattern recognition problems however conventional nmf learning methods require the entire dataset to reside in the memory and thus cannot be applied to large scale or streaming datasets in this paper we propose an efficient online rsa nmf algorithm or nmf that learns nmf in an incremental fashion and thus solves this problem in particular or nmf receives one sample or a chunk of samples per step and updates the bases via robust stochastic approximation benefitting from the smartly chosen learning rate and averaging technique or nmf converges at the rate of in each update of the bases furthermore we prove that or nmf almost surely converges to a local optimal solution by using the quasi martingale by using a buffering strategy we keep both the time and space complexities of one step of the or nmf constant and make or nmf suitable for large scale or streaming datasets preliminary experimental results on real world datasets show that or nmf outperforms the existing online nmf onmf algorithms in terms of efficiency experimental results of face recognition and image annotation on public datasets confirm the effectiveness of or nmf compared with the existing onmf algorithms

perfect control of the robust servomechanism problem the problem of finding a robust servomechanism controller which achieves exact asymptotic tracking regulation for a modeled class of reference and disturbance signals arbitrarily good approximate error regulation agaer for reference disturbance signals which lie outside of this design class and arbitrarily good transient response is considered it is shown that a necessary condition that must be satisfied to accomplish this requirement is that the initial conditions of the servocompensator should be zero an explicit algorithm for designing such controllers is then given a simple controller called the high gain servomechanism controller is also presented which has the property that it gives agaer and arbitrarily good transient response when the robust exact asymptotic tracking regulation requirement is relaxed this controller shows that dynamics in a controller is not essential for achieving good error regulation in the presence of unknown unmeasurable disturbances it is also shown that using the controllers proposed exact error regulation occurs for a larger class of design signals called the extended disturbance reference input signals even if the conditions for perfect control are not satisfied some examples are included to illustrate the type of results obtained

envelopes and partial least squares regression summary we build connections between envelopes a recently proposed context for efficient estimation in multivariate statistics and multivariate partial least squares pls regression in particular we establish an envelope as the nucleus of both univariate and multivariate pls which opens the door to pursuing the same goals as pls but using different envelope estimators it is argued that a likelihood based envelope estimator is less sensitive to the number of pls components that are selected and that it outperforms pls in prediction and estimation

blind separation of mutually correlated sources using precoders this paper studies the problem of blind source separation bss from instantaneous mixtures with the assumption that the source signals are mutually correlated we propose a novel approach to bss by using precoders in transmitters we show that if the precoders are properly designed some cross correlation coefficients of the coded signals can be forced to be zero at certain time lags then the unique correlation properties of the coded signals can be exploited in receiver to achieve source separation based on the proposed precoders a subspace based algorithm is derived for the blind separation of mutually correlated sources the effectiveness of the algorithm is illustrated by simulation examples

large cone nonnegative matrix factorization nonnegative matrix factorization nmf has been greatly popularized by its parts based interpretation and the effective multiplicative updating rule for searching local solutions in this paper we study the problem of how to obtain an attractive local solution for nmf which not only fits the given training data well but also generalizes well on the unseen test data based on the geometric interpretation of nmf we introduce two large cone penalties for nmf and propose large cone nmf lcnmf algorithms compared with nmf lcnmf will obtain bases comprising a larger simplicial cone and therefore has three advantages 1 the empirical reconstruction error of lcnmf could mostly be smaller 2 the generalization ability of the proposed algorithm is much more powerful and 3 the obtained bases of lcnmf have a low overlapping property which enables the bases to be sparse and makes the proposed algorithms very robust experiments on synthetic and real world data sets confirm the efficiency of lcnmf

on performance limitation in tracking a sinusoid this note studies the performance limitation of a feedback system with a given linear time invariant lti plant in tracking a sinusoidal signal it continues and goes beyond some recent studies in the same topic in which it is assumed that the controller can access all the past and future values of the reference signal in this note we consider the more realistic and more difficult situation where the controller only accesses the current and past values of the reference an explicit formula of the best attainable performance is obtained for a single input single output siso system which depends on the nonminimum phase zeros of the plant and the frequency of the reference sinusoid compared to the previously studied case when the future of the reference is available this formula clearly shows the extra effort one has to pay due to the lack of the reference information a partial result for a multiple input multiple output mimo system is also given

weakly and strongly non minimum phase systems properties and limitations abstractin this paper a new classification of non minimum phase systems based on stabilisability of the internal dynamics is introduced according to this classification the non minimum phase systems are categorised as follows 1 weakly non minimum systems for which a finite value of minimum norm of the regulation error can be obtained and 2 strongly non minimum phase systems for which this minimum norm of the regulation becomes infinite due to the presence of both transient and steady state errors while for the weakly non minimum phase systems there is no steady state regulation error it is worth mentioning that based on this new classification a level of non minimum phase ness can be further defined for any given desired trajectory furthermore some numerical simulations are carried out to show consistency with theoretical analysis

multivariate calibration with more variables than observations multivariate calibration involves the use of an estimated linear relationship between a q variate response vector y and a p dimensional explanatory vector x to estimate or predict future unknown x from further observed y in some applications for example when y represents automatic intensity measurements at q different wavelengths on n chemical standard samples the feasible sample size n may be restricted whereas the dimension q can be chosen quite large in this article the singular cases appearing when n q the traditional solutions to the estimation and prediction problems that is the generalized least squares estimator and the estimated best linear predictor are both unique whereas for smaller n q n 1 dimensional hyperplanes of solutions are obtained the same in both problems the properties of the predictor are also empirically studied in an example with p 1 q 6 and varying n

variational regularized 2 d nonnegative matrix factorization a novel approach for adaptive regularization of 2 d nonnegative matrix factorization is presented the proposed matrix factorization is developed under the framework of maximum a posteriori probability and is adaptively fine tuned using the variational approach the method enables 1 a generalized criterion for variable sparseness to be imposed onto the solution and 2 prior information to be explicitly incorporated into the basis features the method is computationally efficient and has been demonstrated on two applications that is extracting features from image and separating single channel source mixture in addition it is shown that the basis features of an information bearing matrix can be extracted more efficiently using the proposed regularized priors experimental tests have been rigorously conducted to verify the efficacy of the proposed method

maximum likelihood regression on relevant components summary most regression methods for multicollinear data are only based on assumptions about the conditional distribution of the dependent variable given the explanatory variables here we propose a new prediction method taking a joint multinormal distribution for all the variables as the starting point a hypothesis is formulated stating that only a fixed small number of components constructed from the explanatory variables is relevant for prediction and maximum likelihood type estimates of the parameters in the model are developed under this hypothesis

solvability conditions and solutions to perfect regulation problem under measurement output feedback the problem of perfect regulation is to design a family of control laws for a given plant such that the resulting overall closed loop system is internally stable and its controlled output can be reduced to zero arbitrarily fast from any initial condition such a problem was heavily studied by many researchers in the 1970s and early 1980s however to the best of our knowledge all of the earlier results deal only with the problem under full state feedback in this paper we solve the long standing problem of perfect regulation via measurement output feedback for general linear time invariant multivariable systems in particular we derive necessary and sufficient conditions under which the problem of perfect regulation via measurement output feedback is solvable for general systems and under these conditions construct two families of feedback laws one of full order and the other reduced order that solve the problem

adaptive method for nonsmooth nonnegative matrix factorization nonnegative matrix factorization nmf is an emerging tool for meaningful low rank matrix representation in nmf explicit constraints are usually required such that nmf generates desired products or factorizations especially when the products have significant sparseness features it is known that the ability of nmf in learning sparse representation can be improved by embedding a smoothness factor between the products motivated by this result we propose an adaptive nonsmooth nmf ans nmf method in this paper in our method the embedded factor is obtained by using a data related approach so it matches well with the underlying products implying a superior faithfulness of the representations besides due to the usage of an adaptive selection scheme to this factor the sparseness of the products can be separately constrained leading to wider applicability and interpretability furthermore since the adaptive selection scheme is processed through solving a series of typical linear programming problems it can be easily implemented simulations using computer generated data and real world data show the advantages of the proposed ans nmf method over the state of the art methods

h2 optimal rejection with preview in the continuous time domain the synthesis of a feedforward unit for h 2 optimal rejection of previewed signals in continuous time invariant linear systems is considered the h 2 optimal compensator herein devised consists of a finite impulse response system working in connection with a standard dynamic unit the design strategy is based on a simple interpretation of the h 2 optimal rejection problem with preview as a compound optimal control problem i e as a problem consisting of a finite horizon lq control problem with constrained final state an infinite horizon lq control problem and the problem of minimizing the global cost functional an explicit description of the compensator is derived by exploiting some results on the parameterization of the solutions of the associated autonomous hamiltonian system

performance limitations in the robust servomechanism problem for discrete time lti systems fundamental limitations for error tracking regulation are obtained for the robust servomechanism problem rsp for a sampled system in studying this problem the cheap control problem for a multi input multi output discrete time system is considered and explicit expressions are obtained for the limiting steady state solution of its associated algebraic riccati equation are as the weight on the control energy tends to zero application of these results is then made to obtain explicit expressions for the limiting performance costs associated with error tracking regulation in the rsp these limitations can be characterized by the system order the dimension of the outputs the number of the system s transmission zeros and the location of the system s nonminimum phase transmission zeros

on the problem of robust and perfect tracking for linear systems with external disturbances we consider in this paper the robust and perfect tracking rpt problem for multivariable linear systems with external disturbances the problem is to design a proper controller such that the resulting overall closed loop system is asymptotically stable and the controlled output almost perfectly tracks a given reference signal with an arbitrarily fast settling time in the face of external disturbances and initial conditions the contribution of this paper is two fold 1 we derive a set of necessary and sufficient conditions under which the rpt problem is solvable and 2 under these solvability conditions we develop algorithms for constructing state and output feedback laws explicitly parameterized in epsilon that solve the rpt problem in our construction of feedback laws we propose a controller structure which enables us to design a tracking controller without introducing additional integrators regardless of what type the system is

underdetermined blind source separation based on subspace representation this paper considers the problem of blindly separating sub and super gaussian sources from underdetermined mixtures the underlying sources are assumed to be composed of two orthogonal components one lying in the rowspace and the other in the nullspace of a mixing matrix the mapping from the rowspace component to the mixtures by the mixing matrix is invertible using the pseudo inverse of the mixing matrix the mapping from the nullspace component to zero by the mixing matrix is noninvertible and there are infinitely many solutions to the nullspace component the latent nullspace component which is of lower complexity than the underlying sources is estimated based on a mean square error mse criterion this leads to a source estimator that is optimal in the mse sense in order to characterize and model sub and super gaussian source distributions the parametric generalized gaussian distribution is used the distribution parameters are estimated based on the expectation maximization em algorithm when the mixing matrix is unavailable it must be estimated and a novel algorithm based on a single source detection algorithm which detects time frequency regions of single source occupancy is proposed in our simulations the proposed algorithm compared to other conventional algorithms estimated the mixing matrix with higher accuracy and separated various sources with higher signal to interference ratio

partial inverse regression in regression with a vector of quantitative predictors sufficient dimension reduction methods can effectively reduce the predictor dimension while preserving full regression information and assuming no parametric model however all current reduction methods require the sample size n to be greater than the number of predictors p it is well known that partial least squares can deal with problems with n p we first establish a link between partial least squares and sufficient dimension reduction motivated by this link we then propose a new dimension reduction method entitled partial inverse regression we show that its sample estimator is consistent and that its performance is similar to or superior to partial least squares when n p especially when the regression model is nonlinear or heteroscedastic an example involving the spectroscopy analysis of biscuit dough is also given

feedback minimax sensitivity and optimal robustness in this paper we look for feedbacks that minimize the sensitivity function of a linear single variable feedback system represented by its frequency responses sensitivity to disturbances and robustness under plant perturbations are measured in a weighted h infty norm in an earlier paper zames proposed an approach to feedback design involving the measurement of sensitivity by multiplicative seminorms which have certain advantages over the widely used quadratic norm in problems where there is plant uncertainty or where signal power spectra are not fixed but belong to sets the problem was studied in a general setting and some h infty examples were solved here a detailed study of the single variable case is undertaken the results are extended to unstable plants and explicit formulas for the general situation of a finite number of right half plane rhp plant zeros or poles are provided the q or approximate inverse parametrization of feedbacks that maintain closed loop stability is extended to the ease of unstable plants the h infty and wiener hopf approaches are compared

linear control theory with an h 0e optimality criterion this expository paper sets out the principal results in bf h infty control theory in the context of continuous time linear systems the focus is on the mathematical theory rather than computational methods

additive risk models for survival data with high dimensional covariates as a useful alternative to cox s proportional hazard model the additive risk model assumes that the hazard function is the sum of the baseline hazard function and the regression function of covariates this article is concerned with estimation and prediction for the additive risk models with right censored survival data especially when the dimension of the covariates is comparable to or larger than the sample size principal component regression is proposed to give unique and numerically stable estimators asymptotic properties of the proposed estimators component selection based on the weighted bootstrap and model evaluation techniques are discussed this approach is illustrated with analysis of the primary biliary cirrhosis clinical data and the diffuse large b cell lymphoma genomic data it is shown that this methodology is numerically stable and effective in dimension reduction while still being able to provide satisfactory prediction and classification results

time frequency approach to underdetermined blind source separation this paper presents a new time frequency tf underdetermined blind source separation approach based on wigner ville distribution wvd and khatri rao product to separate n non stationary sources from m m n mixtures first an improved method is proposed for estimating the mixing matrix where the negative value of the auto wvd of the sources is fully considered then after extracting all the auto term tf points the auto wvd value of the sources at every auto term tf point can be found out exactly with the proposed approach no matter how many active sources there are as long as n 2m 1 further discussion about the extraction of auto term tf points is made and finally the numerical simulation results are presented to show the superiority of the proposed algorithm by comparing it with the existing ones

path following for nonminimum phase systems removes performance limitations we highlight an essential difference between path following and reference tracking for nonminimum phase systems it is well known that in the reference tracking for nonminimum phase systems there exists a fundamental performance limitation in terms of a lower bound on the l sub 2 norm of the tracking error even when the control effort is free we show that this is not the case for the less stringent path following problem where the control objective is to force the output to follow a geometric path without a timing law assigned to it furthermore the same is true even when an additional desired speed assignment is imposed

tensor envelope partial least squares regression abstractpartial least squares pls is a prominent solution for dimension reduction and high dimensional regressions recent prevalence of multidimensional tensor data has led to several tensor versions of the pls algorithms however none offers a population model and interpretation and statistical properties of the associated parameters remain intractable in this article we first propose a new tensor partial least squares algorithm then establish the corresponding population interpretation this population investigation allows us to gain new insight on how the pls achieves effective dimension reduction to build connection with the notion of sufficient dimension reduction and to obtain the asymptotic consistency of the pls estimator we compare our method both analytically and numerically with some alternative solutions we also illustrate the efficacy of the new method on simulations and two neuroimaging data analyses supplementary materials for this article are available online

a convergent algorithm for orthogonal nonnegative matrix factorization this paper proposes a convergent algorithm for nonnegative matrix factorization nmf with orthogonality constraint on the factors we design the algorithm based on the additive update rule algorithm for the standard nmf proposed by lee and seung and derive the convergent version by generalizing the convergence proof of the algorithm developed by lin further we use the proposed algorithms to improve clustering capability of the standard nmf using the reuter document corpus a standard dataset in clustering research

convolution profiles for right inversion of multivariable non minimum phase discrete time systems the problem of the non causal inversion of linear multivariable discrete time systems is analyzed in the geometric approach framework and is solved through the computation of convolution profiles which guarantee perfect tracking under the assumption of infinite length preaction and postaction time intervals it is shown how the shape of the convolution profiles is related to both the relative degree and the invariant zeros of the plant a computational setting for the convolution profiles is derived by means of the standard geometric approach tools feasibility constraints are also taken into account a possible implementation scheme based on a finite impulse response system acting on a stabilized control loop is provided

a fast algorithm for nonnegative matrix factorization and its convergence nonnegative matrix factorization nmf has recently become a very popular unsupervised learning method because of its representational properties of factors and simple multiplicative update algorithms for solving the nmf however for the common nmf approach of minimizing the euclidean distance between approximate and true values the convergence of multiplicative update algorithms has not been well resolved this paper first discusses the convergence of existing multiplicative update algorithms we then propose a new multiplicative update algorithm for minimizing the euclidean distance between approximate and true values based on the optimization principle and the auxiliary function method we prove that our new algorithm not only converges to a stationary point but also does faster than existing ones to verify our theoretical results the experiments on three data sets have been conducted by comparing our proposed algorithm with other existing methods

natural gradient works efficiently in learning when a parameter space has a certain underlying structure the ordinary gradient of a function does not represent its steepest direction but the natural gradient does information geometry is used for calculating the natural gradients in the parameter space of perceptrons the space of matrices for blind source separation and the space of linear dynamical systems for blind source deconvolution the dynamical behavior of natural gradient online learning is analyzed and is proved to be fisher efficient implying that it has asymptotically the same performance as the optimal batch estimation of parameters this suggests that the plateau phenomenon which appears in the backpropagation learning algorithm of multilayer perceptrons might disappear or might not be so serious when the natural gradient is used an adaptive method of updating the learning rate is proposed and analyzed

downdating the moore penrose generalized inverse for cross validation of centred least squares prediction summary given ap x p symmetric matrix b and ap vector c outside the range of b a simple expression is formulated for the action of the moore penrose generalized inverse b on a vector b in the range of b in terms of b c and b cc a special case of this formula gives a downdating formula for the vector of least squares regression coefficients based on the moore penrose inverse of the centred sum of squares and products matrix the formula is analogous to that given by a albert for uncentred least squares fitting an explicit downdating formula for the moore penrose inverse b itself is derived and then more general downdating expressions for the inverse and the vector b b when the p vector c is replaced by a matrix array c of such vectors outside the range of b

fast and robust fixed point algorithms for independent component analysis independent component analysis ica is a statistical method for transforming an observed multidimensional random vector into components that are statistically as independent from each other as possible we use a combination of two different approaches for linear ica comon s information theoretic approach and the projection pursuit approach using maximum entropy approximations of differential entropy we introduce a family of new contrast functions for ica these contrast functions enable both the estimation of the whole decomposition by minimizing mutual information and estimation of individual independent components as projection pursuit directions the statistical properties of the estimators based on such contrast functions are analyzed under the assumption of the linear mixture model and it is shown how to choose contrast functions that are robust and or of minimum variance finally we introduce simple fixed point algorithms for practical optimization of the contrast functions

estimating the error rate of a prediction rule improvement on cross validation abstract we construct a prediction rule on the basis of some data and then wish to estimate the error rate of this rule in classifying future observations cross validation provides a nearly unbiased estimate using only the original data cross validation turns out to be related closely to the bootstrap estimate of the error rate this article has two purposes to understand better the theoretical basis of the prediction problem and to investigate some related estimators which seem to offer considerably improved estimation in small samples

comparison of prediction methods when only a few components are relevant abstract we consider prediction in a multiple regression model where we also look on the explanatory variables as random if the number of explanatory variables is large then the common least squares multiple regression solution may not be the best one we give a methodology for comparing certain alternative prediction methods by asymptotic calculations and perform such a comparisons for four specific methods the results indicate that none of these methods dominates the others and that the difference between the methods typically but not always is small when the number of observations is large in particular principal component regression does well when the eigenvalues corresponding to components not correlated with the dependent variables i e the irrelevant eigenvalues are extremely small or extremely large partial least squares regression does well for intermediate irrelevant eigenvalues a maximum likelihood type method dominates the others asymptotically at least in the case of one relevan

analysis of sparse representation and blind source separation in this letter we analyze a two stage cluster then l1 optimization approach for sparse representation of a data matrix which is also a promising approach for blind source separation bss in which fewer sensors than sources are present first sparse representation factorization of a data matrix is discussed for a given overcomplete basis matrix the corresponding sparse solution coefficient matrix with minimum l1 norm is unique with probability one which can be obtained using a standard linear programming algorithm the equivalence of the l1 norm solution and the l0 norm solution is also analyzed according to a probabilistic framework if the obtained l1 norm solution is sufficiently sparse then it is equal to the l0 norm solution with a high probability furthermore the l1 norm solution is robust to noise but the l0 norm solution is not showing that the l1 norm is a good sparsity measure these results can be used as a recoverability analysis of bss as discussed the basis matrix in this article is estimated using a clustering algorithm followed by normalization in which the matrix columns are the cluster centers of normalized data column vectors zibulevsky pearlmutter boll and kisilev 2000 used this kind of two stage approach in underdetermined bss our recoverability analysis shows that this approach can deal with the situation in which the sources are overlapped to some degree in the analyzed domain and with the case in which the source number is unknown it is also robust to additive noise and estimation error in the mixing matrix finally four simulation examples and an eeg data analysis example are presented to illustrate the algorithm s utility and demonstrate its performance

on the convergence of multiplicative update algorithms for nonnegative matrix factorization nonnegative matrix factorization nmf is useful to find basis information of nonnegative data currently multiplicative updates are a simple and popular way to find the factorization however for the common nmf approach of minimizing the euclidean distance between approximate and true values no proof has shown that multiplicative updates converge to a stationary point of the nmf optimization problem stationarity is important as it is a necessary condition of a local minimum this paper discusses the difficulty of proving the convergence we propose slight modifications of existing updates and prove their convergence techniques invented in this paper may be applied to prove the convergence for other bound constrained optimization problems

minimum volume constrained nonnegative matrix factorization enhanced ability of learning parts nonnegative matrix factorization nmf with minimum volume constraint mvc is exploited in this paper our results show that mvc can actually improve the sparseness of the results of nmf this sparseness is l0 norm oriented and can give desirable results even in very weak sparseness situations thereby leading to the significantly enhanced ability of learning parts of nmf the close relation between nmf sparse nmf and the mvc nmf is discussed first then two algorithms are proposed to solve the mvc nmf model one is called quadratic programming mvc nmf qp mvc nmf which is based on quadratic programming and the other is called negative glow mvc nmf ng mvc nmf because it uses multiplicative updates incorporating natural gradient ingeniously the qp mvc nmf algorithm is quite efficient for small scale problems and the ng mvc nmf algorithm is more suitable for large scale problems simulations show the efficiency and validity of the proposed methods in applications of blind source separation and human face images analysis

when the data are functions a datum is often a continuous functionx t of a variable such as time observed over some interval one or more such functions are observed for each subject or unit of observation the extension of classical data analytic techniques designed forp variate observations to such data is discussed the essential step is the expression of the classical problem in the language of functional analysis after which the extension to functions is a straightforward matter a schematic device called the duality diagram is a very useful tool for describing an analysis and for suggesting new possibilities least squares approximation descriptive statistics principal components analysis and canonical correlation analysis are discussed within this broader framework

gain margin and time lag tolerance constraints applied to the stabilization problem and robust servomechanism problem definitions of gain margin tolerance and time lag tolerance for a multivariable linear time invariant system are given which correspond to a generalization of the notions of gain and phase margin of classical control system theory existence conditions are then given to solve the stabilization problem and the robust servomechanism problem with gain margin and time lag tolerance constraints in particular the following is shown for a system described in terms of a state state model 1 generically there exists a controller to stabilize an unstable m input r output system so that the resultant closed loop system possesses a nonzero gain margin if and only if min r m 1 2 for stable plants there always exists a controller to solve the robust servomechanism problem so that it has an arbitrary specified gain margin time lag tolerance constraint if and only if there exists a solution to the robust servomechanism problem for the plant a controller synthesis method is then given which solves the robust servomechanism problem so that it has a specified gain margin time lag tolerance various examples of the proposed algorithm are given to illustrate the type of results that may be obtained

multivariate calibration when the error covariance matrix is structured this article concerns multivariate calibration in linear models the error covariance matrix is assumed to have linear factor structure a new approach to calibration incorporating distributional properties of the observations of the calibration set is proposed the proposed predictor is based on estimating the parameters in the best linear predictor in the assumed model the predictor is tested on two data sets meat data and fish data

multivariable feedback sensitivity and decentralized control in this paper the problem of sensitivity reduction by feedback is studied and related to a problem of decentralized control a plant will be represented by an n times n matrix of frequency responses which may be unstable or irrational the object will be to find conditions on p s under which a diagonal feedback f s can make the sensitivity parallel i p s f s 1 parallel arbitrarily small over some specified frequency interval j omega 0 j omega 0 without violating a global sensitivity bound parallel i p s f s 1 parallel leq m m geq some const 1 for re s geq 0 it will be shown that such a diagonal feedback of the high gain type can be constructed whenever p 1 s is analytic in re s geq 0 p s satisfies an attenuation condition near s infty and p s approaches diagonal dominance at high frequencies it will also be shown that these conditions on the plant can be interpreted as conditions for the existence of a decentralized wide band control scheme

equivariant adaptive source separation source separation consists of recovering a set of independent signals when only mixtures with unknown coefficients are observed this paper introduces a class of adaptive algorithms for source separation that implements an adaptive version of equivariant estimation and is henceforth called equivariant adaptive separation via independence easi the easi algorithms are based on the idea of serial updating this specific form of matrix updates systematically yields algorithms with a simple structure for both real and complex mixtures most importantly the performance of an easi algorithm does not depend on the mixing matrix in particular convergence rates stability conditions and interference rejection levels depend only on the normalized distributions of the source signals closed form expressions of these quantities are given via an asymptotic performance analysis the theme of equivariance is stressed throughout the paper the source separation problem has an underlying multiplicative structure the parameter space forms a matrix multiplicative group we explore the favorable consequences of this fact on implementation performance and optimization of easi algorithms

projection pursuit based method for blind separation of nonnegative sources this paper presents a projection pursuit pp based method for blind separation of nonnegative sources first the available observation matrix is mapped to construct a new mixing model in which the inaccessible source matrix is normalized to be column sum to 1 then the pp method is proposed to solve this new model where the mixing matrix is estimated column by column through tracing the projections to the mapped observations in specified directions which leads to the recovery of the sources the proposed method is much faster than chan s method which has similar assumptions to ours due to the usage of optimal projection it is also more advantageous in separating cross correlated sources than the independence and uncorrelation based methods as it does not employ any statistical information of the sources furthermore the new method does not require the mixing matrix to be nonnegative simulation results demonstrate the superior performance of our method

efficient online subspace learning with an indefinite kernel for visual tracking and recognition we propose an exact framework for online learning with a family of indefinite not positive kernels as we study the case of nonpositive kernels we first show how to extend kernel principal component analysis kpca from a reproducing kernel hilbert space to krein space we then formulate an incremental kpca in krein space that does not require the calculation of preimages and therefore is both efficient and exact our approach has been motivated by the application of visual tracking for which we wish to employ a robust gradient based kernel we use the proposed nonlinear appearance model learned online via kpca in krein space for visual tracking in many popular and difficult tracking scenarios we also show applications of our kernel framework for the problem of face recognition

continuum regression cross validated sequentially constructed prediction embracing ordinary least squares partial least squares and principal components regression read before the royal statistical society at a meeting organized by the research section on wednesday october 25th 1989 professor d v hinkley in the chair summary the paper addresses the evergreen problem of construction of regressors for use in least squares multiple regression in the context of a general sequential procedure for doing this it is shown that with a particular objective criterion for the construction the procedures of ordinary least squares and principal components regression occupy the opposite ends of a continuous spectrum with partial least squares lying in between there are two adjustable parameters controlling the procedure alpha in the continuum 0 11 and omega the number of regressors finally accepted these control parameters are chosen by crossvalidation the method is illustrated by a range of examples of its application

convergence analysis of the focuss algorithm focal underdetermined system solver focuss is a powerful and easy to implement tool for basis selection and inverse problems one of the fundamental problems regarding this method is its convergence which remains unsolved until now we investigate the convergence of the focuss algorithm in this paper we first give a rigorous derivation for the focuss algorithm by exploiting the auxiliary function following this we further prove its convergence by stability analysis

on optimal output regulation for linear systems the classical output regulation problem formulation for linear systems has a number of shortcomings among them a primary one is that it does not take into account the transient response although this problem has been studied since the 1970s a complete picture has not emerged yet we formulate and study here a number of output regulation problems which incorporate the notion of transient performance into them by defining a performance index that involves transient response we define optimal and suboptimal control problems which are constrained by the steady state output regulation requirement such constrained optimal and suboptimal control problems for a given system are studied by transforming them to unconstrained optimal and suboptimal control problems for certain auxiliary systems several issues such as obtaining expressions for the infimal performance measure solvability conditions regulator construction are studied in detail both state and measurement feedback controllers are considered

fundamental limit of discrete time systems in tracking multi tone sinusoidal signals this paper studies the tracking performance of linear time invariant multi variable discrete time systems the specific problem under consideration is to track a multi tone sinusoidal reference signal consisting of linear combinations of a step and several sinusoidal signals whereas the tracking performance is measured by the energy of the error response between the output of the plant and the reference signal our purpose is to find the fundamental limit for the best attainable performance under any control structures and parameters and we seek to determine this limit analytically in terms of the given plant and reference characteristics both the full information and partial information tracking schemes are formulated and investigated to address these goals which are concerned with whether or not the reference information is fully available for tracking analytical expressions are developed in full generality under full information tracking and for a more specialized case under partial information scheme in addition an optimal cheap control design is constructed to show that the performance limit can be attained asymptotically in the full information case the results show that in general plant nonminimum phase zeros and reference modes can interact to fundamentally constrain a system s tracking ability they also show that absence of full reference information can degrade the tracking performance thus demonstrating an intrinsic trade off between the tracking objective and the availability of the reference information

incremental subspace learning via non negative matrix factorization in this paper we introduce an incremental non negative matrix factorization inmf scheme in order to overcome the difficulties that conventional nmf has in online processing of large data sets the proposed scheme enables incrementally updating its factors by reflecting the influence of each observation on the factorization appropriately this is achieved via a weighted cost function which also allows controlling the memorylessness of the factorization unlike conventional nmf with its incremental nature and weighted cost function the inmf scheme successfully utilizes adaptability to dynamic data content changes with a lower computational complexity test results reported for two video applications namely background modeling in video surveillance and clustering demonstrate that inmf is capable of online representing data content while reducing dimension significantly

nonnegative least correlated component analysis for separation of dependent sources by volume maximization although significant efforts have been made in developing nonnegative blind source separation techniques accurate separation of positive yet dependent sources remains a challenging task in this paper a joint correlation function of multiple signals is proposed to reveal and confirm that the observations after nonnegative mixing would have higher joint correlation than the original unknown sources accordingly a new nonnegative least correlated component analysis nlca method is proposed to design the unmixing matrix by minimizing the joint correlation function among the estimated nonnegative sources in addition to a closed form solution for unmixing two mixtures of two sources the general algorithm of nlca for the multisource case is developed based on an iterative volume maximization ivm principle and linear programming the source identifiability and required conditions are discussed and proven the proposed nlca algorithm denoted by nlca ivm is evaluated with both simulation data and real biomedical data to demonstrate its superior performance over several existing benchmark methods

candid covariance free incremental principal component analysis appearance based image analysis techniques require fast computation of principal components of high dimensional image vectors we introduce a fast incremental principal component analysis ipca algorithm called candid covariance free ipca ccipca used to compute the principal components of a sequence of samples incrementally without estimating the covariance matrix so covariance free the new method is motivated by the concept of statistical efficiency the estimate has the smallest variance given the observed data to do this it keeps the scale of observations and computes the mean of observations incrementally which is an efficient estimate for some well known distributions e g gaussian although the highest possible efficiency is not guaranteed in our case because of unknown sample distribution the method is for real time applications and thus it does not allow iterations it converges very fast for high dimensional image vectors some links between ipca and the development of the cerebral cortex are also discussed

partial least squares estimator for single index models the partial least squares pls approach first constructs new explanatory variables known as factors or components which are linear combinations of available predictor variables a small subset of these factors is then chosen and retained for prediction we study the performance of pls in estimating single index models especially when the predictor variables exhibit high collinearity we show that pls estimates are consistent up to a constant of proportionality we present three simulation studies that compare the performance of pls in estimating single index models with that of sliced inverse regression sir in the first two studies we find that pls performs better than sir when collinearity exists in the third study we learn that pls performs well even when there are multiple dependent variables the link function is non linear and the shape of the functional form is not known

bootstrap and cross validation estimates of the prediction error for linear regression models different estimates of the mean squared error of prediction for linear regression models are derived by the bootstrap and cross validation approaches a comparison is made under normal error distributions especially by the biases and the mean square errors the results indicate that the bias corrected bootstrap estimator is best unbiased and should be the first choice while its simulated variant has approximately the same behaviour on the other hand if only a comparison between uncorrected estimators is made with implications for nonlinear regression models in mind then other variants of bootstrap estimates are preferable for a large or a small dimension of the model parameter for a small dimension the cross validation estimate and sometimes grouped variants of it seem also to be acceptable if the model error is known to be small

linear dimensionality reduction survey insights and generalizations linear dimensionality reduction methods are a cornerstone of analyzing high dimensional data due to their simple geometric interpretations and typically attractive computational properties these methods capture many data features of interest such as covariance dynamical structure correlation between data sets input output relationships and margin between data classes methods have been developed with a variety of names and motivations in many fields and perhaps as a result the connections between all these methods have not been highlighted here we survey methods from this disparate literature as optimization programs over matrix manifolds we discuss principal component analysis factor analysis linear multidimensional scaling fisher s linear discriminant analysis canonical correlations analysis maximum autocorrelation factors slow feature analysis sufficient dimensionality reduction undercomplete independent component analysis linear regression distance metric learning and more this optimization framework gives insight to some rarely discussed shortcomings of well known methods such as the suboptimality of certain eigenvector solutions modern techniques for optimization over matrix manifolds enable a generic linear dimensionality reduction solver which accepts as input data and an objective to be optimized and returns as output an optimal low dimensional projection of the data this simple optimization framework further allows straightforward generalizations and novel variants of classical methods which we demonstrate here by creating an orthogonal projection canonical correlations analysis more broadly this survey and generic solver suggest that linear dimensionality reduction can move toward becoming a blackbox objective agnostic numerical technology

universal regulators for optimal tracking in discrete time systems affected by harmonic disturbances considers the problem of controlling a discrete time linear system by output feedback so as to have a second output z sub t track an observed reference signal r sub t the authors turn to an approach to the tracking problem which allows for damping of other system and control variables and this is our main result the measure of performance is given by a natural quadratic cost function the object is to design an optimal regulator which is universal in the sense that it does not depend on the unknown amplitudes and phases of r sub t and w sub t and is optimal for all choices of r sub t and w sub t the authors prove that an optimal universal regulator exists in a wide class of stabilizing and possibly nonlinear regulators under natural technical conditions and that this regulator is in fact linear provided that the second dimensionality condition above is satisfied on the other hand if it is not satisfied the existence of an optimal universal regulator is not a generic property so as a rule no optimal universal regulator exists

local coordinate concept factorization for image representation learning sparse representation of high dimensional data is a state of the art method for modeling data matrix factorization based techniques such as nonnegative matrix factorization and concept factorization cf have shown great advantages in this area especially useful for image representation both of them are linear learning problems and lead to a sparse representation of the images however the sparsity obtained by these methods does not always satisfy locality conditions for example the learned new basis vectors may be relatively far away from the original data thus we may not be able to achieve the optimal performance when using the new representation for other learning tasks such as classification and clustering in this paper we introduce a locality constraint into the traditional cf by requiring the concepts basis vectors to be as close to the original data points as possible each datum can be represented by a linear combination of only a few basis concepts thus our method is able to achieve sparsity and locality simultaneously we analyze the complexity of our novel algorithm and demonstrate the effectiveness in comparison with the state of the art approaches through a set of evaluations based on real world applications

application of the robust servomechanism controller to systems with periodic tracking disturbance signals the following problem is considered given a system which has tracking disturbance signals arising from a class of periodic signals with period t say assume that such a periodic signal has p dominant harmonic components contained in 1 l 0 1 v then it is desired to find a finite dimensional linear controller which gives exact robust asymptotic regulation for the case of sinusoidal signals with frequencies 1 l 0 1 v subject to certain controller gain magnitude constraints and or gain margin tolerance constraints which may be imposed on the problem this is called the approximate robust servomechanism problem for periodic signals of the class 1 l 0 1 v it is to be noted that such a problem cannot in general be solved by using high gain control synthesis methods existence conditions and a controller design method are given to solve the problem a number of examples is given to illustrate the type of results that may be achieved