application of the karhunen loeve procedure for the characterization of human faces the use of natural symmetries mirror images in a well defined family of patterns human faces is discussed within the framework of the karhunen loeve expansion this results in an extension of the data and imposes even and odd symmetry on the eigenfunctions of the covariance matrix without increasing the complexity of the calculation the resulting approximation of faces projected from outside of the data set onto this optimal basis is improved on average

robust methods for model order estimation model order estimation is a subject in time series analysis that deals with fitting a parametric model to a vector of observations this paper focuses on several model order estimators known in the literature and examines their performance under small deviations of the probability distribution of the noise with respect to a nominal distribution assumed in the model it is demonstrated that the standard estimators suffer from high sensitivity to deviations from the nominal distribution and a drastic performance degradation is experienced to overcome this problem robust estimators that are insensitive to small deviations from the nominal distribution are developed these estimators are based on a composition between model order estimation methods and robust statistical inference techniques known in the literature in addition a new estimator based on a locally best test for weak signals is presented both in nonrobust and robust versions the proposed robust model order estimators are developed on a heuristic basis and there is no claim of optimality but experimental results indicate that they provide significant improvement over the standard estimators

algebraic approach to system identification a variety of identification procedures exists for estimating the parameters of an autoregressive moving average arma process from noise free excitation and noise contaminated response data in this paper an identification procedure is proposed for the more realistic situation in which both the excitation and response are contaminated by white noises the method is based upon the null space characterization of an associated data matrix some of the more important algebraic properties possessed by this data matrix are first established in the ideal noise free data case in particular it is found that an overordering of the arma model will not impair the identification of the underlying system in the more realistic noise contaminated data case an approximation of the data matrix s null space is affected by using an eigenvalue eigenvector decomposition by incorporating this null space approximation the deleterious effects of the noise are significantly reduced thereby giving rise to improved modeling performance this improvement is demonstrated by means of a standard example in which the proposed identification method is shown to produce a better modeling behavior than does the classical least squares method the corresponding iterative generalized least squares method and a commonly employed instrumental variable method

eigenfaces vs fisherfaces recognition using class specific linear projection we develop a face recognition algorithm which is insensitive to large variation in lighting direction and facial expression taking a pattern classification approach we consider each pixel in an image as a coordinate in a high dimensional space we take advantage of the observation that the images of a particular face under varying illumination but fixed pose lie in a 3d linear subspace of the high dimensional image space if the face is a lambertian surface without shadowing however since faces are not truly lambertian surfaces and do indeed produce self shadowing images will deviate from this linear subspace rather than explicitly modeling this deviation we linearly project the image into a subspace in a manner which discounts those regions of the face with large deviation our projection method is based on fisher s linear discriminant and produces well separated classes in a low dimensional subspace even under severe variation in lighting and facial expressions the eigenface technique another method based on linearly projecting the image space to a low dimensional subspace has similar computational requirements yet extensive experimental results demonstrate that the proposed fisherface method has error rates that are lower than those of the eigenface technique for tests on the harvard and yale face databases

the rise algorithm for recursive eigenspace decomposition a recursive method for generating the eigenvalues and eigenvectors of hermitian matrices is presented this algorithm is closely related to recursive iterative toeplitz eigenspace decomposition rite and thus may appropriately be named recursive iterative self adjoint eigenspace decomposition rise since it is formulated for the more general hermitian matrix rise may be performed at a cost of o n sup 3 multiplies per matrix order while rite requires o 4n sup 3 it is demonstrated that rise as is also true with rite is numerically stable and thus does not appreciably accumulate errors as the recursion progresses additionally a variation of rite is presented which allows the computation of the eigenvalues of successively smaller hermitian toeplitz matrices the initial presentation of rite only provided for the determination of the eigenvalues of successively larger matrices

constructing descriptive and discriminative nonlinear features rayleigh coefficients in kernel feature spaces we incorporate prior knowledge to construct nonlinear algorithms for invariant feature extraction and discrimination employing a unified framework in terms of a nonlinearized variant of the rayleigh coefficient we propose nonlinear generalizations of fisher s discriminant and oriented pca using support vector kernel functions extensive simulations show the utility of our approach

detection of signals by information theoretic criteria a new approach is presented to the problem of detecting the number of signals in a multichannel time series based on the application of the information theoretic criteria for model selection introduced by akaike aic and by schwartz and rissanen mdl unlike the conventional hypothesis testing based approach the new approach does not requite any subjective threshold settings the number of signals is obtained merely by minimizing the aic or the mdl criteria simulation results that illustrate the performance of the new method for the detection of the number of signals received by a sensor array are presented

an approach to time series analysis and arma spectral estimation this paper presents an approach to time series analysis and arma spectral estimation from only the output data corrupted by noise it is shown that the generalized not well known modified yule walker myw equations hold when the residual is some correlated noise to solve such equations a new version of the generalized least squares gls method is proposed yielding ar parameter estimates with higher accuracy this gls method can also be used to enhance the estimation accuracy of ar parameters for short and noisy data in case of the myw equation holding theoretically furthermore a simple procedure for improving ma parameter estimates is studied our approach as cadzow s singular value decomposition svd method has provided significantly higher performance spectral estimates for a low order arma model than those obtained via usual techniques which are based upon direct solution of the myw equations and which use a high order model without considering an additive noise

vector arma estimation a reliable subspace approach a parameter estimation method for finite dimensional multivariate linear stochastic systems which is guaranteed to produce valid models approximating the true underlying system in a computational time of a polynomial order in the system dimension is presented this is achieved by combining the main features of certain stochastic subspace identification techniques with sound matrix schur restabilizing procedures and multivariate covariance fitting both of which are formulated as linear matrix inequality problems all aspects of the identification method are discussed with an emphasis on the two issues mentioned above and examples of the overall performance are provided for two different systems

arma parameter estimation using a novel recursive estimation algorithm with selective updating the authors investigate an extension of a recursive estimation algorithm the so called optimal bounding ellipsoid or obe algorithm which features a discerning update strategy in particular an extension of the algorithm to arma autoregressive moving average parameter estimation is presented along with a convergence analysis the extension is similar to the extended least squares els algorithm however the convergence analysis is complicated due to the discerning update strategy which incorporates an information dependent updating factor the virtues of such an update strategy are more efficient use of the input data in terms of information processing and a modular adaptive filter structure which would facilitate the development of a parallel pipelined signal processing architecture it is shown that if the input noise is bounded and the moving average parameters satisfy a certain magnitude bound then the a posteriori prediction errors are uniformly bounded with an additional persistence of excitation condition the parameter estimates are shown to converge to a neighborhood of the true parameters and the a priori prediction errors are shown to be asymptotically bounded simulation results show that the parameter estimation error for the eobe algorithm is comparable to that for the els algorithm

a novel multiset integrated canonical correlation analysis framework and its application in feature fusion multiset canonical correlation analysis mcca is difficult to effectively express the integrated correlation among multiple feature vectors in feature fusion thus this paper firstly presents a novel multiset integrated canonical correlation analysis micca framework the micca establishes a discriminant correlation criterion function of multi group variables based on generalized correlation coefficient the criterion function can clearly depict the integrated correlation among multiple feature vectors then the paper presents a multiple feature fusion theory and algorithm using the micca method the detailed process of the algorithm is as follows firstly extract multiple feature vectors from the same patterns by using different feature extraction methods then extract multiset integrated canonical correlation features using micca finally form effective discriminant feature vectors through two given feature fusion strategies for pattern classification the multi group feature fusion method based on micca not only achieves the aim of feature fusion but also removes the redundancy between features the experiment results on cenparmi handwritten arabic numerals and uci multiple features database show that the micca method has better recognition rates and robustness than the fusion methods based on canonical correlation analysis cca and mcca

two dimensional autoregressive 2 d ar model order estimation much research has been devoted to the area of one dimensional autoregressive 1 d ar and autoregressive moving average arma model order selection the most well known solutions for this problem are the akaike information criterion aic mdl and the minimum eigenvalue mev criteria on the other hand all works in the 2 d case have focused on the problem of parameter estimation in this correspondence we extend the previous criteria to the 2 d ar model order determination the model is assumed causal stable and spatially invariant with p sub 1 spl times p sub 2 quarter plane qp support numerical examples are given to illustrate the effectiveness of each method

a genetic algorithm approach to the spectral estimation of time series with noise and missed observations this study considers the problem of estimating the autoregressive moving average arma power spectral density when measurements are corrupted by noise and by missed observations the missed observations model is based on a probabilistic structure unlike conventional cases of missed observation in parameter estimation problems the variance of noise is unavailable that is the time points of missed observations are unknown and the probability of missing data needs to be estimated in this situation spectral estimation is more difficult to solve and becomes a highly nonlinear optimization problem with many local minima in this paper we use the genetic algorithm ga method to achieve a global optimal solution with a fast convergence rate for this spectral estimation problem from the simulation results we have determined that the performance is significantly improved if the probability of data loss is considered in the spectral estimation problem

a new method of feature fusion and its application in image recognition a new method of feature extraction based on feature fusion is proposed in this paper according to the idea of canonical correlation analysis cca at first the theory framework of cca used in pattern recognition and its reasonable description are discussed the process can be explained as follows extract two groups of feature vectors with the same pattern establish the correlation criterion function between the two groups of feature vectors and extract their canonical correlation features to form effective discriminant vector for recognition then the problem of canonical projection vectors is solved when two total scatter matrixes are singular such that it fits for the case of high dimensional space and small sample size in this sense the applicable range of cca is extended at last the inherent essence of this method used in recognition is analyzed further in theory experimental results on concordia university cenparmi database of handwritten arabic numerals and yale face database show that recognition rate is far higher than that of the algorithm adopting single feature or the existing fusion algorithm

simultaneous identification and adaptive control of unknown systems over finite parameter sets the problem considered is one of simultaneously identifying an unknown system while adequately controlling it the system can be any fairly general discrete time system and the cost criterion can be either of a discounted type or of a long term average type the chief restriction being that the unknown parameter lies in a finite parameter set for a previously introduced scheme of identification and control based on biased maximum likelihood estimates it is shown that 1 every cesaro limit point of the parameter estimates is closed loop equivalent to the unknown parameter 2 for both the discounted and long term average cost criteria the adaptive control law cesaro converges to the set of optimal control laws and 3 in the case of the long term average cost criterion the actual cost incurred by the use of the adaptive controller is optimal and cannot be bettered even if one knew the value of the unknown parameter at the start

brief papers linear identification of arma processes a new method for estimating the parameters of an arma process is presented the method consists of three linear least squares estimations in the first an autoregressive model is fitted to the observation sequence yielding an estimate of the values of the driving white noise sequence linear least squares is then used to fit an arma model to the observation and estimated white noise sequences this model is used to filter the observation and estimated white noise sequences finally an arma model is fitted to the filtered sequences it is shown that the resultant estimator is p consistent the asymptotic bias tends to zero as the degree p of the autoregressive model tends to infinity and is p efficient the asymptotic efficiency approaches the theoretical maximum as p tends to infinity

discriminative learning and recognition of image set classes using canonical correlations we address the problem of comparing sets of images for object recognition where the sets may represent variations in an object s appearance due to changing camera pose and lighting conditions canonical correlations also known as principal or canonical angles which can be thought of as the angles between two d dimensional subspaces have recently attracted attention for image set matching canonical correlations offer many benefits in accuracy efficiency and robustness compared to the two main classical methods parametric distribution based and nonparametric sample based matching of sets here this is first demonstrated experimentally for reasonably sized data sets using existing methods exploiting canonical correlations motivated by their proven effectiveness a novel discriminative learning method over sets is proposed for set classification specifically inspired by classical linear discriminant analysis lda we develop a linear discriminant function that maximizes the canonical correlations of within class sets and minimizes the canonical correlations of between class sets image sets transformed by the discriminant function are then compared by the canonical correlations classical orthogonal subspace method osm is also investigated for the similar purpose and compared with the proposed method the proposed method is evaluated on various object recognition problems using face image sets with arbitrary motion captured under different illuminations and image sets of 500 general objects taken at different views the method is also applied to object category recognition using eth 80 database the proposed method is shown to outperform the state of the art methods in terms of accuracy and efficiency

a universal prior for integers and estimation by minimum description length of the number of bits required to write down the observed data has been reformulated to extend the classical maximum likelihood principle the principle permits estimation of the number of the parameters in statistical models in addition to their values and even of the way the parameters appear in the models i e of the model structures the principle rests on a new way to interpret and construct a universal prior distribution for the integers which makes sense even when the parameter is an individual object truncated realvalued parameters are converted to integers by dividing them by their precision and their prior is determined from the universal prior for the integers by optimizing the precision 1 introduction in this paper we study estimation based upon the principle of minimizing the total number of binary digits required to rewrite the observed data when each observation is given with some precision instead of attempting at an absolutely shortest description which would be futile we look for the optimum relative to a class of parametrically given distributions this minimum description length mdl principle which we introduced in a less comprehensive form in 25 turns out to degenerate to the more familiar maximum likelihood ml principle in case the number of parameters in the models is fixed so that the description length of the parameters themselves can be ignored in another extreme case where the parameters determine the data it similarly degenerates to jaynes s principle of maximum entropy 14 but the main power of the new criterion is that it permits estimates of the entire model its parameters their number and even the way the parameters appear in the model i e the model structure hence there will be no need to supplement the estimated parameters with a separate hypothesis test to decide whether a model is adequately parameterized or perhaps over parameterized

strong consistency of ml estimators using partial fraction and elementary subsystem representation of multivariable systems the strong consistency of ml estimators for partial fraction matrix pfm and elementary subsystem ess representations of vector arma processes will be discussed for the case when the structure is overestimated the main results of the paper are the off line strong consistency theorems of ml estimators for pfm representation when the number of poles is overestimated and for the ess representations when the number of ess s associated with the poles is overestimated

bootstrap modeling of a class of nonstationary signals the problem of modeling polynomial phase signals is considered techniques based on the bootstrap and multiple hypotheses tests for optimal model selection of constant amplitude polynomial phase signals embedded in stationary noise are developed phase parameter estimators based on both the least squares method and the polynomial phase transform are used the proposed techniques are compared with existing ones including akaike s 1978 1974 information criterion and rissanen s 1978 minimum description length criterion and are shown to outperform these procedures for the considered small sample sizes

periodic boundary value problems for second order impulsive differential systems on etudie le probleme aux valeurs limites periodiques pour un systeme d equations differentielles d ordre 2 faiblement couplees avec impulsion

brief identification of dynamic errors in variables models approaches based on two dimensional arma modeling of the data in this paper we propose a parametric and a non parametric identification algorithm for dynamic errors in variables model we show that the two dimensional process composed of the input output data admits a finite order arma representation the non parametric method uses the arma structure to compute a consistent estimate of the joint spectrum of the input and the output a frisch scheme is then employed to extract an estimate of the joint spectrum of the noise free input output data which in turn is used to estimate the transfer function of the system the parametric method exploits the arma structure to give estimates of the system parameters the performances of the algorithms are illustrated using the results obtained from a numerical simulation study

the determination of optimum structures for the state space representation of multivariate stochastic processes when identifying a model for a multivariate stationary stochastic process an important problem is that of determining the structure of the state variable model several overlapping parameterizations can usually be fitted to a given process and the question arises as to which structure leads to the most accurate parameter estimates the accuracy of parameter estimates is often measured by the determinant of the fisher information matrix we show that all admissible structures will give asymptotically the same value to this criterion for finite data some structures may still be better than others and two heuristic structure estimation methods are analyzed some simulation results are also presented

characterization and identification of matrix fraction descriptions for lti systems this paper shows that the matrix fraction description given by a pair a z b z of matrix polynomials of z for a linear time invariant system may not be unique even if a z is monic a z and b z have no common left factor and the matrix coefficients corresponding to the highest order terms of a z and b z are full row rank the orders of all possible matrix fraction descriptions mfds of a given system are completely characterized testing criteria for determining whether a matrix pair is an mfd of the system are derived which involve rank tests of certain toeplitz matrices derived from either the impulse response or output correlation functions of the system a decision procedure is devised that generates sequentially all mfds for a given system identification algorithms are introduced that estimate all mfds of a given system from its input output data or output data only the results are then extended to cover armax systems

a new method for ar order determination of an arma process using a combination of the minimum description length mdl criterion and overdetermined instrumental variable method a computationally efficient method for ar order determination of an arma process is proposed it is shown that the well known singular value decomposition svd method for ar order determination is a special case of the proposed method numerical simulations are presented to show the effectiveness of this method

discriminative feature extraction by a neural implementation of canonical correlation analysis the canonical correlation analysis cca aims at measuring linear relationships between two sets of variables views that can be used for feature extraction in classification problems with multiview data however the correlated features extracted by the cca may not be class discriminative since cca does not utilize the class labels in its traditional formulation although there is a method called discriminative cca dcca that aims to increase the discriminative ability of cca inspired from the linear discriminant analysis lda it has been shown that the extracted features with this method are identical to those by the lda with respect to an orthogonal transformation therefore dcca is simply equivalent to applying single view regular lda to each one of the views separately besides dcca and the other similar dcca approaches have generalization problems due to the sample covariance matrices used in their computation which are sensitive to outliers and noisy samples in this paper we propose a method called discriminative alternating regression d ar to explore correlated and also discriminative features d ar utilizes two alternating multilayer perceptrons each with a linear hidden layer learning to predict both the class labels and the outputs of each other we show that the features found by d ar on training sets significantly accomplish higher classification accuracies on test sets of facial expression recognition object recognition and image retrieval experimental data sets

finite sample criteria for autoregressive order selection the quality of selected ar models depends on the true process in the finite sample practice on the number of observations on the estimation algorithm and on the order selection criterion samples are considered to be finite if the maximum candidate model order for selection is greater than n 10 where n denotes the number of observations finite sample formulae give empirical approximations for the statistical average of the residual energy and of the squared error of prediction for several autoregressive estimation algorithms this leads to finite sample criteria for order selection that depend on the estimation method the special finite sample information criterion fsic and combined information criterion cic are necessary because of the increase of the variance of the residual energy for higher model orders that has not been accounted for in other criteria only the expectation of the logarithm of the residual energy as a function of the model order has been the basis for the previous classes of asymptotical and finite sample criteria however the behavior of the variance causes an undesirable tendency to select very high model orders without the special precautions of fsic or cic

multiple view semi supervised dimensionality reduction multiple view data together with some domain knowledge in the form of pairwise constraints arise in various data mining applications how to learn a hidden consensus pattern in the low dimensional space is a challenging problem in this paper we propose a new method for multiple view semi supervised dimensionality reduction the pairwise constraints are used to derive embedding in each view and simultaneously the linear transformation is introduced to make different embeddings from different pattern spaces comparable hence the consensus pattern can be learned from multiple embeddings of multiple representations we derive an iterating algorithm to solve the above problem some theoretical analyses and out of sample extensions are also provided promising experiments on various data sets together with some important discussions are also presented to demonstrate the effectiveness of the proposed algorithm

model selection through a statistical analysis of the global minimum of a weighted nonlinear least squares cost function this paper presents a model selection algorithm for the identification of parametric models that are linear in the measurements it is based on the mean and variance expressions of the global minimum of a weighted nonlinear least squares cost function the method requires the knowledge of the noise covariance matrix but does not assume that the true model belongs to the model set unlike the traditional order estimation methods available in literature the presented technique allows to detect undermodeling the theory is illustrated by simulations on signal modeling and system identification problems and by one real measurement example

multiset canonical correlations using globality preserving projections with applications to feature extraction and recognition multiset features extracted from the same patterns always represent different characteristics of data thus it is very valuable to perform the extraction on multiple feature sets this paper addresses the issue of multiset correlation feature extraction mcfe in multiple feature representations a novel method is proposed to carry out the mcfe for classification called multiset canonical correlations using globality preserving projections mcc gps which can perform joint dimensionality reduction for high dimensional data mcc gp integrates correlational characteristics of feature pairs and global geometric information of data in the transformed low dimensional space this makes mcc gps have better discriminant ability than a previous method proposed by the authors called multiset integrated canonical correlation analysis micca which only considers correlations for recognition tasks furthermore mcc gp can subsume two popular feature extraction methods into its framework under some constraints this also provides a new insight for these two methods the proposed method is applied to pattern recognition and examined using the coil 100 and eth 80 object databases and ar cmu pie and yale face databases extensive experimental results show that mcc gp outperforms micca and multiset canonical correlation analysis in terms of classification accuracy and efficiency

an algorithm for pole zero system model order estimation in system identification estimates of the unknown system model orders are often required an algorithm for estimating model orders is described that looks at input output data covariance matrix eigenvectors when model orders are overestimated zeros appear in the noise subspace eigenvectors the number of zeros present can be used to estimate model orders

multiset canonical correlations analysis and multispectral truly multitemporal remote sensing data this paper describes two and multiset canonical correlations analysis cca for data fusion multisource multiset or multitemporal exploratory data analysis these techniques transform multivariate multiset data into new orthogonal variables called canonical variates cvs which when applied in remote sensing exhibit ever decreasing similarity as expressed by correlation measures over sets consisting of 1 spectral variables at fixed points in time r mode analysis or 2 temporal variables with fixed wavelengths t mode analysis the cvs are invariant to linear and affine transformations of the original variables within sets which means for example that the r mode cvs are insensitive to changes over time in offset and gain in a measuring device in a case study cvs are calculated from landsat thematic mapper tm data with six spectral bands over six consecutive years both rand t mode cvs clearly exhibit the desired characteristic they show maximum similarity for the low order canonical variates and minimum similarity for the high order canonical variates these characteristics are seen both visually and in objective measures the results from the multiset cca r and t mode analyses are very different this difference is ascribed to the noise structure in the data the cca methods are related to partial least squares pls methods this paper very briefly describes multiset cca based multiset pls also the cca methods can be applied as multivariate extensions to empirical orthogonal functions eof techniques multiset cca is well suited for inclusion in geographical information systems gis

non parametric detection of the number of signals hypothesis testing and random matrix theory detection of the number of signals embedded in noise is a fundamental problem in signal and array processing this paper focuses on the non parametric setting where no knowledge of the array manifold is assumed first we present a detailed statistical analysis of this problem including an analysis of the signal strength required for detection with high probability and the form of the optimal detection test under certain conditions where such a test exists second combining this analysis with recent results from random matrix theory we present a new algorithm for detection of the number of sources via a sequence of hypothesis tests we theoretically analyze the consistency and detection performance of the proposed algorithm showing its superiority compared to the standard minimum description length mdl based estimator a series of simulations confirm our theoretical analysis

robust and accurate arx and arma model order estimation of non gaussian processes the field of higher order statistics is emerging rapidly for analyzing non gaussian processes there are several motivations behind the use of higher order statistics the emphasis of this paper is based on the property that higher order cumulants are blind to any kind of gaussian process hence when the processed signal is non gaussian stationary and the additive noise is stationary gaussian the noise will vanish in the cumulant domain this paper presents an extension to the results of liang et al 1993 this extension is a straightforward generalization of liang s approach to third order cumulants tocs the new cumulant based algorithm provides a higher level of accuracy in the presence of noise than the original second order algorithm in addition the original results of liang have been extended to the case of colored gaussian noise examples are given to demonstrate the performance of this algorithm even when the observed signal is heavily corrupted by gaussian noise

rapid and brief communication a theorem on the generalized canonical projective vectors this paper proposes a kind of generalized canonical projective vectors gcpv based on the framework of canonical correlation analysis cca applying image recognition apart from canonical projective vectors cpv the process of obtaining gcpv contains the class information of samples such that the combined features extracted according to the basis of gcpv can give a better classification performance the experimental result based on the concordia university cenparmi handwritten arabian numeral database has proved that our method is superior to the method based on cpv

adaptive stochastic filters with no strict positive real condition a new adaptive stochastic filter structure is introduced which avoids the strict passivity test used as a sufficient condition for convergence required by existing adaptive schemes the proposed algorithm consists of three stages in the first stage an autoregressive model is fitted and the residue obtained is used as an estimate of the noise in the second stage an autoregressive recursive moving average model is fitted using the residual of the first stage a modified residual is then filtered using a parameter and the model obtained from the second stage to generate an improved estimate of the noise in the third stage this improved estimate of the noise is used to obtain a better autoregressive moving average model it is shown that the proposed algorithm will also reduce the bias in the estimated parameters the simulation results given show that the proposed filter compares favorably to the algorithm introduced by mayne and clark and also landau this filter is then applied to the adaptive line enhancement sinusoidal detection and adaptive spectral estimation problems to illustrate its usefulness

sequential estimation of hidden arma processes by particle filtering part ii this is part ii of a series of two papers where we address sequential estimation of wide sense stationary autoregressive moving average arma state processes by particle filtering in part i we considered a state space model where the state was an arma process of known order and where the parameters of the process could be known or unknown in this paper we extend our work from part i by considering the same type of models with the added complexity that the arma processes are now of unknown order instead of working on a scheme that first tracks the state by operating with different assumed models and then selects the best model by using a predefined criterion we present a method that directly estimates the state without the need of knowing the model order we derive the transition density of the state for unknown arma model order and propose a particle filter based on that density and the empirical bayesian methodology we demonstrate the performance of the proposed method with computer simulations and compare it with the methods from part i

joint blind source separation by multiset canonical correlation analysis in this paper we introduce a simple and effective scheme to achieve joint blind source separation bss of multiple datasets using multiset canonical correlation analysis m cca j r kettenring canonical analysis of several sets of variables biometrika vol 58 pp 433 451 1971 we first propose a generative model of joint bss based on the correlation of latent sources within and between datasets we specify source separability conditions and show that when the conditions are satisfied the group of corresponding sources from each dataset can be jointly extracted by m cca through maximization of correlation among the extracted sources we compare source separation performance of the m cca scheme with other joint bss methods and demonstrate the superior performance of the m cca scheme in achieving joint bss for a large number of datasets group of corresponding sources with heterogeneous correlation values and complex valued sources with circular and non circular distributions we apply m cca to analysis of functional magnetic resonance imaging fmri data from multiple subjects and show its utility in estimating meaningful brain activations from a visuomotor task

nonlinear component analysis as a kernel eigenvalue problem a new method for performing a nonlinear form of principal component analysis is proposed by the use of integral operator kernel functions one can efficiently compute principal components in high dimensional feature spaces related to input space by some nonlinear map for instance the space of all possible five pixel products in 16 16 images we give the derivation of the method and present experimental results on polynomial feature extraction for pattern recognition

nonlinear second order periodic boundary value problems abstract in this paper we study a periodic boundary value problem pbvp for brevity of second order it is well known that if and are upper and lower solutions respectively with then there exists a solution u such that u when nothing was known the main purpose of this paper is to consider this last case also we study the structure of the set of solutions of the pbvp in the sector when and we apply the results to the case

an optimal instrumental variable method for arma spectral estimation a multistep iterative and fast recursive algorithm for autoregressive moving average arma spectral estimation is presented the ar parameters of an arma process are estimated using the extended instrumental variable eiv method the optimal choice of instruments prefilter and weighting matrix is investigated a bootstrapping procedure that has computational convenience is proposed for the algorithm the statistical analysis and experiments show that the optimal iv estimate is unbiased consistent efficient asymptotically normal and equivalent to the maximum likelihood ml estimate and the prediction error pe estimate and the proposed algorithm has the advantages of sharper resolution less frequency bias and better efficiency of convergence

existence of periodic solutions for first order differential equations in this paper we study the periodic boundary value problem for first order differential equations by combining techniques of the theory of differential inequalities namely the method of upper and lower solutions and the alternative method for nonlinear problems at resonance the results obtained are in terms of the behavior of the nonlinear part at infinity

periodic solutions of some lienard and duffing equations on etudie certains aspects du probleme periodique u f u u g t u 0 u 0 u 2 u 0 u 2 ou f est une fonction continue sur r et g t u est une fonction de caratheodory soumise a certaines restrictions de croissances on donne deux resultats d existence

a covariance extension approach to identification of time series in this paper we consider a three step procedure for identification of time series based on covariance extension and model reduction and we present a complete analysis of its statistical convergence properties a partial covariance sequence is estimated from statistical data then a high order maximum entropy model is determined which is finally approximated by a lower order model by stochastically balanced model reduction such procedures have been studied before in various combinations but an overall convergence analysis comprising all three steps has been lacking supposing the data is generated from a true finite dimensional system which is minimum phase it is shown that the transfer function of the estimated system tends in h to the true transfer function as the data length tends to infinity if the covariance extension and the model reduction is done properly the proposed identification procedure and some variations of it are evaluated by simulations

new method of order estimation for arma armax processes let the observation y k be generated by the multivariate arma process a z y k c z w k with unknown coefficients theta a theta c and orders p r where w k is a sequence of independent and identically distributed i i d random vectors with zero mean and unknown covariance matrix r w 0 a new method for estimating the orders p r is introduced in contrast to most of the existing results the new method is not based on optimizing a certain criterion and the order estimates given in the paper are rather easy to update computationally in comparison with the criterion optimization based methods when new data arrive the method is then extended to determining the orders of armax processes under reasonable conditions the estimates are proved to converge to the true orders with probability one as time tends to infinity

consistent order estimation and minimal penalties consider an i i d sequence of random variables whose distribution f lies in one of the nested families of models mq q 1 the smallest index q such that mq contains f is called the model order the aim of this paper is to explore the consistency properties of penalized likelihood model order estimators such as bayesian information criterion we show in a general setting that the minimal strongly consistent penalty is of order q loglogn where q is a dimensional quantity in contrast to previous work an a priori upper bound on the model order is not assumed the results rely on a sharp characterization of the pathwise fluctuations of the generalized likelihood ratio statistic under entropy assumptions on the model classes our results are applied to the geometrically complex problem of location mixture order estimation which is widely used but poorly understood

a two stage linear discriminant analysis via qr decomposition linear discriminant analysis lda is a well known method for feature extraction and dimension reduction it has been used widely in many applications involving high dimensional data such as image and text classification an intrinsic limitation of classical lda is the so called singularity problems that is it fails when all scatter matrices are singular many lda extensions were proposed in the past to overcome the singularity problems among these extensions pca lda a two stage method received relatively more attention in pca lda the lda stage is preceded by an intermediate dimension reduction stage using principal component analysis pca most previous lda extensions are computationally expensive and not scalable due to the use of singular value decomposition or generalized singular value decomposition in this paper we propose a two stage lda method namely lda qr which aims to overcome the singularity problems of classical lda while achieving efficiency and scalability simultaneously the key difference between lda qr and pca lda lies in the first stage where lda qr applies qr decomposition to a small matrix involving the class centroids while pca lda applies pca to the total scatter matrix involving all training data points we further justify the proposed algorithm by showing the relationship among lda qr and previous lda methods extensive experiments on face images and text documents are presented to show the effectiveness of the proposed algorithm

kernel independent component analysis we present a class of algorithms for independent component analysis ica which use contrast functions based on canonical correlations in a reproducing kernel hilbert space on the one hand we show that our contrast functions are related to mutual information and have desirable mathematical properties as measures of statistical dependence on the other hand building on recent developments in kernel methods we show that these criteria and their derivatives can be computed efficiently minimizing these criteria leads to flexible and robust algorithms for ica we illustrate with simulations involving a wide variety of source distributions showing that our algorithms outperform many of the presently known algorithms

3 d ar model order selection via rank test procedure this paper deals with the problem of three dimensional autoregressive 3 d ar model order estimation we show that the information for the 3 d ar model order is implicitly contained in an appropriate matrix rank built from the autocorrelation function acf of the underlying 3 d gaussian process exploiting this property we develop an algorithm to estimate the order p1 p2 p3 corresponding to the quarter space qs region of support the proposed method is based upon a rank test procedure rtp using singular value decomposition svd and solving nonlinear system equations numerical simulations are presented to illustrate the performances of the proposed algorithm

arma spectral estimation of narrow band processes via model reduction the problem of estimating autoregressive moving average arma models for narrowband processes is considered the following approach is proposed estimate a high order autoregressive ar approximation of the process by model reduction based on a truncated internally balanced realization or optimal hankel norm model reduction reduce the order of this high order ar estimate to find a lower order arma model this algorithm gives arma spectral estimates with excellent resolution properties without using iterative numerical minimization methods as for the maximum likelihood method how to take the narrowband assumption into account in the model reduction step is discussed in detail

state and parameter estimation for dynamic systems with colored noise an accuracy approach based on the minimum discrepancy measure an autoregressive moving average arma order determination criterion based on a minimum discrepancy measure mdm is proposed to model colored measurement noise based on this criterion an algorithm is presented to perform the state and parameter estimation the criterion is consistent in the sense that the mdm decision rule will choose the correct order as the number of observations tends to infinity experimental results are included to illustrate the effectiveness of this technique

kpca plus lda a complete kernel fisher discriminant framework for feature extraction and recognition this paper examines the theory of kernel fisher discriminant analysis kfd in a hilbert space and develops a two phase kfd framework i e kernel principal component analysis kpca plus fisher linear discriminant analysis lda this framework provides novel insights into the nature of kfd based on this framework the authors propose a complete kernel fisher discriminant analysis ckfd algorithm ckfd can be used to carry out discriminant analysis in double discriminant subspaces the fact that it can make full use of two kinds of discriminant information regular and irregular makes ckfd a more powerful discriminator the proposed algorithm was tested and evaluated using the feret face database and the cenparmi handwritten numeral database the experimental results show that ckfd outperforms other kfd algorithms

a reverse formulation of the rise algorithm the recently developed recursive iterative self adjoint eigenspace rise decomposition algorithm recursively computes the complete eigenspace decomposition of successively larger hermitian matrices however some practical applications also require that the decomposition of successively smaller matrices be computed a modification to the rise algorithm is presented that makes it possible to run this algorithm backward on successively smaller hermitian matrices this important modification increases the number of practical applications of this algorithm

periodic solutions of nonlinear boundary value problems one of the well known techniques in the theory of nonlinear boundary value problems bvp is the method of differential inequalities or the method of upper and lower solutions the method of alternative problems a global variant of the lyapunov schmidt method has been used in the study of problems at resonance the investigation of periodic bvp s forms an important subclass of problems at resonance our aim is to combine the two approaches to discuss nonlinear problems at resonance we restrict ourselves in this paper to the discussion of periodic boundary value problems in section 1 we shall indicate the method of upper and lower solutions the results for systems are given in a general way so as to include known results and also offer new directions section 2 deals with the abstract existence results at resonance in the desired framework in section 3 we consider the scalar periodic bvp s by combining the two techniques namely alternative methods and the method of differential inequalities

a method for autoregressive moving average estimation summary the problem considered is that of estimating an autoregressive moving average system including estimating the degrees of the autoregressive and moving average lag operators the basic method is that introduced by hannan rissanen 1982 however that method may sometimes overestimate the degrees and modifications are here introduced to correct this the problem is itself due to the use of a long autoregression of order clog t when t is large in the first stage of the process the effect of this is investigated and in particular its effect on the speed of convergence of the estimates

face recognition using laplacianfaces we propose an appearance based face recognition method called the laplacianface approach by using locality preserving projections lpp the face images are mapped into a face subspace for analysis different from principal component analysis pca and linear discriminant analysis lda which effectively see only the euclidean structure of face space lpp finds an embedding that preserves local information and obtains a face subspace that best detects the essential face manifold structure the laplacianfaces are the optimal linear approximations to the eigenfunctions of the laplace beltrami operator on the face manifold in this way the unwanted variations resulting from changes in lighting facial expression and pose may be eliminated or reduced theoretical analysis shows that pca lda and lpp can be obtained from different graph models we compare the proposed laplacianface approach with eigenface and fisherface methods on three different face data sets experimental results suggest that the proposed laplacianface approach provides a better representation and achieves lower error rates in face recognition

arma model order estimation based on the eigenvalues of the covariance matrix an approach to model order determination based on the minimum description length mdl criterion is proposed and shown to depend on the minimum eigenvalues of a covariance matrix derived from the observed data a selection procedure for estimating the model order by means of the mdl method is proposed examples are given to illustrate the significantly improved accuracy of the technique

correlation metric for generalized feature extraction beyond linear and kernel based feature extraction we propose in this paper the generalized feature extraction formulation based on the so called graph embedding framework two novel correlation metric based algorithms are presented based on this formulation correlation embedding analysis cea which incorporates both correlational mapping and discriminating analysis boosts the discriminating power by mapping data from a high dimensional hypersphere onto another low dimensional hypersphere and preserving the intrinsic neighbor relations with local graph modeling correlational principal component analysis cpca generalizes the conventional principal component analysis pca algorithm to the case with data distributed on a high dimensional hypersphere their advantages stem from two facts 1 tailored to normalized data which are often the outputs from the data preprocessing step and 2 directly designed with correlation metric which shows to be generally better than euclidean distance for classification purpose extensive comparisons with existing algorithms on visual classification experiments demonstrate the effectiveness of the proposed algorithms

parametric identification using weighted null space fitting in identification of dynamical systems the prediction error method with a quadratic cost function provides asymptotically efficient estimates under gaussian noise but in general it requires solving a nonconvex optimization problem which may imply convergence to nonglobal minima an alternative class of methods uses a nonparametric model as intermediate step to obtain the model of interest weighted null space fitting wnsf belongs to this class starting with the estimate of a nonparametric arx model with least squares then the reduction to a parametric model is a multistep procedure where each step consists of the solution of a quadratic optimization problem which can be obtained with weighted least squares the method is suitable for both open and closed loop data and can be applied to many common parametric model structures including output error armax and box jenkins the price to pay is the increase of dimensionality in the nonparametric model which needs to tend to infinity as function of the sample size for certain asymptotic statistical properties to hold in this paper we conduct a rigorous analysis of these properties namely consistency and asymptotic efficiency also we perform a simulation study illustrating the performance of wnsf and identify scenarios where it can be particularly advantageous compared with state of the art methods

model reductions of high order estimated models the asymptotic ml approach abstract the reduction of order of high order models obtained from an identification experiment is discussed from a statistical point of view the asymptotic maximum likelihood ml approach is defined to reduce the order of an estimated model this approach considers the maximum likelihood criterion given the asymptotic statistics both in the number of data and the order of the estimated model and corresponds to frequency weighted l 2 norm model reduction by using the insight from the asymptotic ml approach an identification algorithm is proposed based on a high order arx estimate and model reduction via a frequency weighted balanced realization the advantage of this algorithm is that iterative minimization methods are not required to find the estimate

noisy data and impulse response estimation this paper investigates the impulse response estimation of linear time invariant lti systems when only noisy finite length input output data of the system is available the competing parametric candidates are the least square impulse response estimates of possibly different lengths it is known that the presence of noise prohibits using model sets with large number of parameters as the resulting parameter estimation error can be quite large model selection methods acknowledge this problem hence they provide metrics to compare estimates in different model classes such metrics typically involve a combination of the available least square output error which decreases as the number of parameters increases and a function that penalizes the size of the model in this paper we approach the model class selection problem from a different perspective that is closely related to the involved denoising problem the method primarily focuses on estimating the parameter error in a given model class of finite order using the available least square output error we show that such an estimate which is provided in terms of upper and lower bounds with certain level of confidence contains the appropriate tradeoffs between the bias and variance of the estimation error consequently these measures can be used as the basis for model comparison and model selection furthermore we demonstrate how this approach reduces to the celebrated aic method for a specific confidence level the performance of the method as the noise variance and or the data length varies is explored and consistency of the approach as the data length grows is analyzed

laplacian eigenmaps for dimensionality reduction and data representation one of the central problems in machine learning and pattern recognition is to develop appropriate representations for complex data we consider the problem of constructing a representation for data lying on a low dimensional manifold embedded in a high dimensional space drawing on the correspondence between the graph laplacian the laplace beltrami operator on the manifold and the connections to the heat equation we propose a geometrically motivated algorithm for representing the high dimensional data the algorithm provides a computationally efficient approach to nonlinear dimensionality reduction that has locality preserving properties and a natural connection to clustering some potential applications and illustrative examples are discussed

technical communique linear estimation of arma processes the method of estimating arma parameters first discussed by mayne and firoozan is investigated it is shown that if at the first stage the order of the fitted autoregression is allowed to depend on the number of time points in a reasonable manner then it will still be true that the final estimate of the parameter vector will converge almost surely to the true value this is to be compared to the result in the original paper where the order is fixed and it is shown that as the sample size increases the estimate converges to a value which if the order of the autoregression is high enough will be arbitrarily near to the true value some comments are made on other extensions on the law of the iterated logarithm on the central limit theorem and on the choice of the order of the fitted autoregression the innovation sequence need not be gaussian and for the convergence result only a natural condition relating to prediction needs be imposed

on the use of autoregressive order determination criteria in multivariate white noise tests testing the hypothesis of multivariate white noise is seen as the selection of the order of a multivariate autoregressive model for the observed time series therefore multivariate white noise tests can be carried out by applying autoregressive order determination criteria such as aic bic etc it is known for example that the bic criterion estimates consistently the order of an autoregression an order determination criterion with this property leads to a white noise test with a significance level approaching zero as n the number of observations increases the order of an autoregressive moving average model is proposed to be determined by applying this kind of white noise test the resulting model building procedure is a generalization of the procedure proposed by g e p box and g m jenkins 1970

appearance models based on kernel canonical correlation analysis this paper introduces a new approach to constructing appearance models based on kernel canonical correlation analysis kernel cca kernel cca is a non linear extension of cca where a non linear transformation of the input data is performed implicitly using kernel methods although in this respect it is similar to other generalized linear methods kernel cca is especially well suited for relating two sets of measurements the benefits of our method compared to standard feature extraction methods based on pca will be illustrated experimentally for the task of estimating an object s pose from raw brightness images

parameter and structure identification of linear multivariable systems abstract in this paper a parameter and structure identification scheme which can be used for both continuous and discrete time systems is presented this scheme has an on line nature and it incorporates standard techniques such as state variable filters and adaptation laws developed through stability analysis the canonical form being used is based on the output injection which is constructed to assign all the poles of the system to an arbitrary chosen real number it is therefore named the canonical form

paper a new algorithm for recursive estimation of parameters in controlled arma processes a recursive algorithm for estimating the constant but unknown parameters of a controlled arma process is presented the algorithm is a recursive version of an off line algorithm using three stages of standard least squares in the first stage the parameters of a controlled ar model of degree p are estimated the residuals used in this stage are employed in the second stage to estimate the parameters of a controlled arma process the first two stages constitute a recursive version of durbin s algorithm the model obtained in the second stage is used to filter the input output and residuals and these filtered variables are used in the final stage to obtain improved estimates of the controlled arma process it is shown that the estimate is globally p consistent i e that the estimate converges a s as the number of data tends to infinity to a vector which in turn converges to the true parameter vector as the degree p of the ar model tends to infinity

learning the kernel matrix with semidefinite programming kernel based learning algorithms work by embedding the data into a euclidean space and then searching for linear relations among the embedded data points the embedding is performed implicitly by specifying the inner products between each pair of points in the embedding space this information is contained in the so called kernel matrix a symmetric and positive semidefinite matrix that encodes the relative positions of all points specifying this matrix amounts to specifying the geometry of the embedding space and inducing a notion of similarity in the input space classical model selection problems in machine learning in this paper we show how the kernel matrix can be learned from data via semidefinite programming sdp techniques when applied to a kernel matrix associated with both training and test data this gives a powerful transductive algorithm using the labeled part of the data one can learn an embedding also for the unlabeled part the similarity between test points is inferred from training points and their labels importantly these learning problems are convex so we obtain a method for learning both the model class and the function without local minima furthermore this approach leads directly to a convex method for learning the 2 norm soft margin parameter in support vector machines solving an important open problem

regularized multiple set canonical correlation analysis multiple set canonical correlation analysis generalized cano or gcano for short is an important technique because it subsumes a number of interesting multivariate data analysis techniques as special cases more recently it has also been recognized as an important technique for integrating information from multiple sources in this paper we present a simple regularization technique for gcano and demonstrate its usefulness regularization is deemed important as a way of supplementing insufficient data by prior knowledge and or of incorporating certain desirable properties in the estimates of parameters in the model implications of regularized gcano for multiple correspondence analysis are also discussed examples are given to illustrate the use of the proposed technique

a fast estimation method for arma processes an improvement of a batch algorithm for parameter estimation of an arma process is presented a fast and iterative algorithm is presented and provides estimates with the same asymptotic distribution as the maximum likelihood estimates this algorithm is a modified version of the method suggested by mayne and firoozan 1982 linear identification of arma processes automatica 18 461 466 and requires one less regression than the previous method if only one pass is executed

on a multivariate eigenvalue problem part i algebraic theory and a power method multivariate eigenvalue problems for symmetric and positive definite matrices arise from multivariate statistics theory where coefficients are to be determined so that the resulting linear combinations of sets of random variables are maximally correlated by using the method of lagrange multipliers such an optimization problem can be reduced to the multivariate eigenvalue problem for over 30 years an iterative method proposed by horst psychometrika 26 1961 pp 129 149j has been used for solving the multivariate eigenvalue problem yet the theory of convergence has never been complete the number of solutions to the multivariate eigenvalue problem also remains unknown this paper contains two new results by using the degree theory a closed form on the cardinality of solutions for the multivariate eigenvalue problem is first proved a convergence property of horst s method by forming it as a generalization of the so called power method is then proved the discussion leads to new formulations of nume