unified approach to h optimization hankel approximation and balanced realization problems a unified approach is proposed here to solve three problems simultaneously namely h optimization hankel approximation and minimal balanced realization for scalar systems it is shown that all these problems can be reduced to the same standard form wherein merely the singular value decomposition svd of an easily constructed square matrix is required to establish the final solutions the formulation is solely in terms of the coefficients of a transfer function and the resulting algorithm can be performed in a very computationally efficient way

latent classification models for binary data one of the simplest and yet most consistently well performing set of classifiers is the naive bayes models a special class of bayesian network models however these models rely on the naive assumption that all the attributes used to describe an instance are conditionally independent given the class of that instance to relax this independence assumption we have in previous work proposed a family of models called latent classification models lcms lcms are defined for continuous domains and generalize the naive bayes model by using latent variables to model class conditional dependencies between the attributes in addition to providing good classification accuracy the lcm has several appealing properties including a relatively small parameter space making it less susceptible to over fitting in this paper we take a first step towards generalizing lcms to hybrid domains by proposing an lcm for domains with binary attributes we present algorithms for learning the proposed model and we describe a variational approximation based inference procedure finally we empirically compare the accuracy of the proposed model to the accuracy of other classifiers for a number of different domains including the problem of recognizing symbols in black and white images

parameter expansion to accelerate em the px em algorithm summary the em algorithm and its extensions are popular tools for modal estimation but ar e often criticised for their slow convergence we propose a new method that can often make em much faster the intuitive idea is to use a covariance adjustment to correct the analysis of the m step capitalising on extra information captured in the imputed complete data the way we accomplish this is by parameter expansion we expand the complete data model while preserving the observed data model and use the expanded complete data model to generate em this parameter expanded ei m px em algorithm shares the simplicity and stability of ordinary em but has a faster rate of convergence since its m step performs a more efficient analysis the px em algorithm is illustrated for the multivariate t distribution a random effects model factor analysis probit regression and a poisson imaging model

exploratory longitudinal factor analysis in multiple populations for multiple populatios a longtidinal factor analytic model which is entirely exploratory that is no explicit identification constraints is proposed factorial collapse and period practice effects are allowed an invariant and or stationary factor pattern is permitted this model is formulated stochastically to implement this model a stagewise em algorithm is developed finally a numerical illustration utilizing nesselroade and baltes data is presented

component analysis in cross sectional and longitudinal data an extension of component analysis to longitudinal or cross sectional data is presented in this method components are derived under the restriction of invariant and or stationary compositing weights optimal compositing weights are found numerically the method can be generalized to allow differential weighting of the observed variables in deriving the component solution some choices of weightings are discussed an illustration of the method using real data is presented

alternating least squares algorithms for simultaneous components analysis with equal component weight matrices in two or more populations millsap and meredith 1988 have developed a generalization of principal components analysis for the simultaneous analysis of a number of variables observed in several populations or on several occasions the algorithm they provide has some disadvantages the present paper offers two alternating least squares algorithms for their method suitable for small and large data sets respectively lower and upper bounds are given for the loss function to be minimized in the millsap and meredith method these can serve to indicate whether or not a global optimum for the simultaneous components analysis problem has been attained

variational learning in nonlinear gaussian belief networks we view perceptual tasks such as vision and speech recognition as inference problems where the goal is to estimate the posterior distribution over latent variables e g depth in stereo vision given the sensory input the recent flurry of research in independent component analysis exemplifies the importance of inferring the continuousvalued latent variables of input data the latent variables found by this method are linearly related to the input but perception requires nonlinear inferences such as classification and depth estimation in this paper we present a unifying framework for stochastic neural networks with nonlinear latent variables nonlinear units are obtained by passing the outputs of linear gaussian units through various nonlinearities we present a general variational method that maximizes a lower bound on the likelihood of a training set and give results on two visual feature extraction problems we also show how the variational method can be used for pattern classification and compare the performance of these nonlinear networks with other methods on the problem of handwritten digit recognition

stochastic approximation em for large scale exploratory irt factor analysis a stochastic approximation em algorithm saem is described for exploratory factor analysis of dichotomous or ordinal variables the factor structure is obtained from sufficient statistics that are updated during iterations with the robbins monro procedure two large scale simulations are reported that compare accuracy and cpu time of the proposed saem algorithm to the metropolis hasting robbins monro procedure and to a generalized least squares analysis of the polychoric correlation matrix a smaller scale application to real data is also reported including a method for obtaining standard errors of rotated factor loadings a simulation study based on the real data analysis is conducted to study bias and error estimates the saem factor algorithm requires minimal lines of code no derivatives and no large matrix inversion it is programmed entirely in r

complex factor analysis and extensions many subspace based array signal processing algorithms assume that the noise is spatially white in this case the noise covariance matrix is a multiple of the identity and the eigenvectors of the data covariance matrix are not affected by it if the noise covariance is an unknown arbitrary diagonal e g for an uncalibrated array the eigenvalue decomposition leads to incorrect subspace estimates and it has to be replaced by a more general factor analysis decomposition fa which then reveals all relevant information we consider this data model and several extensions where the noise covariance matrix has a more general structure such as banded sparse block diagonal and cases where we have multiple data covariance matrices that share the same noise covariance matrix starting from a nonlinear weighted least squares formulation we propose new estimation algorithms for both classical fa and its extensions the optimization is based on gauss newton gradient descent generally this leads to an iteration involving the inversion of a very large matrix using the structure of the problem we show how this can be reduced to the inversion of a matrix with dimension equal to the number of unknown noise covariance parameters this results in new algorithms that have faster numerical convergence and lower complexity compared to several maximum likelihood based algorithms that could be considered state of the art the new algorithms scale well to large dimensions and can replace eigenvalue decompositions in many applications even if the noise can be assumed to be white

internally balanced minimal realization of discrete siso systems given the transfer function or the state equations in phase canonical form of a discrete siso system a method is proposed to obtain an internally balanced state space representation of it

maximization of sums of quotients of quadratic forms and some generalizations monotonically convergent algorithms are described for maximizing six constrained functions of vectors x or matrices x with columns x 1 x r these functions are h 1 x sigma k x a k x x c k x 1 h 1 x sigma k tr x a k x x c k x 1 h over tilde 1 x sigma k sigma l x l a k x l x l c k x l 1 with x constrained to be columnwise orthonormal h 2 x sigma k x a k x 2 x c k x 1 subject to x x 1 h 2 x sigma k tr x a k x x a k x x c k x 1 subject to x x i and h over tilde 2 x sigma k sigma l x l a k x l 2 x l c k x l 1 subject to x x i in these functions the matrices c k are assumed to be positive definite the matrices a k can be arbitrary square matrices the general formulation of the functions and the algorithms allows for application of the algorithms in various problems that arise in multivariate analysis several applications of the general algorithms are given specifically algorithms are given for reciprocal principal components analysis binormamin rotation generalized discriminant analysis variants of generalized principal components analysis simple structure rotation for one of the latter variants and set component analysis for most of these methods the algorithms appear to be new for the others the existing algorithms turn out to be special cases of the newly derived general algorithms

em and beyond the basic theme of the em algorithm to repeatedly use complete data methods to solve incomplete data problems is also a theme of several more recent statistical techniques these techniques multiple imputation data augmentation stochastic relaxation and sampling importance resampling combine simulation techniques with complete data methods to attack problems that are difficult or impossible for em

computation of the maximum likelihood estimator in low rank factor analysis factor analysis is a classical multivariate dimensionality reduction technique popularly used in statistics econometrics and data science estimation for factor analysis is often carried out via the maximum likelihood principle which seeks to maximize the gaussian likelihood under the assumption that the positive definite covariance matrix can be decomposed as the sum of a low rank positive semidefinite matrix and a diagonal matrix with nonnegative entries this leads to a challenging rank constrained nonconvex optimization problem for which very few reliable computational algorithms are available we reformulate the low rank maximum likelihood factor analysis task as a nonlinear nonsmooth semidefinite optimization problem study various structural properties of this reformulation and propose fast and scalable algorithms based on difference of convex optimization our approach has computational guarantees gracefully scales to large problems is applicable to situations where the sample covariance matrix is rank deficient and adapts to variants of the maximum likelihood problem with additional constraints on the model parameters our numerical experiments validate the usefulness of our approach over existing state of the art approaches for maximum likelihood factor analysis

an incomplete data approach to the analysis of covariance structures in this paper linear structural equation models with latent variables are considered it is shown how many common models arise from incomplete observation of a relatively simple system subclasses of models with conditional independence interpretations are also discussed using an incomplete data point of view the relationships between the incomplete and complete data likelihoods assuming normality are highlighted for computing maximum likelihood estimates the em algorithm and alternatives are surveyed for the alternative algorithms simplified expressions for computing function values and derivatives are given likelihood ratio tests based on complete and incomplete data are related and an example on using their relationship to improve the fit of a model is given

factor analysis models via i divergence optimization given a positive definite covariance matrix formula see text of dimension n we approximate it with a covariance of the form formula see text where h has a prescribed number formula see text of columns and formula see text is diagonal the quality of the approximation is gauged by the i divergence between the zero mean normal laws with covariances formula see text and formula see text respectively to determine a pair h d that minimizes the i divergence we construct by lifting the minimization into a larger space an iterative alternating minimization algorithm aml a la csiszar tusnady as it turns out the proper choice of the enlarged space is crucial for optimization the convergence of the algorithm is studied with special attention given to the case where d is singular the theoretical properties of the aml are compared to those of the popular em algorithm for exploratory factor analysis inspired by the ecme a newton raphson variation on em we develop a similar variant of aml called acml and in a few numerical experiments we compare the performances of the four algorithms

latent curve analysis as a method for representing development latent trait theory is presented in terms of a statistical model containing individual parameters and a structure on both the first and second moments of the random variables reflecting growth maximum likelihood parameter estimates and associated asymptotic tests follow directly these procedures may be viewed as an alternative to standard repeated measures anova and to first order auto regressive methods as formulated the model encompasses cohort sequential designs and allow for period or practice effects a numerical illustration using data initially collected by nesselroade and baltes is presented

a family of association coefficients for metric scales four types of metric scales are distinguished the absolute scale the ratio scale the difference scale and the interval scale a general coefficient of association for two variables of the same metric scale type is developed some properties of this general coefficient are discussed it is shown that the matrix containing these coefficients between any number of variables is gramian the general coefficient reduces to specific coefficients of association for each of the four metric scales two of these coefficients are well known the product moment correlation and tucker s congruence coefficient applications of the new coefficients are discussed

estimation of discrete distributions with a class of simplex constraints abstract simplex constraints such as monotonicity and convexity or concavity on the probabilities of a set of discrete distributions are useful for modeling and analyzing discrete data this article considers both maximum likelihood estimation and bayesian estimation of discrete distribution with a class of simplex constraints using the expectation maximization em algorithm and the data augmentation da algorithm the formulation and implementation of em and da for binomial poisson hierarchical poisson binomial multinomial and hierarchical multinomial distributions are considered in detail and illustrated with examples

maximum likelihood estimation of nonlinear structural equation models the existing maximum likelihood theory and its computer software in structural equation modeling are established based on linear relationships among manifest variables and latent variables however models with nonlinear relationships are often encountered in social and behavioral sciences in this article an em type algorithm is developed for maximum likelihood estimation of a general nonlinear structural equation model to avoid computation of the complicated multiple integrals involved the e step is completed by a metropolis hastings algorithm it is shown that the m step can be completed efficiently by simple conditional maximization standard errors of the maximum likelihood estimates are obtained via louis s formula the methodology is illustrated with results from a simulation study and two real examples

nonconvergence improper solutions and starting values in lisrel maximum likelihood estimation in the framework of a robustness study on maximum likelihood estimation with lisrel three types of problems are dealt with nonconvergence improper solutions and choice of starting values the purpose of the paper is to illustrate why and to what extent these problems are of importance for users of lisrel the ways in which these issues may affect the design and conclusions of robustness research is also discussed

balanced minimal realization via singular value decomposition of sarason operator abstract in this paper a method is proposed to obtain a balanced minimal realization of a stable rational transfer function matrix the allison young algorithm for the interpolation problems is directly extended to matrix value cases and is applied in the realization procedure this method uses the finite matrix of a sarason operator instead of the infinite block hankel matrix and the coefficients of the transfer function matrix instead of any initial realization the advantage of this approach in connection with the solution of l optimization problems and the balanced realization is also discussed

transformation to achieve a longitudinally stationary factor pattern matrix meredith s method of extracting a factorially invariant solution is adapted to longitudinal settings an explorational estimation procedure is presented which attempts to identify the longitudinal factor components of an across occasion variance covariance matrix this is effected by transforming an initial factor pattern matrix to stationarity the estimation is performed in two parts the first employing a stepwise algorithm to ascertain the dimensionality and existence of the longitudinal components and the second being the direct estimation of the existing factor pattern

algebraic factor analysis tetrads pentads and beyond factor analysis refers to a statistical model in which observed variables are conditionally independent given fewer hidden variables known as factors and all the random variables follow a multivariate normal distribution the parameter space of a factor analysis model is a subset of the cone of positive definite matrices this parameter space is studied from the perspective of computational algebraic geometry grobner bases and resultants are applied to compute the ideal of all polynomial functions that vanish on the parameter space these polynomials known as model invariants arise from rank conditions on a symmetric matrix under elimination of the diagonal entries of the matrix besides revealing the geometry of the factor analysis model the model invariants also furnish useful statistics for testing goodness of fit

expandable factor analysis bayesian sparse factor models have proven useful for characterizing dependence in multivariate data but scaling computation to large numbers of samples and dimensions is problematic we propose expandable factor analysis for scalable inference in factor models when the number of factors is unknown the method relies on a continuous shrinkage prior for efficient maximum a posteriori estimation of a low rank and sparse loadings matrix the structure of the prior leads to an estimation algorithm that accommodates uncertainty in the number of factors we propose an information criterion to select the hyperparameters of the prior expandable factor analysis has better false discovery rates and true positive rates than its competitors across diverse simulation settings we apply the proposed approach to a gene expression study of ageing in mice demonstrating superior results relative to four competing methods

the harris kaiser independent cluster rotation as a method for rotation to simple component weights procedures for oblique rotation of factors or principal components typically focus on rotating the pattern matrix such that it becomes optimally simple an important oblique rotation method that does so is harris and kaiser s 1964 independent cluster hkic rotation in principal components analysis a case can be made for interpreting the components on the basis of the component weights rather than on the basis of the pattern so it seems desirable to rotate the components such that the weights rather than the pattern become optimally simple in the present paper it is shown that hkic rotates the components such that both the pattern and the weights matrix become optimally simple in addition it is shown that the pattern resulting from hkic rotation is columnwise proportional to the associated weights matrix which implies that the interpretation of the components does not depend on whether it is based on the pattern or on the component weights matrix it is also shown that the latter result only holds for hkic rotation and slight modifications of it

em algorithms for ml factor analysis the details of em algorithms for maximum likelihood factor analysis are presented for both the exploratory and confirmatory models the algorithm is essentially the same for both cases and involves only simple least squares regression operations the largest matrix inversion required is for aq q symmetric matrix whereq is the matrix of factors the example that is used demonstrates that the likelihood for the factor analysis model may have multiple modes that are not simply rotations of each other such behavior should concern users of maximum likelihood factor analysis and certainly should cast doubt on the general utility of second derivatives of the log likelihood as measures of precision of estimation

fitting full information item factor models and an empirical investigation of bridge sampling abstract based on item response theory bock and aitken introduced a method of item factor analysis termed full information item factor fiif analysis by bartholomew because it uses all distinct item response vectors as data but a limitation of their fitting algorithm is its reliance on fixed point gauss hermite quadrature which can produce appreciable numerical errors especially in high dimension problems the first purpose of this article is to offer more reliable methods by using recent advances in statistical computation specifically we illustrate two ways of implementing monte carlo expectation maximization em algorithm to fit a fiif model using the gibbs sampler to carry out the computation for the e steps we also show how to use bridge sampling to simulate the likelihood ratios for monitoring the convergence of a monte carlo em a strategy that is useful in general simulations demonstrate substantial improvement over bock and aitken s algorithm in recovering known factor loadings in hig

improved approximations via rayleigh s quotient two methods for obtaining excellent results based on one term approximations via rayleigh s quotient are described along with their applications a combination of these methods can be used to obtain accurate resutls for a wide class of problems as well after a few iterations depending on the starting function the stodola vianello method sometimes leads to overly complicated formulae so that symbolic manipulation software may become bogged down

singular perturbational approximations for discrete time balanced systems a new zero order singular perturbational type approximation is developed for model reduction of discrete time linear systems this approximation is also suitable for model reduction of systems which have fast subsystems and which are represented in the internally balanced format subsystem elimination as suggested by moore for continuous time systems does not generalize for the discrete time case and the singular perturbational approximation gives a particular solution to the discrete time internally balanced model reduction problem

on component analyses principal components analysis can be redefined in terms of the regression of observed variables upon component variables two criteria for the adequacy of a component representation in this context are developed and are shown to lead to different component solutions both criteria are generalized to allow weighting the choice of weights determining the scale invariance properties of the resulting solution a theorem is presented giving necessary and sufficient conditions for equivalent component solutions under different choices of weighting applications of the theorem are discussed that involve the components analysis of linearly derived variables and of external variables

independent factor analysis we introduce the independent factor analysis ifa method for recovering independent hidden sources from their observed mixtures ifa generalizes and unifies ordinary factor analysis fa principal component analysis pca and independent component analysis ica and can handle not only square noiseless mixing but also the general case where the number of mixtures differs from the number of sources and the data are noisy ifa is a two step procedure in the first step the source densities mixing matrix and noise covariance are estimated from the observed data by maximum likelihood for this purpose we present an expectation maximization em algorithm which performs unsupervised learning of an associated probabilistic model of the mixing situation each source in our model is described by a mixture of gaussians thus all the probabilistic calculations can be performed analytically in the second step the sources are reconstructed from the observed data by an optimal nonlinear estimator a variational approximation of this algorithm is derived for cases with a large number of sources where the exact algorithm becomes intractable our ifa algorithm reduces to the one for ordinary fa when the sources become gaussian and to an em algorithm for pca in the zero noise limit we derive an additional em algorithm specifically for noiseless ifa this algorithm is shown to be superior to ica since it can learn arbitrary source densities from the data beyond blind separation ifa can be used for modeling multidimensional data by a highly constrained mixture of gaussians and as a tool for nonlinear signal encoding

nonlinear models using dirichlet process mixtures we introduce a new nonlinear model for classification in which we model the joint distribution of response variable y and covariates x non parametrically using dirichlet process mixtures we keep the relationship between y and x linear within each component of the mixture the overall relationship becomes nonlinear if the mixture contains more than one component with different regression coefficients we use simulated data to compare the performance of this new approach to alternative methods such as multinomial logit mnl models decision trees and support vector machines we also evaluate our approach on two classification problems identifying the folding class of protein sequences and detecting parkinson s disease our model can sometimes improve predictive accuracy moreover by grouping observations into sub populations i e mixture components our model can sometimes provide insight into hidden structure in the data

probabilistic principal component analysis principal component analysis pca is a ubiquitous technique for data analysis and processing but one which is not based upon a probability model in this paper we demonstrate how the principal axes of a set of observed data vectors may be determined through maximum likelihood estimation of parameters in a latent variable model closely related to factor analysis we consider the properties of the associated likelihood function giving an em algorithm for estimating the principal subspace iteratively and discuss the advantages conveyed by the definition of a probability density function for pca

directed principal component analysis we consider a problem involving estimation of a high dimensional covariance matrix that is the sum of a diagonal matrix and a low rank matrix and making a decision based on the resulting estimate such problems arise for example in portfolio management where a common approach employs principal component analysis pca to estimate factors used in constructing the low rank term of the covariance matrix the decision problem is typically treated separately with the estimated covariance matrix taken to be an input to an optimization problem we propose directed pca an efficient algorithm that takes the decision objective into account when estimating the covariance matrix directed pca effectively adjusts factors that would be produced by pca so that they better guide the specific decision at hand we demonstrate through computational studies that directed pca yields significant benefit and we prove theoretical results establishing that the degree of improvement over conventional pca can be arbitrarily large

lateral torsional buckling of cantilevered elastically coupled composite strip and i beams the lateral torsional buckling of composite strip and i beams is considered the geometrically exact governing equations are simplified by consistently regarding certain configuration parameters as small the assumption that these parameters are indeed small is equivalent to the assumption that the square of the maximum prebuckling cross sectional rotation due to bending is small compared to unity the analysis takes into account various refinements of previously published results including the vlasov eaect elastic coupling the oaset of the load from the centroid and of course prebuckling deflections the analysis is thereby reduced to a single fourth order diaerential equation and boundary conditions all of which are derivable from a corresponding energy expression from the form of matched asymptotic expansions of the buckling mode when small parameters are ignored altogether a single comparison function is found which gives the correct buckling load to within 1 for a wide range of the warping rigidity using this comparison function a formula for the buckling load as a function of the small parameters of the problem is found and validated with certain exceptions regarding the load oaset parameter the formula provides results which agree quite well with the numerical solution of the exact equations as long as all the small parameters remain small however the load oaset parameter always appears in the governing equations as multiplied by a ratio of stianesses which can become large especially for composite i beams for this case a special treatment is required 2001 elsevier science ltd all rights reserved

divide conquer and coordinate globally coordinated switching linear dynamical system the goal of this work is to learn a parsimonious and informative representation for high dimensional time series conceptually this comprises two distinct yet tightly coupled tasks learning a low dimensional manifold and modeling the dynamical process these two tasks have a complementary relationship as the temporal constraints provide valuable neighborhood information for dimensionality reduction and conversely the low dimensional space allows dynamics to be learned efficiently solving these two tasks simultaneously allows important information to be exchanged mutually if nonlinear models are required to capture the rich complexity of time series then the learning problem becomes harder as the nonlinearities in both tasks are coupled a divide conquer and coordinate method is proposed the solution approximates the nonlinear manifold and dynamics using simple piecewise linear models the interactions and coordinations among the linear models are captured in a graphical model the model structure setup and parameter learning are done using a variational bayesian approach which enables automatic bayesian model structure selection hence solving the problem of overfitting by exploiting the model structure efficient inference and learning algorithms are obtained without oversimplifying the model of the underlying dynamical process evaluation of the proposed framework with competing approaches is conducted in three sets of experiments dimensionality reduction and reconstruction using synthetic time series video synthesis using a dynamic texture database and human motion synthesis classification and tracking on a benchmark data set in all experiments the proposed approach provides superior performance

latent variable structural equation modeling with categorical data abstract structural equation modeling with latent variables is overviewed for situations involving a mixture of dichotomous ordered polytomous and continuous indicators of latent variables special emphasis is placed on categorical variables models in psychometrics econometrics and biometrics are interrelated via a general model due to muthen limited information least squares estimators and full information estimation are discussed an example is estimated with a model for a four wave longitudinal data set where dichotomous responses are related to each other and a set of independent variables via latent variables with a variance component structure

indirect estimation of large conditionally heteroskedastic factor models with an application to the dow 30 stocks we derive indirect estimators of conditionally heteroskedastic factor models in which the volatilities of common and idiosyncratic factors depend on their past unobserved values by calibrating the score of a kalman filter approximation with inequality constraints on the auxiliary model parameters we also propose alternative indirect estimators for large scale models and explain how to apply our procedures to many other dynamic latent variable models we analyse the small sample behaviour of our indirect estimators and several likelihood based procedures through an extensive monte carlo experiment with empirically realistic designs finally we apply our procedures to weekly returns on the dow 30 stocks

improved first order approximation of eigenvalues and eigenvectors a method based on reduced basis approximation concepts is presented for improved first order approximation of eigenvalues and eigenvectors of modified structural dynamic systems the terms of a local approximation based on taylor or matrix power series are used as basis vectors for approximating the perturbed eigenparameters for each eigenmode a reduced eigensystem is generated by using the baseline eigenvector and the first order approximation term as ritz vectors the solution of the reduced eigensystem leads to two possible estimates of each perturbed eigenvalue and eigenvector criteria for selection of the best approximation are presented the zero and first order rayleigh quotient approximation can be directly recovered as special cases of the present method results are presented for approximate dynamic reanalysis of a 25 bar planar truss structure it is shown that high quality approximation of the perturbed eigenparameters can be obtained for moderate perturbations in the stiffness and mass matrices for very large local perturbations of the structure including deletion of some structural members it is shown that the present method yields reasonable quality approximations