the equivalence of the total least squares and minimum norm methods signal processing it is shown that the minimum norm solution is equivalent to the total least squares solution it is noted that two versions of the total least squares solution exist one based on the signal subspace and another based on the noise subspace

the smallest perturbation of a submatrix which lowers the rank and constrained total least squares problems let t left begin array 20 c a b c d end array right be a partitioned rectangular matrix we exhibit perturbations of d of smallest norm which lower the rank of t we apply this result to generalize the total least squares problem sx b when only a rectangular subset of the data s b is considered uncertain

consistent normalized least mean square filtering with noisy data matrix when the ordinary least squares method is applied to the parameter estimation problem with noisy data matrix it is well known that the estimates turn out to be biased while this bias term can be somewhat reduced by the use of models of higher order or by requiring a high signal to noise ratio snr it can never be completely removed consistent estimates can be obtained by means of the instrumental variable method ivm or the total data least squares method tls dls in the adaptive setting for the such problem a variety of least mean squares lms type algorithms have been researched rather than their recursive versions of ivm or tls dls that cost considerable computations motivated by these observations we propose a consistent lms type algorithm for the data least square estimation problem this novel approach is based on the geometry of the mean squared error mse function rendering the step size normalization and the heuristic filtered estimation of the noise variance respectively for fast convergence and robustness to stochastic noise monte carlo simulations of a zero forcing adaptive finite impulse response fir channel equalizer demonstrate the efficacy of our algorithm

an efficient recursive total least squares algorithm for fir adaptive filtering an algorithm for recursively computing the total least squares tls solution to the adaptive filtering problem is described this algorithm requires o n multiplications per iteration to effectively track the n dimensional eigenvector associated with the minimum eigenvalue of an augmented sample covariance matrix it is shown that the recursive least squares rls algorithm generates biased adaptive filter coefficients when the filter input vector contains additive noise the tls solution on the other hand is seen to produce unbiased solutions examples of standard adaptive filtering applications that result in noise being added to the adaptive filter input vector are cited computer simulations comparing the relative performance of rls and recursive tls are described

on the approximability of minimizing nonzero variables or unsatisfied relations in linear systems we investigate the computational complexity of two closely related classes of combinatorial optimization problems for linear systems which arise in various fields such as machine learning operations research and pattern recognition in the first class min ulr one wishes given a possibly infeasible system of linear relations to find a solution that violates as few relations as possible while satisfying all the others in the second class min rvls the linear system is supposed to be feasible and one looks for a solution with as few nonzero variables as possible for both min ulr and min rvls the four basic types of relational operators and are considered while min rvls with equations was mentioned to be np hard in garey and johnson 1979 we established in amaldi 1992 amaldi and kann 1995 that min ulr with equalities and inequalities are np hard even when restricted to homogeneous systems with bipolar coefficients the latter problems have been shown hard to approximate in arora et al 1993 in this paper we determine strong bounds on the approximability of various variants of min rvls and min ulr including constrained ones where the variables are restricted to take binary values or where some relations are mandatory while others are optional the various np hard versions turn out to have different approximability properties depending on the type of relations and the additional constraints but none of them can be approximated within any constant factor unless p np particular attention is devoted to two interesting special cases that occur in discriminant analysis and machine learning in particular we disprove a conjecture of van horn and martinez 1992 regarding the existence of a polynomial time algorithm to design linear classifiers or perceptrons that involve a close to minimum number of features

esprit estimation of signal parameters via rotational invariance techniques an approach to the general problem of signal parameter estimation is described the algorithm differs from its predecessor in that a total least squares rather than a standard least squares criterion is used although discussed in the context of direction of arrival estimation esprit can be applied to a wide variety of problems including accurate detection and estimation of sinusoids in noise it exploits an underlying rotational invariance among signal subspaces induced by an array of sensors with a translational invariance structure the technique when applicable manifests significant performance and computational advantages over previous algorithms such as mem capon s mlm and music

fast branch bound algorithms for optimal feature selection a novel search principle for optimal feature subset selection using the branch bound method is introduced thanks to a simple mechanism for predicting criterion values a considerable amount of time can be saved by avoiding many slow criterion evaluations we propose two implementations of the proposed prediction mechanism that are suitable for use with nonrecursive and recursive criterion forms respectively both algorithms find the optimum usually several times faster than any other known branch bound algorithm as the algorithm computational efficiency is crucial due to the exponential nature of the search problem we also investigate other factors that affect the search performance of all branch bound algorithms using a set of synthetic criteria we show that the speed of the branch bound algorithms strongly depends on the diversity among features feature stability with respect to different subsets and criterion function dependence on feature set size we identify the scenarios where the search is accelerated the most dramatically finish in linear time as well as the worst conditions we verify our conclusions experimentally on three real data sets using traditional probabilistic distance criteria

an algorithm for pole zero system model order estimation in system identification estimates of the unknown system model orders are often required an algorithm for estimating model orders is described that looks at input output data covariance matrix eigenvectors when model orders are overestimated zeros appear in the noise subspace eigenvectors the number of zeros present can be used to estimate model orders

the statistical performance of the music and the minimum norm algorithms in resolving plane waves in noise this paper presents an asymptotic statistical analysis of the null spectra of two eigen assisted methods music 1 and minimum norm 2 for resolving independent closely spaced plane waves in noise particular attention is paid to the average deviation of the null spectra from zero at the true angles of arrival for the plane waves these deviations are expressed as functions of signal to noise ratios number of array elements angular separation of emitters and the number of snapshots in the case of music an approximate expression is derived for the resolution threshold of two plane waves with equal power in noise this result is validated by monte carlo simulations

scaled total least squares fundamentals the standard approaches to solving overdetermined linear systems bx approx c construct minimal corrections to the vector c and or the matrix b such that the corrected system is compatible in ordinary least squares ls the correction is restricted to c while in data least squares dls it is restricted to b in scaled total least squares stls 22 corrections to both c and b are allowed and their relative sizes depend on a real positive parameter gamma stls unifies several formulations since it becomes total least squares tls when gamma 1 and in the limit corresponds to ls when gamma rightarrow 0 and dls when gamma rightarrow infty this paper analyzes a particularly useful formulation of the stls problem the analysis is based on a new assumption that guarantees existence and uniqueness of meaningful stls solutions for all parameters gamma 0 it makes the whole stls theory consistent our theory reveals the necessary and sufficient condition for preserving the smallest singular value of a matrix while appending or deleting a column this condition represents a basic matrix theory result for updating the singular value decomposition as well as the rank one modification of the hermitian eigenproblem the paper allows complex data and the equivalences in the limit of stls with dls and ls are proven for such data it is shown how any linear system bx approx c can be reduced to a minimally dimensioned core system satisfying our assumption consequently our theory and algorithms can be applied to fully general systems the basics of practical algorithms for both the stls and dls problems are indicated for either dense or large sparse systems our assumption and its consequences are compared with earlier approaches

qr based tls and mixed ls tls algorithms with applications to adaptive iir filtering the least squares ls total least squares tls and mixed ls tls approaches are compared as to their properties and performance on several classical filtering problems mixed ls tls is introduced as a qr decomposition based algorithm for unbiased equation error adaptive infinite impulse response iir filtering the algorithm is based on casting adaptive iir filtering into a mixed ls tls framework this formulation is shown to be equivalent to the minimization of the mean square equation error subject to a unit norm constraint on the denominator parameter vector an efficient implementation of the mixed ls tls solution is achieved through the use of back substitution and inverse iteration unbiasedness of the system parameter estimates is established for the mixed ls tls solution in the case of uncorrelated output noise and the algorithm is shown to converge to this solution ls tls and mixed ls tls performance is then compared for the problems of echo cancellation noise reduction and frequency equalization

the constrained total least squares technique and its applications to harmonic superresolution the constrained total least squares ctls method is a natural extension of tls to the case when the noise components of the coefficients are algebraically related the ctls technique is developed and some of its applications to superresolution harmonic analysis are presented the ctls problem is reduced to an unconstrained minimization problem over a small set of variables a perturbation analysis of the ctls solution is derived and from it the root mean square error rmse of the ctls solution which is valid for small noise levels is obtained in closed form the complex version of the newton method is derived and applied to determine the ctls solution it is also shown that the ctls problem is equivalent to a constrained parameter maximum likelihood problem the ctls technique is applied to estimate the frequencies of sinusoids in white noise and the angle of arrival of narrowband wavefronts at a linear uniform array in both cases the ctls method shows superior or similar accuracy to other advanced techniques

a generalization of the eckart young mirsky matrix approximation theorem abstract the eckart young mirsky theorem solves the problem of approximating a matrix by one of lower rank however the approximation generally differs from the original in all its elements in this paper it is shown how to obtain a best approximation of lower rank in which a specified set of columns of the matrix remains fixed the paper concludes with some applications of the generalization

a recursive restricted total least squares algorithm we show that the generalized total least squares gtls problem with a singular noise covariance matrix is equivalent to the restricted total least squares rtls problem and propose a recursive method for its numerical solution the method is based on the generalized inverse iteration the estimation error covariance matrix and the estimated augmented correction are also characterized and computed recursively the algorithm is cheap to compute and is suitable for online implementation simulation results in least squares ls data least squares dls total least squares tls and restricted total least squares rtls noise scenarios show fast convergence of the parameter estimates to their optimal values obtained by corresponding batch algorithms

sensor array signal processing via a procrustes rotations based eigenanalysis of the esprit data pencil the authors propose the procrustes rotation based esprit algorithm pro esprit for estimating characteristic parameters associated with a scene of radiating sources given data derived from two identical sensor arrays the x and y arrays which are displaced relative to one another pro esprit uses the inherent redundancy built into the esprit array system to compensate for deviations from the ideal array model this redundancy is manifested by two fundamental properties of the noiseless x and y data matrices comprising the esprit data pencil a they have the same row space the signal subspace and 2 they have the same column space the source subspace these properties are utilized by invoking two primary tools 1 the solution to the procrustes problem for optimally approximating an invariant subspace rotation and 2 the total least squares method for perturbing each of two estimates of a common subspace in a minimal fashion until the two perturbed spaces are the same

dea based dimensionality reduction for classification problems satisfying strict non satiety assumption abstract this study shows how data envelopment analysis dea can be used to reduce vertical dimensionality of certain data mining databases the study illustrates basic concepts using a real world graduate admissions decision task it is well known that cost sensitive mixed integer programming mip problems are np complete this study shows that heuristic solutions for cost sensitive classification problems can be obtained by solving a simple goal programming problem by that reduces the vertical dimension of the original learning dataset using simulated datasets and a misclassification cost performance metric the performance of proposed goal programming heuristic is compared with the extended dea discriminant analysis mip approach the holdout sample results of our experiments shows that the proposed heuristic approach outperforms the extended dea discriminant analysis mip approach

using simulated annealing to optimize the feature selection problem in marketing applications the feature selection also specification problem is concerned with finding the most influential subset of predictors in predictive modeling from a much larger set of potential predictors that can contain hundreds of predictors the problem belongs to the realm of combinatorial optimization where the objective is to find the subset of variables that optimize the value of some goodness of fit function due to the dimensionality of the problem the feature selection problem belongs to the group of np hard problems most of the available predictors are noisy or redundant and add very little if any to the prediction power of the model using all the predictors in the model often results in strong over fitting and very poor predictions constructing a prediction model by checking out all possible subsets is impractical due to computational volume looking on the contribution of each predictor separately is not accurate because it ignores the inter correlations between predictors as a result no analytic solution is available for the feature selection problem requiring that one resorts to heuristics in this paper we employ the simulated annealing sa approach which is one of the leading stochastic search methods for specifying a large scale linear regression model the sa results are compared to the results of the more common stepwise regression swr approach for model specification the models are applied on realistic data sets in database marketing we also use simulated data sets to investigate what data characteristics make the swr approach equivalent to the supposedly more superior sa approach

a perturbation property of the tls lp method it is shown that the tls lp total least squares linear prediction method and the svd prony method are equivalent to the first order perturbation approximation in practice it means that the two methods yield the same estimation variances when the signal to noise ratio snr is above a threshold

solving feature subset selection problem by a parallel scatter search the aim of this paper is to develop a parallel scatter search metaheuristic for solving the feature subset selection problem in classification given a set of instances characterized by several features the classification problem consists of assigning a class to each instance feature subset selection problem selects a relevant subset of features from the initial set in order to classify future instances we propose two methods for combining solutions in the scatter search metaheuristic these methods provide two sequential algorithms that are compared with a recent genetic algorithm and with a parallelization of the scatter search this parallelization is obtained by running simultaneously the two combination methods parallel scatter search presents better performance than the sequential algorithms

total least squares approach for frequency estimation using linear prediction the resolution of the estimated closely spaced frequencies of the multiple sinusoids degrades as the signal to noise ratio snr of the received signal becomes low this resolution can be improved by using the total least squares tls method in solving the linear prediction lp equation this approach makes use of the singular value decomposition svd of the augmented matrix for low rank approximation to reduce the noise effect from both the observation vector and the lp data matrix simultaneously comparison is made to the principle eigenvector pe method of tufts and kumaresan both on theoretical and experimental grounds the tls algorithm exhibits superior performance over the pe method where low rank approximation is applied to the data matrix only

survey paper structured low rank approximation and its applications fitting data by a bounded complexity linear model is equivalent to low rank approximation of a matrix constructed from the data the data matrix being hankel structured is equivalent to the existence of a linear time invariant system that fits the data and the rank constraint is related to a bound on the model complexity in the special case of fitting by a static model the data matrix and its low rank approximation are unstructured we outline applications in system theory approximate realization model reduction output error and errors in variables identification signal processing harmonic retrieval sum of damped exponentials and finite impulse response modeling and computer algebra approximate common divisor algorithms based on heuristics and local optimization methods are presented generalizations of the low rank approximation problem result from different approximation criteria e g weighted norm and constraints on the data matrix e g nonnegativity related problems are rank minimization and structured pseudospectra

phase noise compensation for ofdm systems we describe a low complexity method for time domain compensation of phase noise in ofdm systems we extend existing methods in several respects first we suggest using the karhunen loeve representation of the phase noise process to estimate the phase noise we then derive an improved data directed choice of basis elements for ls phase noise estimation and present its total least square counterpart problem the proposed method helps overcome one of the major weaknesses of ofdm systems we also generalize the time domain phase noise compensation to the multiuser mimo context finally we present simulation results using both simulated and measured phased noise we quantify the tracking performance in the presence of residual carrier offset

on the total least squares linear prediction method for frequency estimation the total least squares tls linear prediction lp method recently presented by rahman and yu 1987 and the equivalent improved pisarenko s ip method by kumaresan 1986 are reviewed and generalized by the whitening approach the resulting whitened tls lp method yields higher estimation accuracy than the tls lp this simulation was carried out in double precision fortran 77 on vax 8810 the imsl routines were used to perform eigendecompositions compute the polynomial roots and to generate the pseudo gaussian random numbers

structured least squares problems and robust estimators a novel approach is proposed to provide robust and accurate estimates for linear regression problems when both the measurement vector and the coefficient matrix are structured and subject to errors or uncertainty a new analytic formulation is developed in terms of the gradient flow of the residual norm to analyze and provide estimates to the regression the presented analysis enables us to establish theoretical performance guarantees to compare with existing methods and also offers a criterion to choose the regularization parameter autonomously theoretical results and simulations in applications such as blind identification multiple frequency estimation and deconvolution show that the proposed technique outperforms alternative methods in mean squared error for a significant range of signal to noise ratio values

a discrete particle swarm optimization method for feature selection in binary classification problems abstract this paper investigates the feature subset selection problem for the binary classification problem using logistic regression model we developed a modified discrete particle swarm optimization pso algorithm for the feature subset selection problem this approach embodies an adaptive feature selection procedure which dynamically accounts for the relevance and dependence of the features included the feature subset we compare the proposed methodology with the tabu search and scatter search algorithms using publicly available datasets the results show that the proposed discrete pso algorithm is competitive in terms of both classification accuracy and computational performance

a variable selection method based on tabu search for logistic regression models a tabu search method is proposed and analysed for selecting variables that are subsequently used in logistic regression models the aim is to find from among a set of m variables a smaller subset which enables the efficient classification of cases reducing dimensionality has some very well known advantages that are summarized in literature the specific problem consists in finding for a small integer value of p a subset of size p of the original set of variables that yields the greatest percentage of hits in logistic regression the proposed tabu search method performs a deep search in the solution space that alternates between a basic phase that uses simple moves and a diversification phase to explore regions not previously visited testing shows that it obtains significantly better results than the stepwise backward or forward methods used by classic statistical packages some results of applying these methods are presented