benchmarking optimization software with performance profiles we propose performance profiles distribution functions for a performance metric as a tool for benchmarking and comparing optimization software we show that performance profiles combine the best features of other tools for performance evaluation

a fuzzy set preference model for consumer choice abstract a fuzzy set model for predicting consumer choice is developed by combining the vector conjoint model with fuzzy measurement in a new model that requires only ordinal data fuzzy set measurement of consumer preferences is examined by testing alternative set representations and an interactive set definition method a comparison test of predictive validity using identical data from 139 subjects showed large improvements in rank order prediction validity compared to a non fuzzy approach prediction improvements of 83 are attributable to the use of fuzzy set definitions for subject ratings with the first two of six ordered ranking predicted on average

an augmented lagrangian interior point method using directions of negative curvature we describe an efficient implementation of an interior point algorithm for non convex problems that uses directions of negative curvature these directions should ensure convergence to second order kkt points and improve the computational efficiency of the procedure some relevant aspects of the implementation are the strategy to combine a direction of negative curvature and a modified newton direction and the conditions to ensure feasibility of the iterates with respect to the simple bounds the use of multivariate barrier and penalty parameters is also discussed as well as the update rules for these parameters we analyze the convergence of the procedure both the linesearch and the update rule for the barrier parameter behave appropriately as the main goal of the paper is the practical usage of negative curvature a set of numerical results on small test problems is presented based on these results the relevance of using directions of negative curvature is discussed

fast iterative kernel principal component analysis we develop gain adaptation methods that improve convergence of the kernel hebbian algorithm kha for iterative kernel pca kim et al 2005 kha has a scalar gain parameter which is either held constant or decreased according to a predetermined annealing schedule leading to slow convergence we accelerate it by incorporating the reciprocal of the current estimated eigenvalues as part of a gain vector an additional normalization term then allows us to eliminate a tuning parameter in the annealing schedule finally we derive and apply stochastic meta descent smd gain vector adaptation schraudolph 1999 2002 in reproducing kernel hilbert space to further speed up convergence experimental results on kernel pca and spectral clustering of usps digits motion capture and image denoising and image super resolution tasks confirm that our methods converge substantially faster than conventional kha to demonstrate scalability we perform kernel pca on the entire mnist data set

combining and scaling descent and negative curvature directions the aim of this paper is the study of different approaches to combine and scale in an efficient manner descent information for the solution of unconstrained optimization problems we consider the situation in which different directions are available in a given iteration and we wish to analyze how to combine these directions in order to provide a method more efficient and robust than the standard newton approach in particular we will focus on the scaling process that should be carried out before combining the directions we derive some theoretical results regarding the conditions necessary to ensure the convergence of combination procedures following schemes similar to our proposals finally we conduct some computational experiments to compare these proposals with a modified newton s method and other procedures in the literature for the combination of information

quadratic cost flow and the conjugate gradient method by introducing quadratic penalty terms a convex non separable quadratic network program can be reduced to an unconstrained optimization problem whose objective function is a piecewise quadratic and continuously differentiable function a conjugate gradient method is applied to the reduced problem and its convergence is proved the computation exploits the special network data structures originated from the network simplex method this algorithmic framework allows direct extension to multicommodity cost flows some preliminary computational results are presented

general results on the convergence of stochastic algorithms a deterministic approach is proposed for proving the convergence of stochastic algorithms of the most general form under necessary conditions on the input noise and reasonable conditions on the nonnecessarily continuous mean field emphasis is placed on the case where more than one stationary point exists we also use this approach to prove the convergence of a stochastic algorithm with markovian dynamics

a new conjugate gradient method with guaranteed descent and an efficient line search a new nonlinear conjugate gradient method and an associated implementation based on an inexact line search are proposed and analyzed with exact line search our method reduces to a nonlinear version of the hestenes stiefel conjugate gradient scheme for any inexact line search our scheme satisfies the descent condition bf g k sf t bf d k le frac 7 8 bf g k 2 moreover a global convergence result is established when the line search fulfills the wolfe conditions a new line search scheme is developed that is efficient and highly accurate efficiency is achieved by exploiting properties of linear interpolants in a neighborhood of a local minimizer high accuracy is achieved by using a convergence criterion which we call the approximate wolfe conditions obtained by replacing the sufficient decrease criterion in the wolfe conditions with an approximation that can be evaluated with greater precision in a neighborhood of a local minimum than the usual sufficient decrease criterion numerical comparisons are given with both l bfgs and conjugate gradient methods using the unconstrained optimization problems in the cute library

fast principal component extraction by a weighted information criterion principal component analysis pca is an essential technique in data compression and feature extraction and there has been much interest in developing fast pica algorithms on the basis of the concepts of both weighted subspace and information maximization this paper proposes a weighted information criterion winc for searching the optimal solution of a linear neural network we analytically show that the optimum weights globally asymptotically converge to the principal eigenvectors of a stationary vector stochastic process we establish a dependent relation of choosing the weighting matrix on statistics of the input process through the analysis of stability of the equilibrium of the proposed criterion therefore we are able to reveal the constraint on the choice of a weighting matrix we develop two adaptive algorithms based on the winc for extracting in parallel multiple principal components both algorithms are able to provide adaptive step size which leads to a significant improvement in the learning performance furthermore the recursive least squares rls version of winc algorithms has a low computational complexity o np where n is the input vector dimension and p is the number of desired principal components in fact the winc algorithm corresponds to a three layer linear neural network model capable of performing in parallel the extraction of multiple principal components the winc algorithm also generalizes some well known pca psa algorithms just by adjusting the corresponding parameters since the weighting matrix does not require an accurate value it facilitates the system design of the winc algorithm for practical applications the accuracy and speed advantages of the winc algorithm are verified through simulations

a decoupled first second order steps technique for nonconvex nonlinear unconstrained optimization with improved complexity bounds in order to be provably convergent towards a second order stationary point optimization methods applied to nonconvex problems must necessarily exploit both first and second order information however as revealed by recent complexity analyses of some of these methods the overall effort to reach second order points is significantly larger when compared to the one of approaching first order ones on the other hand there are other algorithmic schemes initially designed with first order convergence in mind that do not appear to maintain the same first order performance when modified to take second order information into account in this paper we propose a technique that separately computes first and second order steps and that globally converges to second order stationary points it consists in better connecting the steps to be taken and the stationarity criteria potentially guaranteeing larger steps and decreases in the objective our approach is shown to lead to an improvement of the corresponding complexity bound with respect to the first order optimality tolerance while having a positive impact on the practical behavior although the applicability of our ideas is wider we focus the presentation on trust region methods with and without derivatives and motivate in both cases the interest of our strategy

a distributed framework for dimensionality reduction and denoising with the recent upsurge in the amount of data transmission and use in networks a lot of emphasis is placed on designing efficient methods to extract useful information by reducing the size of large data sets a major approach in data dimensionality reduction relies on the estimation of the principal data covariance eigenspace in this paper a distributed algorithmic framework is put forth for finding the principal eigenspace of spatially scattered sensor data toward this end the standard principal component analysis framework is reformulated as a separable constrained minimization problem which is solved by utilizing coordinate descent techniques combined with the alternating direction method of multipliers computationally simple local updating recursions are obtained that involve only single hop intersensor communications the proposed distributed algorithm is shown to be robust even in the presence of inter sensor communication noise and converge to the principal eigenspace when inter sensor links are ideal the proposed distributed framework is used in a data denoising setting while extensive numerical tests using both synthetic and real data demonstrate a fast convergence rate better steady state performance of the novel algorithm over existing alternatives and communication noise resilience

a cross associative neural network for svd of non squared data matrix in signal processing this paper proposes a cross associative neural network cann for singular value decomposition svd of a non squared data matrix in signal processing in order to improve the convergence speed and avoid the potential instability of the deterministic networks associated with the cross correlation neural network models we study the global asymptotic stability of the network for tracking all the singular components and show that the selection of its learning rate in the iterative algorithm is independent of the singular value distribution of a non squared matrix the performances of cann are shown via simulations

least squares type algorithm for adaptive implementation of pisarenko s harmonic retrieval method pisarenko s harmonic retrieval method involves determining the minimum eigenvalue and the corresponding eigenvector of the covariance matrix of the observed random process recently thompson 9 suggested a constrained gradient search procedure for obtaining an adaptive version of pisarenko s method and his simulations have verified that the frequency estimates provided by his procedure were unbiased however the main cost of this technique was that the initial convergence rate could be very slow for certain poor initial conditions restating the constrained minimization as an unconstrained nonlinear problem we derived an alternative gauss newton type recursive algorithm which also used the second derivative matrix or hessian this algorithm may also be viewed as an approximate least squares algorithm simulations have been performed to compare this algorithm to a slight variant of thompson s original algorithm the most important conclusions are that the least squares type algorithm has faster convergence in the beginning while its convergence rate close to the true parameters depends on the signal to noise ratio of the input signal the approximate least squares algorithm resolves the sinusoids much faster than the gradient version

nonconvex optimization using negative curvature within a modified linesearch this paper describes a new algorithm for the solution of nonconvex unconstrained optimization problems with the property of converging to points satisfying second order necessary optimality conditions the algorithm is based on a procedure which from two descent directions a newton type direction and a direction of negative curvature selects in each iteration the linesearch model best adapted to the properties of these directions the paper also presents results of numerical experiments that illustrate its practical efficiency

measurement of membership functions and their acquisition abstract three basic views of the representation of membership functions are reviewed together with fundamental measurement of linguistic terms of linguistic variables the conclusion is that such measurements are either on an ordinal or an interval scale based on whether the appropriate axioms are validated by the empirical data with an allowance for stochastic variation the conjoint measurement is introduced for the case of multi dimensional linguistic variables whose linguistic terms are compositions of two or finitely many more component linguistic terms of distinct independent linguistic variables it is shown that any composition of distinct independent component linguistic terms of component linguistic variables by any t norm or s norm or any finite convext linear combination preserves the monotonic weak order property of the components in the composite once the scale properties of the measurement values of particular terms are validated composition procedures may be applied to the experimental data to obtain compound membership functions of fuzzy sets induced by meaningful representations of compositions of linguistic terms of linguistic variables finally four methods of membership acquisition and construction are reviewed from the perspective of fundamental measurement

convergence of learning algorithms with constant learning rates the behavior of neural network learning algorithms with a small constant learning rate epsilon in stationary random input environments is investigated it is rigorously established that the sequence of weight estimates can be approximated by a certain ordinary differential equation in the sense of weak convergence of random processes as epsilon tends to zero as applications backpropagation in feedforward architectures and some feature extraction algorithms are studied in more detail

algorithms for nonnegative independent component analysis we consider the task of solving the independent component analysis ica problem x as given observations x with a constraint of nonnegativity of the source random vector s we refer to this as nonnegative independent component analysis and we consider methods for solving this task for independent sources with nonzero probability density function pdf p s down to s 0 it is sufficient to find the orthonormal rotation y wz of prewhitened sources z vx which minimizes the mean squared error of the reconstruction of z from the rectified version y sup of y we suggest some algorithms which perform this both based on a nonlinear principal component analysis pca approach and on a geodesic search method driven by differential geometry considerations we demonstrate the operation of these algorithms on an image separation problem which shows in particular the fast convergence of the rotation and geodesic methods and apply the approach to a musical audio analysis task

asymptotic performance analysis of subspace adaptive algorithms introduced in the neural network literature in the neural network literature many algorithms have been proposed for estimating the eigenstructure of covariance matrices we first show that many of these algorithms when presented in a common framework show great similitudes with the gradient like stochastic algorithms usually encountered in the signal processing literature we derive the asymptotic distribution of these different recursive subspace estimators a closed form expression of the covariances in distribution of eigenvectors and associated projection matrix estimators are given and analyzed in particular closed form expressions of the mean square error of these estimators are given it is found that these covariance matrices have a structure very similar to those describing batch estimation techniques the accuracy of our asymptotic analysis is checked by numerical simulations and it is found to be valid not only for a small step size but in a very large domain finally convergence speed and deviation from orthonormality of the different algorithms are compared and several tradeoffs are analyzed

global convergence of oja s subspace algorithm for principal component extraction oja s principal subspace algorithm is a well known and powerful technique for learning and tracking principal information in time series a thorough investigation of the convergence property of oja s algorithm is undertaken in this paper the asymptotic convergence rates of the algorithm is discovered the dependence of the algorithm on its initial weight matrix and the singularity of the data covariance matrix is comprehensively addressed

global analysis of oja s flow for neural networks a detailed study of oja s learning equation in neural networks is undertaken in this paper not only are such fundamental issues as existence uniqueness and representation of solutions completely resolved but also the convergence issue is resolved it is shown that the solution of oja s equation is exponentially convergent to an equilibrium from any initial value moreover the necessary and sufficient conditions are given on the initial value for the solution to converge to a dominant eigenspace of the associated autocorrelation matrix as a by product this result confirms one of oja s conjectures that the solution converges to the principal eigenspace from almost all initial values some other characteristics of the limiting solution are also revealed these facilitate the determination of the limiting solution in advance using only the initial information two examples are analyzed demonstrating the explicit dependence of the limiting solution on the initial value in another respect it is found that oja s equation is the gradient flow of generalized rayleigh quotients on a stiefel manifold

performance analysis of adaptive eigenanalysis algorithms we present a rigorous analysis of several popular forms of short memory adaptive eigenanalysis algorithms using a stochastic averaging method a first order analysis shows that the algorithms do not have any equilibrium points despite published claims to the contrary through averaging analysis we show that they hover around an appropriate eigenvector a second order analysis is also given without the gaussian noise assumption and our results greatly outperform an earlier approximation in the literature the second order analysis has been of much interest in the offline study but in the dynamic adaptive case is uncommon

a note on linear function approximation using random projections linear function approximations based on random projections are proposed and justified for a class of fixed point and minimization problems

approximate solution of the trust region problem by minimization over two dimensional subspaces the trust region problem minimization of a quadratic function subject to a spherical trust region constraint occurs in many optimization algorithms in a previous paper the authors introduced an inexpensive approximate solution technique for this problem that involves the solution of a two dimensional trust region problem they showed that using this approximation in an unconstrained optimization algorithm leads to the same theoretical global and local convergence properties as are obtained using the exact solution to the trust region problem this paper reports computational results showing that the two dimensional minimization approach gives nearly optimal reductions in then dimension quadratic model over a wide range of test cases we also show that there is very little difference in efficiency and reliability between using the approximate or exact trust region step in solving standard test problems for unconstrained optimization these results may encourage the application of similar approximate trust region techniques in other contexts

on stochastic approximation of the eigenvectors and eigenvalues of the expectation of a random matrix abstract in applications of signal processing and pattern recognition eigenvectors and eigenvalues of the statistical mean of a random matrix sequence are needed iterative methods are suggested and analyzed in which no sample moments are used convergence is shown by stochastic approximation theory

performance analysis of an adaptive algorithm for tracking dominant subspaces this paper provides a performance analysis of a least mean square lms dominant invariant subspace algorithm based on an unconstrained minimization problem this algorithm is a stochastic gradient algorithm driving the columns of a matrix w to an orthonormal basis of a dominant invariant subspace of a correlation matrix we consider the stochastic algorithm governing the evolution of ww sup h to the projection matrix onto this dominant invariant subspace and study its asymptotic distribution a closed form expression of its asymptotic covariance is given in the case of independent observations and is further analyzed to provide some insights into the behavior of this lms type algorithm in particular it is shown that even though the algorithm does not constrain w to have orthonormal columns there is deviation from orthonormality of the first order we also give a closed form expression of the asymptotic covariance of the doas estimated by the music algorithm applied to the adaptive estimate of the projector it is found that the asymptotic distributions have a structure that is very similar to those describing batch estimation techniques because both algorithms are obtained from the minimization of the same criterion finally the accuracy of the asymptotic analysis is checked by numerical simulations and is found to be valid not only for a small step size but in a very large domain

asymptotic distributions associated to oja s learning equation for neural networks we perform a complete asymptotic performance analysis of the stochastic approximation algorithm denoted subspace network learning algorithm derived from oja s learning equation in the case where the learning rate is constant and a large number of patterns is available this algorithm drives the connection weight matrix w to an orthonormal basis of a dominant invariant subspace of a covariance matrix our approach consists in associating to this algorithm a second stochastic approximation algorithm that governs the evolution of ww sup t to the projection matrix onto this dominant invariant subspace then using a general result of gaussian approximation theory we derive the asymptotic distribution of the estimated projection matrix closed form expressions of the asymptotic covariance of the projection matrix estimated by the snl algorithm and by the smoothed snl algorithm that we introduce are given in the case of independent or correlated learning patterns and are further analyzed it is found that the structures of these asymptotic covariance matrices are similar to those describing batch estimation techniques the accuracy or our asymptotic analysis is checked by numerical simulations and it is found to be valid not only for a small learning rate but in a very large domain finally improvements brought by our smoothed snl algorithm are shown such as the learning speed misadjustment tradeoff and the deviation from orthonormality

fuzzy route choice model for traffic assignment abstract in order to represent traffic assignment on a road network we propose a new route choice model taking account of the imprecisions and the uncertainties lying in the dynamic choice process this model makes possible a more accurate description of the process than those deterministic or stochastic used in the literature we assume that drivers choose a path all the more than it is foreseen to have a lesser cost the predicted cost for each path is modelled by a fuzzy subset which can represent imprecision on network knowledge e g length of links as well as uncertainty on traffic conditions e g congested or uncongested network incident the costs of all possible paths are compared and result in an attractiveness degree for each path various comparison indices can be used to represent the different possible natures of drivers when making their decisions pessimistic optimistic risk taking risk averting the fuzzy choice model was compared with the logit model a widely used stochastic discrete choice model and has been proved to be able to find the same results the effect of advanced travellers information systems atis on drivers is modeled as a modification of the imprecision or the uncertainty of the predicted cost of a route

combining search directions using gradient flows the efficient combination of directions is a significant problem in line search methods that either use negative curvature or wish to include additional information such as the gradient or different approximations to the newton direction

projection approximation subspace tracking subspace estimation plays an important role in a variety of modern signal processing applications we present a new approach for tracking the signal subspace recursively it is based on a novel interpretation of the signal subspace as the solution of a projection like unconstrained minimization problem we show that recursive least squares techniques can be applied to solve this problem by making an appropriate projection approximation the resulting algorithms have a computational complexity of o nr where n is the input vector dimension and r is the number of desired eigencomponents simulation results demonstrate that the tracking capability of these algorithms is similar to and in some cases more robust than the computationally expensive batch eigenvalue decomposition relations of the new algorithms to other subspace tracking methods and numerical issues are also discussed