adaptive filter performance with nonlinearities in the correlation multiplier it is well known that the correlation multiplier used to form the update signal in adaptive transversal filters need not be a true multiplier by using an approximate multiplier such as a sign bit only multiplier implementation can be made less costly in this paper we derive expressions for the convergence rate and misalignment noise of adaptive transversal filters with arbitrary correlation multiplier non linearities the analysis shows any such nonlinearities to always harm performance and quantifies the penalty suffered sign bit only multiplication suffers a significant penalty and power of two multiplication is suggested as a better alternative its performance is almost equal to that of a true multiplier and in a digital implementation at least it is only slightly more difficult to realize than a sign bit only multiplier the theory is developed in a way emphasizing physical understanding and has been verified to be in good agreement with simulation

a simplicial approach to the determination of an integer point of a simplex we propose an algorithm to find an integer point of a simplex the algorithm is based on an integer labeling rule and a triangulation of the space starting at an integer point the algorithm leaves it along one of n rays and generates a sequence of adjacent simplices of varying dimension within a finite number of iterations the algorithm either yields an integer point of the simplex or proves that no such point exists

the 2 n 1 2 ray algorithm a new simplicial algorithm to compute economic equilibria a new variable dimension simplicial restart algorithm is introduced to compute economic equilibria the number of rays along which the algorithm can leave the starting point differs from the thusfar known algorithms more precisely the new algorithm has one ray to each of the 2n 1 2 faces of then dimensional price simplex whereas the existing algorithms haven 1 rays either to each facet or to each vertex of the unit simplex the path of points followed by the algorithm can be interpreted as a globally and universally convergent price adjustment process the process is also economically meaningful and therefore it is a good alternative for the well known walras tatonnement process computational results show that the algorithm is competitive with the most efficient simplicial algorithms developed thusfar

asymptotic behaviour of a class of stochastic approximation procedures letb be a real banach space considering stochastic approximation procedures of robbins monro and kiefer wolfowitz type one is led to recursion formulas of the form u k 1 u k k 1 gamma k u k k rho v k k eta t k and positive real numbers uk vk tkb valued random variables k random linear operatorsb b where in addition to some further assumptions k almost surely andtk t say almost surely letting y k n k 1 n n n be an array ofb valued random variables such that the partial sums sum limits k 1 m v k and sum limits k 1 m y k n do not differ too much uniformly inm 1 n we shall investigate the distance of u m m 1 n and u n m m 1 n where u k 1 left n right u k left n right k 1 gamma u k left n right k rho y k n k eta t fork 1 n 1 the proofs are based on a deterministic argument that enables us to present a unified approach for proving central limit theorems weak and almost sure invariance principles and variants of the bounded and the functional law of the iterated logarithm for the sequence un

variable dimension algorithms basic theory interpretations and extensions of some existing methods in this paper we establish a basic theory for variable dimension algorithms which were originally developed for computing fixed points by van der laan and talman we introduce a new concept primal dual pair of subdivided manifolds and by utilizing it we propose a basic model which will serve as a foundation for constructing a wide class of variable dimension algorithms our basic model furnishes interpretations to several existing methods lemke s algorithm for the linear complementarity problem its extension to the nonlinear complementarity problem a variable dimension algorithm on conical subdivisions and merrill s algorithm we shall present a method for solving systems of equations as an application of the second method and an efficient implementation of the fourth method to which our interpretation leads us a method for constructing triangulations with an arbitrary refinement factor of mesh size is also proposed

locally optimum adaptive signal processing algorithms we propose a new analytic method for comparing constant gain adaptive signal processing algorithms specifically estimates of the convergence speed of the algorithms allow for the definition of a local measure of performance called the efficacy that can be theoretically evaluated by definition the efficacy is consistent with the fair comparison techniques used in signal processing applications using the efficacy as a performance measure we prove that the lms newton algorithm is optimum and is thus the fastest algorithm within a very rich algorithmic class furthermore we prove that regular lms is better than any of its variants that apply the same nonlinear transformation on the elements of the regression vector such as signed regressor quantized regressor etc for an important class of input signals simulations support all our theoretical conclusions

a variable rate refining triangulation a new variable rate refining subdivisionb4 for the solution of equations with piecewise linear homotopies and restart homotopies is described the essential virtue of the subdivisionb4 is that it offers vast latitude to the user in particular the parameters ofb4 can be set so one restriction ofb4 is thatj3 triangulation and another restriction is the octahedral subdivision more generally full flexibility is available in the placement of local focal points and perhaps most importantly b4 permits the new capability of refining coordinates at variable andindependent rates

discrete time stochastic adaptive control this paper establishes global convergence of a stochastic adaptive control algorithm for discrete time linear systems it is shown that with probability one the algorithm will ensure the system inputs and outputs are sample mean square bounded and the conditional mean square output tracking error achieves its global minimum possible value for linear feedback control thus asymptotically the adaptive control algorithm achieves the same performance as could be achieved if the system parameters were known

adaptive system optimization using random directions stochastic approximation we present new algorithms for simulation optimization using random directions stochastic approximation rdsa these include first order gradient as well as second order newton schemes we incorporate both continuous valued as well as discrete valued perturbations into both types of algorithms the former are chosen to be independent and identically distributed i i d symmetric uniformly distributed random variables r v while the latter are i i d asymmetric bernoulli r v s our newton algorithm with a novel hessian estimation scheme requires n dimensional perturbations and three loss measurements per iteration whereas the simultaneous perturbation newton search algorithm of 1 requires 2n dimensional perturbations and four loss measurements per iteration we prove the asymptotic unbiasedness of both gradient and hessian estimates and asymptotic strong convergence for both first order and second order schemes we also provide asymptotic normality results which in particular establish that the asymmetric bernoulli variant of newton rdsa method is better than 2spsa of 1 numerical experiments are used to validate the theoretical results

the complex lms adaptive algorithm transient weight mean and covariance with applications to the ale the transient and steady state mean and covariance of the complex valued lms adaptive element weights are investigated when the inputs are samples from circularly normal processes it is shown that the data covariance diagonalizing matrix also diagonalizes the weight covariance matrix this result permits describing the complete transient behavior of the adaptive line enhancer ale weight covariance matrix in closed form for the case of multiple equal power narrow band statistically independent orthogonal rayleigh fading sinusoids in broad band noise based on these transient covariance results it is shown that for any stage of adaptation there exists an optimum ale gain constant that minimizes the excess mean squared prediction error this result is of particular significance when processing time limited data

on error saturation nonlinearities in lms adaptation the effect of a saturation type error nonlinearity in the weight update equation in least mean squares lms adaptation is investigated for a white gaussian data model nonlinear difference equations are derived for the eight first and second moments which include the effect of an error function erf saturation type nonlinearity on the error sequence driving the algorithm a nonlinear difference equation for the mean norm is explicitly solved using a differential equation approximation and integration by quadratures the steady state second moment weight behavior is evaluated exactly for the erf nonlinearity using the above results the tradeoff between the extent of error saturation steady state excess mean square error and rate of algorithm convergence is studied the tradeoff shows that 1 starting with a sign detector the convergence rate is increased by nearly a factor of two for each additional bit and 2 as the number of bits is increased further the additional bit by very little in convergence speed asymptotically approaching the behavior of the linear algorithm

newton based stochastic optimization using q gaussian smoothed functional algorithms we present the first q gaussian smoothed functional sf estimator of the hessian and the first newton based stochastic optimization algorithm that estimates both the hessian and the gradient of the objective function using q gaussian perturbations our algorithm requires only two system simulations regardless of the parameter dimension and estimates both the gradient and the hessian at each update epoch using these we also present a proof of convergence of the proposed algorithm in a related recent work ghoshdastidar dukkipati bhatnagar 2014 we presented gradient sf algorithms based on the q gaussian perturbations our work extends prior work on sf algorithms by generalizing the class of perturbation distributions as most distributions reported in the literature for which sf algorithms are known to work turn out to be special cases of the q gaussian distribution besides studying the convergence properties of our algorithm analytically we also show the results of numerical simulations on a model of a queuing network that illustrate the significance of the proposed method in particular we observe that our algorithm performs better in most cases over a wide range of q values in comparison to newton sf algorithms with the gaussian and cauchy perturbations as well as the gradient q gaussian sf algorithms

a method for discrete stochastic optimization this paper addresses the problem of optimizing a function over a finite or countably infinite set of alternatives in situations where this objective function cannot be evaluated exactly but has to be estimated or measured a special focus is on situations where simulation is used to evaluate the objective function we present two versions of a new iterative method for solving such discrete stochastic optimization problems in each iteration of the proposed method a neighbor of the current alternative is selected and estimates of the objective function evaluated at the current and neighboring alternatives are compared the alternative that has a better observed function value becomes the next current alternative we show how one version of the proposed method can be used to solve discrete optimization problems where the objective function is evaluated using transient or steady state simulation and we show how the other version can be applied to solve a special class of discrete stochastic optimizati

information and asymptotic efficiency in parametric nonparametric models asymptotic lower bounds for estimation of the parameters of models with both parametric and nonparametric components are given in the form of representation theorems for regular estimates and asymptotic minimax bounds the methods used involve i the notion of a hellinger differentiable root density where part of the differentiation is with respect to the nonparametric part of the model to obtain appropriate scores and ii calculation of the effective score for the real or vector finite dimensional parameter of interest as that component of the score function orthogonal to all nuisance parameter scores perhaps infinite dimensional the resulting asymptotic information for estimation of the parametric component of the model is just 4 times the squared l 2 norm of the effective score a corollary of these results is a simple necessary condition for adaptive estimation adaptation is possible only if the scores for the parameter of interest are orthogonal to the scores for the nuisance function or nonparametric part of the model examples considered include the one sample location model with and without symmetry mixture models the two sample shift model and cox s proportional hazards model

model free control of nonlinear stochastic systems with discrete time measurements consider the problem of developing a controller for general nonlinear and stochastic systems where the equations governing the system are unknown using discrete time measurement this paper presents an approach for estimating a controller without building or assuming a model for the system such an approach has potential advantages in accommodating complex systems with possibly time varying dynamics the controller is constructed through use of a function approximator such as a neural network or polynomial this paper considers the use of the simultaneous perturbation stochastic approximation algorithm which requires only system measurements a convergence result for stochastic approximation algorithms with time varying objective functions and feedback is established it is shown that this algorithm can greatly enhance the efficiency over more standard stochastic approximation algorithms based on finite difference gradient approximations

a simplicial algorithm for stationary point problems on polytopes a simplicial variable dimension restart algorithm for the stationary point problem or variational inequality problem on a polytope is proposed given a polytope c in rn and a continuous function f c rn find a point x in c such that fx x fx x for any point x in c starting from an arbitrary point v in c the algorithm generates a piecewise linear path of points in c this path is followed by alternating linear programming pivot steps to follow a linear piece of the path and replacement steps in a simplicial subdivision of c within a finite number of function evaluations and linear programming pivot steps the algorithm finds an approximate stationary point the algorithm leaves the starting point v along a ray pointing to one of the vertices w of c the vertex w is obtained from the optimum solution of the linear programming problem maximize fv x subject to x c

computing an integer point of a simplex with an arbitrary starting homotopy like simplicial algorithm abstract an arbitrary starting homotopy like simplicial algorithm is developed for computing an integer point of an n dimensional simplex the algorithm is derived from the use of an integer labeling rule and a triangulation of r n 0 1 and consists of two interchanging phases one phase of the algorithm constitutes a homotopy simplicial algorithm which generates n 1 dimensional simplices in r n 0 1 and the other phase of the algorithm constitutes a pivoting procedure which generates n dimensional simplices in either r n 0 or r n 1 the algorithm varies from one phase to the other when the matrix defining the simplex is in the so called canonical form starting at an arbitrary integer point in r n 0 the algorithm within a finite number of iterations either yields an integer point of the simplex or proves that no such point exists

speech enhancement in discontinuous transmission systems using the constrained stability least mean squares algorithm in this paper a novel constrained stability least mean squares lms algorithm for filtering speech sounds is proposed in the adaptive noise cancellation anc problem it is based on the minimization of the squared euclidean norm of the weight vector change under a stability constraint over the a posteriori estimation errors to this purpose the lagrangian methodology has been used in order to propose a nonlinear adaptation in terms of the product of differential input and error convergence analysis is also studied in terms of the evolution of the natural modes to the optimal wiener hopf solution so that the stability performance depends exclusively on the adaptation parameter and the eigenvalues of the difference matrix r1 the algorithm shows superior performance over the referenced algorithms in the anc problem of speech discontinuous transmission systems which are characterized by rapid transitions of the desired signal the experimental analysis carried out on the aurora 3 speech databases provides an extensive performance evaluation together with an exhaustive comparison to the standard lms algorithms i e the normalized lms nlms and other recently reported lms algorithms such as the modified nlms the error nonlinearity lms or the normalized data nonlinearity lms adaptation

optimization of the transient and steady state behavior of discrete event systems we present a general framework for applying simulation to optimize the behavior of discrete event systems our approach involves modeling the discrete event system under study as a general state space markov chain whose distribution depends on the decision parameters we then show how simulation and the likelihood ratio method can be used to evaluate the performance measure of interest and its gradient and we present conditions that guarantee that the robbins monro stochastic approximation algorithm will converge almost surely to the optimal values of the decision parameters both transient and steady state performance measures are considered for steady state performance measures we consider both the case when the markov chain of interest is regenerative in the standard sense as well as the case when this markov chain is harris recurrent and thereby regenerative in a wider sense

invariance principles for martingales and sums of independent random variables on donne des ameliorations des theoremes d approximation de strassen dans le cas de margingales

efficient implementation of second order stochastic approximation algorithms in high dimensional problems stochastic approximation sa algorithms have been widely applied in minimization problems when the loss functions and or the gradient information are only accessible through noisy evaluations stochastic gradient sg descent a first order algorithm and a workhorse of much machine learning is perhaps the most famous form of sa among all sa algorithms the second order simultaneous perturbation stochastic approximation 2spsa and the second order stochastic gradient 2sg are particularly efficient in handling high dimensional problems covering both gradient free and gradient based scenarios however due to the necessary matrix operations the per iteration floating point operations flops cost of the standard 2spsa 2sg is o p 3 where p is the dimension of the underlying parameter note that the o p 3 flops cost is distinct from the classical spsa based per iteration o 1 cost in terms of the number of noisy function evaluations in this work we propose a technique to efficiently implement the 2spsa 2sg algorithms via the symmetric indefinite matrix factorization and show that the flops cost is reduced from o p 3 to o p 2 the formal almost sure convergence and rate of convergence for the newly proposed approach are directly inherited from the standard 2spsa 2sg the improvement in efficiency and numerical stability is demonstrated in two numerical studies

implementation of a continuation method for normal maps this paper presents an implementation of a nonsmooth continuation method of which the idea was originally put forward by alexander et al we show how the method can be computationally implemented and present numerical results for variational inequality problems in up to 96 variables

control theoretic design of the lms and the sign algorithms in nonstationary environments the feedback structure of the lms least mean squared algorithm proposed by b widrow et al proc ieee vol 64 p 1151 62 aug 1976 is reexamined from a control system design viewpoint the minimization of the misadjustments due to gradient noise and lag can then be recast as the disturbance rejection and tracking problems in control a frequency domain approach to the latter problems that has the advantages of transparency ease of computation and generality compared with the time domain approach previously used is presented with the same set of assumptions of white input and a markovian plant it is shown that the optimum step size obtained by the present approach is identical to that obtained by widrow applying the same approach the optimum step size of a simplified version of the lms algorithm the sign algorithm is derived for the case when the plant is slowly varying and the input signals are gaussian

on error saturation nonlinearities for lms adaptation in impulsive noise this correspondence extends the analytic results in n j bershad ldquoon error nonlinearities in lms adaptation rdquo ieee transactions on acoustics speech and signal processing vol assp 36 no 4 pp 440 452 april 1988 to least mean square lms adaptation in impulsive observation noise a scalar recursion for the weight misadjustment is derived for the white input data case monte carlo simulations verify the accuracy of the theoretical model the theoretical recursion is then used to study the effects of the impulse noise on algorithm convergence speed and steady state weight misadjustment for a wide variety of parameter values

a recursive least m estimate algorithm for robust adaptive filtering in impulsive noise fast algorithm and convergence performance analysis this paper studies the problem of robust adaptive filtering in impulsive noise environment using a recursive least m estimate algorithm rlm the rlm algorithm minimizes a robust m estimator based cost function instead of the conventional mean square error function mse previous work has showed that the rlm algorithm offers improved robustness to impulses over conventional recursive least squares rls algorithm in this paper the mean and mean square convergence behaviors of the rlm algorithm under the contaminated gaussian impulsive noise model is analyzed a lattice structure based fast rlm algorithm called the huber prior error feedback least squares lattice h pef lsl algorithm is derived part of the h pef lsl algorithm was presented in icassp 2001 it has an order o n arithmetic complexity where n is the length of the adaptive filter and can be viewed as a fast implementation of the rlm algorithm based on the modified huber m estimate function and the conventional pef lsl adaptive filtering algorithm simulation results show that the transversal rlm and the h pef lsl algorithms have better performance than the conventional rls and other rls like robust adaptive algorithms tested when the desired and input signals are corrupted by impulsive noise furthermore the theoretical and simulation results on the convergence behaviors agree very well with each other

the octahedral algorithm a new simplicial fixed point algorithm a new variable dimension simplicial algorithm for the computation of solutions of systems of nonlinear equations or the computation of fixed points is presented it uses the restrart technique of merrill to improve the accuracy of the solution the algorithm is shown to converge quadratically under certain conditions the algorithm should be efficient and relatively easy to implement

on the computation of fixed points in the product space of unit simplices and an application to noncooperative n person games in this paper an algorithm based on the principle of simplicial approximation is introduced to compute fixed points of upper semicontinuous point to set mappings from the product space s of unit simplices into itself the algorithm is a modification of an algorithm introduced in an earlier paper the main feature is that it starts with an arbitrary chosen point in s and that the triangulation of s depends on the starting point moreover the algorithm can terminate with a non full dimensional subsimplex yielding a good approximation an application is given for non cooperative n person games where s is the strategy space some computational experiences are given

a nonlinear analytical model for the quantized lms algorithm the power of two step size case presents a study of the quantization effects in the finite precision lms algorithm with power of two step sizes deterministic nonlinear recursions are presented for the mean and second moment matrix of the weight vector about the wiener weight for white gaussian data models and small algorithm step size spl mu the numerical solutions of these recursions are shown to agree very closely with the monte carlo simulations during all phases of the adaptation process design examples demonstrate the selection of the number of quantizer bits and the adaptation step size spl mu to yield a desired transient behavior and cancellation performance the results obtained indicate that previous models are too conservative in predicting the converged mse for a given number of bits

a unified approach to the implementation of several restart fixed point algorithms and a new variable dimension algorithm we present a unified description of a class of restart fixed point algorithms including merrill s method and several variable dimension algorithms for their implementation on computers based on this description we show that some pivot saving techniques originally developed for the homotopy methods can be applied to the class we also propose a new variable dimension algorithm having 3 n 1 rays along which we can move toward a solution some numerical comparisons of the simplicial restart algorithms merrill s method the 2n method the octahedral method and the new one support that the latter two methods are more efficient than the others

adaptive threshold nonlinear algorithm for adaptive filters with robustness against impulse noise a new adaptation algorithm for adaptive filters is proposed introducing adaptive threshold in the nonlinear correlation function the adaptive threshold controlled by the long term average of the error signal power makes the filtering system highly robust against impulse noise which is demonstrated by the results of simulation and theoretical calculation

a continuous deformation algorithm on the product space of unit simplices a continuous deformation algorithm is introduced on s 1 where s denotes the product space of unit simplices with arbitrary grid refinement between two subsequent levels the set s 1 is triangulated in such a way that for each m m 1 2 s m is triangulated by the so called v triangulation the algorithm starts by applying a variable dimension algorithm on s until an approximating simplex has been found on level 1 then the algorithm follows a path of approximating simplices in s 1 starting on level 1 until a certain level or a certain accuracy of a solution of the underlying problem has been reached if the algorithm returns to level 1 then we again apply the variable dimension algorithm until a new approximating simplex is found on level 1 etc we allow solutions to lie on the boundary of s 1 in which case the algorithm in general will follow a path on the boundary of s 1

the d 2 triangulation for simplicial homotopy algorithms for computing solutions of nonlinear equations a triangulation of arbitrary refinement of grid sizes of 0 1 n is proposed for simplicial homotopy algorithms for computing solutions of nonlinear equations on each level the new triangulation called thed2 triangulation subdivides n into simplices according to thed1 triangulation we prove that thed2 triangulation is superior to thek2 triangulation andj2 triangulation in the number of simplices numerical tests show that the simplicial homotopy algorithm based on thed2 triangulation indeed is much more efficient

asymptotically efficient self tuning regulators this paper studies the problem of adaptive regulation of linear systems with white noise disturbances the apparent dilemma between the control objective and the need of information for parameter estimation is resolved by occasional use of white noise probing inputs and by a reparametrization of the model insights into the question concerning how often and when such probing inputs should be introduced are provided by the concept of asymptotic efficiency which quantifies the asymptotically minimal cost due to parameter ignorance or equivalently due to the infeasibility of using the optimal regulator that assumes knowledge of the system parameters asymptotically efficient adaptive regulators are constructed by making use of certain basic properties of adaptive predictors involving recursive least squares for the reparametrized model

a continuous deformation algorithm for variational inequality problems on polytopes a continuous deformation algorithm is proposed for solving a variational inequality problem on a polytopek the algorithm embeds the polytopek intok 0 and starts by applying a variable dimension algorithm onk 0 until an approximate solution is found onk 0 then by tracing the path of solutions of a system of equations the algorithm virtually follows a path of approximate solution ink 0 when the path ink 0 returns to level 0 i e k 0 the variable dimension algorithm is again used until a new approximate solution is found onk 0 the setk 0 is triangulated so that the approximate solution on the path improves the accuracy as the level increases a contrivance for a practical implementation of the algorithm is proposed and tested for some test problems

combinatorial theorems on the simplotope that generalize results on the simplex and cube in the context of the theory and computation of fixed points of continuous mappings researchers have developed combinatorial analogs of brouwer s fixed point theorem on the simplex and on the n cube although the simplex and the cube have different combinatorial properties regarding their boundaries they are both instances of a simplotope which is the cross product of simplices this paper presents three combinatorial theorems on the simplotope and shows how each translates into some known and new results on the simplex and cube including various forms of sperner s lemma each combinatorial theorem also implies set covering lemmas on the simplotope the simplex and the cube including the generalized covering lemma the knaster kuratowski mazurkiewicz lemma and a lemma of freidenfelds

on weight update saturation nonlinearities in lms adaptation the effect of a saturation type nonlinearity in the weight update equation in least mean square lms adaptation is investigated for a white gaussian data model nonlinear difference equations that include the effect of a 1 e sup ax saturation type nonlinearity on the update term driving the algorithm are derived for the weight first and second moments a nonlinear difference equation for the mean norm is explicitly solved using a differential equation approximation and integration by quadratures the steady state second moment weight behavior is evaluated approximately for the nonlinearity using these results the tradeoff between the extent of weight update saturation steady state excess mean square error and rate of algorithm convergence is studied for the same steady state misadjustment error the tradeoff results are discussed

maximum likelihood array processing in non gaussian noise with gaussian mixtures many approaches have been studied for the array processing problem when the additive noise is modeled with a gaussian distribution but these schemes typically perform poorly when the noise is non gaussian and or impulsive this paper is concerned with maximum likelihood array processing in non gaussian noise we present the cramer rao bound on the variance of angle of arrival estimates for arbitrary additive independent identically distributed iid symmetric non gaussian noise then we focus on non gaussian noise modeling with a finite gaussian mixture distribution which is capable of representing a broad class of non gaussian distributions that include heavy tailed impulsive cases arising in wireless communications and other applications based on the gaussian mixture model we develop an expectation maximization em algorithm for estimating the source locations the signal waveforms and the noise distribution parameters the important problems of detecting the number of sources and obtaining initial parameter estimates for the iterative em algorithm are discussed in detail the initialization procedure by itself is an effective algorithm for array processing in impulsive noise novel features of the em algorithm and the associated maximum likelihood formulation include a nonlinear beamformer that separates multiple source signals in non gaussian noise and a robust covariance matrix estimate that suppresses impulsive noise while also performing a model based interpolation to restore the low rank signal subspace the em approach yields improvement over initial robust estimates and is valid for a wide snr range the results are also robust to pdf model mismatch and work well with infinite variance cases such as the symmetric stable distributions simulations confirm the optimality of the em estimation procedure in a variety of cases including a multiuser communications scenario we also compare with existing array processing algorithms for non gaussian noise

efficiency and implementation of simplicial zero point algorithms in this paper we compare the efficiency of several simplicial variable dimension algorithms to do so we first treat the issues of degeneracy and accelerating we present a device for solving degeneracy furthermore we compare several accelerating techniques the technique of iterated quasi newton steps after each major cycle of the simplicial algorithm is implemented in a computer code which is used to compare the efficiency of the n 1 ray 2n ray 2n ray and 3n 1 ray algorithms except for the n 1 ray algorithm the number of function evaluations does not differ very much between the various algorithms it appeared however that the 2n algorithm needs considerably less computation time

a simplicial algorithm for the nonlinear stationary point problem on an unbounded polyhedron a path following algorithm is proposed for finding a solution to the nonlinear stationary point problem on an unbounded convex and pointed polyhedron the algorithm can start at an arbitrary point of the polyhedron the path to be followed by the algorithm is described as the path of zeros of some piecewise continuously differentiable function defined on an appropriate subdivided manifold this manifold is induced by a generalized primal dual pair of subdivided manifolds the path of zeros can be approximately followed by dividing the polyhedron into simplices and replacing the original function by its piecewise linear approximation with respect to this subdivision the piecewise linear path of this function can be generated by alternating replacement steps and linear programming pivot steps a condition under which the path of zeros converges to a solution is also stated and a description of how the algorithm operates when the problem is linear or when the polyhedron is the cartesian product of a poly

fat triangulations or solving certain nonconvex matrix optimization problems in an earlier paper the author proposed the problems of determining optimal linear transformations of the triangulationsj1 andk1 in the sense of minimizing their average directional density for a given mesh size these tasks were also formulated as optimization problems where the variable is a matrix here we solve these problems and another one which is analogously related to finding an optimal linear transformation of the new triangulationj we show thatj1 andj are themselves optimal while the ofk1 developed by van der laan and talman is optimal the latter theorem extends partial results of van der laan and talman and eaves

simplicial approximation of solutions to the nonlinear complementarity problem with lower and upper bounds ideas of a simplicial variable dimension restart algorithm to approximate zero points onrn developed by the authors and of a linear complementarity problem pivoting algorithm are combined to an algorithm for solving the nonlinear complementarity problem with lower and upper bounds the algorithm can be considered as a modification of the2n ray zero point finding algorithm onrn it appears that for the new algorithm the number of linear programming pivot steps is typically less than for the2n ray algorithm applied to an equivalent zero point problem this is caused by the fact that the algorithm utilizes the complementarity conditions on the variables

a fully polynomial time approximation algorithm for computing a stationary point of the general linear complementarity problem we apply a potential reduction algorithm to solve the general linear complementarity problem glcp minimize xty subject to ax by cz q and x y z 0 we show that the algorithm is a fully polynomial time approximation scheme fptas for computing an e approximate stationary point of the glcp note that there are some glcps in which every stationary point is a solution xty 0 these include the lcps with row sufficient matrices we also show that the algorithm is a polynomial time algorithm for a special class of glcps

optimal quasi convex combinations for stochastic approximation algorithms with parallel observers abstract motivated by numerous potential applications in decentralized estimation detection and adaptive control monte carlo optimization etc two types of stochastic approximation sa algorithms with parallel observers are developed to find the root of a non linear function with random noise corrupted measurements instead of employing the classical robbins monro rm sa procedures with a single observer a collection of physically separated parallel observers are used to estimate the same parameter a newly formed approximation sequence is obtained by means of an appropriate quasi convex combination of the most current values obtained from all the observers in addition to getting strong consistency and asymptotic normality the optimal convex combination coefficients are derived comparisons of asymptotic performance are made these comparisons indicate that the algorithms suggested here are asymptotically better than the classical approach

comparison of the convergence of two algorithms for adaptive fir digital filters the convergence properties of two different algorithms for the updating of the coefficients of an adaptive fir digital filter are investigated and compared with one another these algorithms are the stochastic iteration algorithm and the sign algorithm in this latter algorithm a one bit gradient estimation is used which makes its implementation very simple the convergence is characterized by the residual echo variance after convergence and a parameter that indicates the speed of the convergence it is shown that the convergence of the sign algorithm can always be assured but is much slower than that of the stochastic iteration algorithm if the same variance of the residual echo is to be obtained

consistency and asymptotic efficiency of slope estimates in stochastic approximation schemes adaptive stochastic approximation schemes for choosing the levels of x at which y is to be observed are useful in applications of the following nature suppose that in 1 1 x is the dosage level of a drug given to the i th patient who turns up for treatment and that yi is the response of the patient suppose also that an optimal response value y is desired without loss of generality we shall replacing y by y y if necessary assume that y 0 if 0 were known then the dosage levels should all be set at 0 since 0 is usually unknown how can the dosage levels xi be chosen so as to approach the

improved convergence analysis of stochastic gradient adaptive filters using the sign algorithm convergence analysis of stochastic gradient adaptive filters using the sign algorithm is presented in this paper the methods of analysis currently available in literature assume that the input signals to the filter are white this restriction is removed for gaussian signals in our analysis expressions for the second moment of the coefficient vector and the steady state error power are also derived simulation results are presented and the theoretical and empirical curves show a very good match

a new variable dimension algorithm for the fixed point problem we propose a new variable dimension algorithm for solving a system ofn equations inn variables the algorithm solves a sequence of subproblems consisting of the first several equations the dimension of subproblems does not however increase monotonically a convergence condition is also presented

adaptive stochastic approximation by the simultaneous perturbation method stochastic approximation sa has long been applied for problems of minimizing loss functions or root finding with noisy input information as with all stochastic search algorithms there are adjustable algorithm coefficients that must be specified and that can have a profound effect on algorithm performance it is known that choosing these coefficients according to an sa analog of the deterministic newton raphson algorithm provides an optimal or near optimal form of the algorithm however directly determining the required hessian matrix or jacobian matrix for root finding to achieve this algorithm form has often been difficult or impossible in practice the paper presents a general adaptive sa algorithm that is based on a simple method for estimating the hessian matrix while concurrently estimating the primary parameters of interest the approach applies in both the gradient free optimization kiefer wolfowitz and root finding stochastic gradient based robbins monro settings and is based on the simultaneous perturbation sp idea introduced previously the algorithm requires only a small number of loss function or gradient measurements per iteration independent of the problem dimension to adaptively estimate the hessian and parameters of primary interest aside from introducing the adaptive sp approach the paper presents practical implementation guidance asymptotic theory and a nontrivial numerical evaluation also included is a discussion and numerical analysis comparing the adaptive sp approach with the iterate averaging approach to accelerated sa

an efficient implementation of merrill s method for sparse or partially separable systems of nonlinear equations when merrill s method without an extra dimension is applied to solving sparse or partially separable systems of nonlinear equations the computational efficiency can be further improved i e a larger piece of linearity can be traversed in one step by using a suitable linear system one of the linear systems is updated and the corresponding technique is shown to update information about the linear system and the large piece in the implementation of the method some numerical results of the method support the claim that it is efficient also some mistakes from a previous paper in which the main technique exploiting sparsity was proposed are corrected

transient analysis of data normalized adaptive filters this paper develops an approach to the transient analysis of adaptive filters with data normalization among other results the derivation characterizes the transient behavior of such filters in terms of a linear time invariant state space model the stability of the model then translates into the mean square stability of the adaptive filters likewise the steady state operation of the model provides information about the mean square deviation and mean square error performance of the filters in addition to deriving earlier results in a unified manner the approach leads to stability and performance results without restricting the regression data to being gaussian or white the framework is based on energy conservation arguments and does not require an explicit recursion for the covariance matrix of the weight error vector

a stationary stochastic approximation method abstract a problem occuring in statistics and econometrics is the minimization or root finding of a function of several parameters expressed as an integral if the integral cannot be evaluated analytically and numerical integration is required evaluation can be prohibitively costly when the dimension of integration exceeds two since monte carlo techniques are natural in such a case stochastic approximation can be viewed as combining integration and optimization while these methods require minimal conditions for convergence implementation as a practical problem solver faces difficulties in verification and stopping a stationary stochastic approximation method is proposed which is less sophisticated mathematically this method is applied to two types of problems a bayesian decision problem and the computation of minimum hellinger distance estimators

detection of random signals in gaussian mixture noise a locally optimal detection algorithm for random signals in dependent noise is derived and applied to independent identically distributed complex valued gaussian mixture noise the resulting detector is essentially a weighted sum of power detectors the power detector is the locally optimal detector for random signals in gaussian noise the weighting functions are modified to enhance the detection performance for small sample sizes an implementation of the mixture detector using the expectation maximization algorithm is described moments of these detectors are calculated from piecewise polynomial approximations of the weighting functions the sum of sufficiently many independent identically distributed detector outputs is then approximated by a normal distribution probability distributions are also derived for the power detector in gaussian mixture noise for a particular set of noise parameters the theoretical distributions are compared with those obtained from monte carlo simulation and seen to be quite close the theoretical distributions are then used to compare the performance of the mixture and power detectors in gaussian mixture noise over a range of parameters and to assess the impact of parameter error on detection performance in this study the signal gain of the mixture detectors varies from 15 to 38 db and the degradation of the probability of detection due to parameter estimation error is relatively minor

a scaled stochastic approximation algorithm consider a stochastic system of such complexity that its performance can only be evaluated by using simulation or direct experimentation to optimize the expected performance of such systems as a function of several continuous input parameters decision variables we present a scaled stochastic approximation algorithm for finding the zero root of the gradient of the response function in each iteration of the scaled algorithm two independent gradient estimates are sampled at the current estimate of the optimal input parameter vector to compute a scale free estimate of the next search direction we establish sufficient conditions to ensure strong consistency and asymptotic normality of the resulting estimator of the optimal input parameter vector strong consistency is also established for a variant of the scaled algorithm with kesten s acceleration an experimental performance comparison of the scaled algorithm and the classical robbins monro algorithm in two simple queueing systems reveals some of the practical advantages of the scaled algorithm

the d1 triangulation of rn for simplicial algorithms for computing solutions of nonlinear equations we present a new triangulation of n which is called the d1 triangulation for computing zero points or fixed points of nonlinear mappings the d1 triangulation subdivides the unit cube and is based on very elementary pivot rules we compare the d1 triangulation to several well known triangulations of n which triangulate the unit cube according to several measures of efficiency the new triangulation is superior such as the number of simplices in the unit cube the diameter of a triangulation the average directional density and the surface density

on the optimum data nonlinearity in lms adaptation the effect of an arbitrary nonlinear operation on the data input to the weight update equation in the lms adaptive algorithm is investigated for a gaussian data model exact difference equations are derived for the weight first and second moments that include the effects of an arbitrary nonlinear operation on the data sequence the difference equations are used to obtain expressions for the transient behavior of the mean square error the mean square error is minimized over the choice of nonlinearity for a fixed transient behavior the best choice of nonlinearity is shown exactly to be of the form x 1 bx2 when the input data are white for an arbitrary data covariance matrix the optimum nonlinearity is shown to be linear when the product of the algorithm step size and input power is much less than unity

average intersection and pivoting densities for the purposes of quantifying efficiency of more than one codimension piecewise linear continuation algorithms higher dimensional analogues of the average directional density are defined and studied two such are explicitly considered the average intersection density which measures the local density of intersections of an implicitly defined piecewise linear manifold with complementing faces of a decomposition of euclidean space and average pivoting density which considers the number of pivots such an intersection entails general results are established which relate the densities to geometric properties of the decomposition explicit closed formulae are presented for the most interesting case of the freudenthal kuhn triangulation and asymptotic expressions are developed

a unifying model based on retraction for fixed point algorithms we present a unifying model based on retraction for several restart fixed point algorithms the model embraces the interpretation of the algorithms in terms of stationary point problem by van der laan and talman and fully explains the 2 ray method

a class of order statistic lms algorithms conventional gradient based adaptive filters as typified by the well known lms algorithm use an instantaneous estimate of the error surface gradient to update the filter coefficients such a strategy leaves the algorithm extremely vulnerable to impulsive interference a class of adaptive algorithms employing order statistic filtering of the sampled gradient estimates is presented these algorithms dubbed order statistic least mean squares oslms are designed to facilitate adaptive filter performance close to the least squares optimum across a wide range of input environments from gaussian to highly impulsive three specific oslms filters are defined the median lms the average lms and the trimmed mean lms the properties of these algorithms are investigated and the potential for improvement demonstrated finally a general adaptive oslms scheme in which the nature of the order statistic operator is also adapted in response to the statistics of the input signal is presented it is shown that this can facilitate performance gains over a wide range of input data types

permutation congruent transformations of the freudenthal triangulation with minimum surface density linear transformations of the freudenthal triangulation with permutation congruent simplexes are characterized and within this class those transformations yielding minimum normalized surface density are identified

acceleration of stochastic approximation by averaging a new recursive algorithm of stochastic approximation type with the averaging of trajectories is investigated convergence with probability one is proved for a variety of classical optimization and identification problems it is also demonstrated for these problems that the proposed algorithm achieves the highest possible rate of convergence

multivariate stochastic approximation using a simultaneous perturbation gradient approximation the problem of finding a root of the multivariate gradient equation that arises in function minimization is considered when only noisy measurements of the function are available a stochastic approximation sa algorithm for the general kiefer wolfowitz type is appropriate for estimating the root the paper presents an sa algorithm that is based on a simultaneous perturbation gradient approximation instead of the standard finite difference approximation of keifer wolfowitz type procedures theory and numerical experience indicate that the algorithm can be significantly more efficient than the standard algorithms in large dimensional problems

feedback and weighting mechanisms for improving jacobian estimates in the adaptive simultaneous perturbation algorithm it is known that a stochastic approximation sa analogue of the deterministic newton raphson algorithm provides an asymptotically optimal or near optimal form of stochastic search however directly determining the required jacobian matrix or hessian matrix for optimization has often been difficult or impossible in practice this paper presents a general adaptive sa algorithm that is based on a simple method for estimating the jacobian matrix while concurrently estimating the primary parameters of interest relative to prior methods for adaptively estimating the jacobian matrix the paper introduces two enhancements that generally improve the quality of the estimates for underlying jacobian hessian matrices thereby improving the quality of the estimates for the primary parameters of interest the first enhancement rests on a feedback process that uses previous jacobian estimates to reduce the error in the current estimate the second enhancement is based on an optimal weighting of per iteration jacobian estimates from the use of simultaneous perturbations the algorithm requires only a small number of loss function or gradient measurements per iteration independent of the problem dimension to adaptively estimate the jacobian matrix and parameters of primary interest

performance analysis of adaptive filters equipped with the dual sign algorithm a convergence analysis for stochastic gradient adaptive filters equipped with the dual sign algorithm dsa is presented expressions for the mean and mean squared values of the coefficient misalignment vector are derived under the assumption that the input signal is gaussian the main differences between the current analysis and previous analyses are 1 the present analysis is valid for arbitrary autocorrelation matrices and 2 the analysis does not approximate the behavior for the dsa as switching between two sign algorithms a certain amount of time after the dsa is initialized a simulation example comparing the analytical results with empirical ones is presented and the two curves show excellent agreement with each other

a better triangulation for wright s 2n ray algorithm abstract we give a triangulation for wright s 2n ray algorithm to compute solutions of nonlinear equations according to measures of efficiency of triangulations it is better than any other available triangulation for the 2n ray algorithm

normalized data nonlinearities for lms adaptation properly designed nonlinearly modified lms algorithms in which various quantities in the stochastic gradient estimate are operated upon by memoryless nonlinearities have been shown to perform better than the lms algorithm in system identification type problems the authors investigate one such algorithm given by w sub k l w sub k spl mu d sub k w sub k sup t x sub k x sub k f x sub k in which the function f x sub k is a scalar function of the sum of the squares of the n elements of the input data vector x sub k this form of algorithm generalizes the so called normalized lms nlms algorithm they evaluate the expected behavior of this nonlinear algorithm for both independent input vectors and correlated gaussian input vectors assuming the system identification model by comparing the nonlinear algorithm s behavior with that of the lms algorithm they then provide a method of optimizing the form of the nonlinearity for the given input statistics in the independent input case they show that the optimum nonlinearity is a single parameter version of the nlms algorithm with an additional constant in the denominator and show that this algorithm achieves a lower excess mean square error mse than the lms algorithm with an equivalent convergence rate additionally they examine the optimum step size sequence for the optimum nonlinear algorithm and show that the resulting algorithm performs better and is less complex to implement than the optimum step size algorithm derived for another form of the nlms algorithm simulations verify the theory and the predicted performance improvements of the optimum normalized data nonlinearity algorithm

a class of simplicial restart fixed point algorithms without an extra dimension in an earlier paper we introduced an algorithm for approximating a fixed point of a mapping on the product space of unit simplices ideas of that paper are used to construct a class of triangulations ofrn more precisely for somek 1 k n and positive integersm1 mk with sumn a triangulation ofrn is obtained by triangulating the cells which are formed by taking the product of given triangulations ofrmj j 1 k the triangulation of each cell will be defined in relation to an arbitrarily chosen pointv inrn being the starting point of the algorithm fork n we obtain thek triangulation originally due to todd each element of the class can be used to find a simplex which approximates a fixed point of a mapping onrn by generating a unique path of adjacent simplices of variable dimension starting with the pointv

rudiments of an average case complexity theory for piecewise linear path following algorithms we examine the efficiency of pl path following algorithms in followingft 1 0 whereft is the pl approximation induced by the simplicial triangulationt to a mapf n n 1 in particular we consider the problem of determining an upper bound on the expected number of pivots made per unit length off 1 0 that is approximated we show that if the sizes of the simplices oft are sufficiently small where sufficiently small is an explicitly given quantity dependent on measurements of how nice f is then the average directional density oft as introduced by todd really does give a good approximation to the expected number of pivots made confirming what researchers have believed on intuitive grounds for a decade because what constitutes sufficiently small is a precisely given quantity i e non asymptotic we are able to provide some rigorous justification for the claim that the expected number of pivots grows only polynomially inn the number of variables several other issues are also examined

a modified integer labeling for complementarity algorithms nonlinear complementary problems can be solved by algorithms based on the principle of simplicial approximation we present such an algorithm that allows the choice of an arbitrary starting point without using an artificial labeling or an extra dimension furthermore the relation between the grid size of the triangulation and the accuracy of an approximate solution is investigated and it is shown that a combination of the algorithm with newton steps yields better accuracy finally the algorithm is adapted for solving systems of equations

multivariate adaptive stochastic approximation on considere une version multivariable de l approximation stochastique adaptative developpee recemment par lai et robbins 1978 1979

equivalence of surface density and average directional density the average directional density criteria for evaluating tilings is shown to be equivalent to surface density and valid for random broken paths just as for straight paths