on error saturation nonlinearities in nlms adaptation the effect of a saturation type error nonlinearity in the weight update equation in normalized least mean square nlms adaptation is investigated for system identification for a white gaussian data model nonlinear recursions are derived for the weight mean error and mean square deviation msd that include the effect of an error function erf saturation type nonlinearity on the error sequence driving the algorithm the nonlinear recursion for the msd is solved numerically and shown in excellent agreement with monte carlo simulations supporting the theoretical model assumptions the theory is extended to tracking a markov channel and accurately predicts the tracking behavior as well the saturation behavior of the algorithm is easily studied by varying a single parameter in the error function varying from a linear device to a hard limiter for the white data case the excess mean square error emse is simply related to the msd the tradeoff between the extent of error saturation steady state emse and algorithm convergence rate is studied using these results

a parallel adaptation algorithm for recursive least squares adaptive filters in nonstationary environments an accurate new expression for the steady state tracking performance of exponentially weighted recursive least squares rls adaptive filters in a random walk scenario is derived this relation is then used to provide a detailed comparison between rls performance and that of normalized least mean squares adaptive filters further a variable forgetting factor algorithm referred to as the parallel adaptation algorithm that approximately achieves the theoretical minimum mean squared error performance in a random walk scenario is developed extensive simulation results are presented to support the present findings and demonstrate the improved performance of the proposed algorithm in a number of different applications

statistical tracking behavior of affine projection algorithm for unity step size since unity step size could guarantee the fastest convergence and more detailed analysis for the affine projection ap algorithm a statistical tracking behavior of ap algorithm is discussed in this paper deterministic recursive equations are derived for the mean weight error and mean square error all the possible correlations between the adaptive filtering coefficients and the past measurement noise are considered as well

upper and lower bounds of the misadjustment in the lms algorithm the misadjustment is a dimensionless measure of the difference between the optimal weiner performance and the actual performance of the lms least mean squares algorithm this difference is attributed to the gradient noise in the adaptive process present estimates relate the misadjustment to the convergence gain and the trace of the autocorrelation matrix the authors establish tighter lower bounds on the estimate of this misadjustment an upper bound on the misadjustment as a function of the convergence gain is found

stochastic modeling of the transform domain varepsilon rm lms algorithm this paper presents a statistical analysis of the transform domain least mean square tdlms algorithm resulting in a more accurate model than those discussed in the current open literature the motivation to analyze such an algorithm comes from the fact that the tdlms presents a higher convergence speed for correlated input signals as compared with other adaptive algorithms possessing a similar computational complexity such a fact makes it a highly competitive alternative to some applications approximate analytical models for the first and second moments of the filter weight vector are obtained the tdlms algorithm has an orthonormal transformation stage accomplishing a decomposition of the input signal into distinct frequency bands in which the interband samples are practically uncorrelated on the other hand the intraband samples are correlated the larger the number of bands the higher their correlation the model is then derived taking into account such a correlation requiring that a high order hyperelliptic integral be computed in addition to the proposed model an approximate procedure for computing high order hyperelliptic integrals is presented a regularization parameter is also considered in the model expressions permitting to assess its impact on the adaptive algorithm behavior an upper bound for the step size control parameter is also obtained through simulation results the accuracy of the proposed model is assessed

a short proof of the xiao massey lemma correlation immune combining functions a short and lucid proof of the xiao massey lemma is given it provides various ways of generalizing the lemma

performance advantage of complex lms for controlling narrow band adaptive arrays in narrow band adaptive array applications the mean square convergence of the discrete time real least mean square lms algorithm is slowed by image frequency noises generated in the lms loops the complex lms algorithm proposed by widrow et al is shown to eliminate these noises yielding convergence of the mean squared error mse at slightly over twice the rate this paper includes a comprehensive analysis of the mse of adaptation for lms the analysis is based upon the method developed in the 1968 dissertation by k d senne and it represents the most complete treatment of the subject published to date

on the optimum gain parameter in lms adaptation in this correspondence the optimum selection of the adaptation parameter in lms adaptive estimation is discussed for a white data covarianee matrix it is shown via analysis and numerical evaluation that the optimum selection of is a complicated function of the number of filter taps the initial weight setting the wiener weights and the number of learning samples is not chosen in general to yield the most rapid transient response for most cases of interest a smaller value of will be selected for slower adaptation and smaller misadjustment error at the end of the learning phase

mmse detection in asynchronous cdma systems an equivalence result the analysis of linear minimum mean square error mmse detection in a band limited code division multiple access cdma system that employs random spreading sequences is considered the key features of the analysis are that the users are allowed to be completely asynchronous and that the chip waveform is assumed to be the ideal nyquist sinc function it is shown that the asymptotic signal to interference ratio sir at the detector output is the same as that in an equivalent chip synchronous system it is hence been established that synchronous analyses of linear mmse detection can provide useful guidelines for the performance in asynchronous band limited systems

a measure of the tracking capability of recursive stochastic algorithms with constant gains a criterion is given for measuring the tracking capability of recursive algorithms when applied to slowly time varying systems the optimal gain for a given disturbance is also calculated this criterion is seen to have some connection with the fisher information matrix and allows us to select a priori the best algorithm for identifying a given unknown parameter which may be subject to smooth unknown disturbances examples of applications are given in the areas of identification theory and data communication theory

a family of shrinkage adaptive filtering algorithms a family of adaptive filtering algorithms that uses a variable step size is proposed a variable step size is obtained by minimizing the energy of the noise free a posteriori error signal which is obtained by using a known l 1 hbox l 2 minimization formulation based on this methodology a shrinkage affine projection shap algorithm a shrinkage least mean squares shlms algorithm and a shrinkage normalized least mean squares shnlms algorithm are proposed the shap algorithm yields a significantly reduced steady state misalignment as compared to the conventional affine projection ap variable step size ap and set membership ap algorithms for the same convergence speed although the improvement is achieved at the cost of an increase in the average computational effort per iteration in the amount of 11 to 14 the shlms algorithm yields a significantly reduced steady state misalignment and faster convergence as compared to the conventional lms and variable step size lms algorithms similarly the shnlms algorithm yields a significantly reduced steady state misalignment and faster convergence as compared to the conventional normalized least mean squares nlms and set membership nlms algorithms

fast converging and low complexity adaptive filtering using an averaged kalman filter kalman filtering is applied to obtain a fast converging low complexity adaptive filter that is of the matrix stepsize normalized least mean square nlms type by replacing certain variables with averages the solution of an averaged diagonal riccati equation allows optimal time varying adaptation gains to be precomputed or computed online with a small number of scalar riccati equations the adaptation gains are computed from prior assumptions on impulse response power and shape this fact results in a systematic procedure for adaptation gain tuning in the time varying matrix stepsize case simulations using music as input show significant performance improvements as compared with the nlms algorithm

adaptive lms l filters for noise suppression in images several adaptive least mean squares lms l filters both constrained and unconstrained ones are developed for noise suppression in images and compared in this paper first the location invariant lms l filter for a nonconstant signal corrupted by zero mean additive white noise is derived it is demonstrated that the location invariant lms l filter can be described in terms of the generalized linearly constrained adaptive processing structure proposed by griffiths and jim 1982 subsequently the normalized and the signed error lms l filters are studied a modified lms l filter with nonhomogeneous step sizes is also proposed in order to accelerate the rate of convergence of the adaptive l filter finally a signal dependent adaptive filter structure is developed to allow a separate treatment of the pixels that are close to the edges from the pixels that belong to homogeneous image regions

a spectral characterization of correlation immune combining functions it is shown that a boolean combining function f x of n variables is mth order correlation immune if and only if its walsh transform f omega vanishes for all omega with hamming weight between 1 and m inclusive this result is used to extend slightly siegenthaler s ieee trans comput vol c 34 pp 81 85 jan 1985 characterization of the algebraic normal form of correlation immune combining functions

steady state performance of spline adaptive filters recently a novel class of nonlinear adaptive filters called spline adaptive filters safs has been introduced and demonstrated to be very effective in many practical applications the learning rules of these architectures are based on the least mean square lms algorithm in order to provide theoretical foundation to the saf in this paper we provide a steady state performance evaluation in particular after the stochastic analysis of the mean behavior of the saf approach under the gaussian assumption the analytical derivation of the theoretical excess mean square error emse and the normalized misadjustment are derived and discussed the proposed analysis of emse and misadjustment is based on the energy conservation approach that has been extended to saf architecture the derived theoretical analysis allows to accurately predict the steady state performance therefore some properties for the correct choice of filter parameters are also provided experimental results demonstrate the effectiveness of the analysis results

the error variance of lms with time varying weights a calculation of weight error variance of the lms least mean square algorithm is made in the presence of time varying true weights with time varying weights the lms error system has in general several time scales operating at once this causes difficulties in the variance calculation which seem hitherto to have passed unnoticed to handle this problem a sort of perturbation expansion is developed based on weak convergence methods or stochastic averaging the main concern in carrying out the error variance calculation is to study the effect on lms performance of adaptation speed as it relates to true speed of parameter change three cases are covered with reference to considering adaptation speed with respect to true speed first where the adaptation speed is too slow second where it is matched and third where it is faster

asymptotic and finite sample properties of estimators based on stochastic gradients stochastic gradient descent procedures have gained popularity for parameter estimation from large data sets however their statistical properties are not well understood in theory and in practice avoiding numerical instability requires careful tuning of key parameters here we introduce implicit stochastic gradient descent procedures which involve parameter updates that are implicitly defined intuitively implicit updates shrink standard stochastic gradient descent updates the amount of shrinkage depends on the observed fisher information matrix which does not need to be explicitly computed thus implicit procedures increase stability without increasing the computational burden our theoretical analysis provides the first full characterization of the asymptotic behavior of both standard and implicit stochastic gradient descent based estimators including finite sample error bounds importantly analytical expressions for the variances of these stochastic gradient based estimators reveal their exact loss of efficiency we also develop new algorithms to compute implicit stochastic gradient descent based estimators for generalized linear models cox proportional hazards m estimators in practice and perform extensive experiments our results suggest that implicit stochastic gradient descent procedures are poised to become a workhorse for approximate inference from large data sets

efficient subspace based algorithm for adaptive bearing estimation and tracking in some practical applications of array processing the directions of the incident signals should be estimated adaptively and or the time varying directions should be tracked promptly in this paper an adaptive bearing estimation and tracking abest algorithm is investigated for estimating and tracking the uncorrelated and correlated narrow band signals impinging on a uniform linear array ula based on the subspace based method without eigendecomposition sumwe where a linear operator is obtained from the array data to form a basis for the null space by exploiting the array geometry and its shift invariance property specifically the null space is estimated using the least mean square lms or normalized lms nlms algorithm and the directions are updated using the approximate newton method the transient analyses of the lms and nlms algorithms are studied where the weight i e the linear operator is in the form of a matrix and there is a correlation between the additive noise and input data that involve the instantaneous correlations of the received array data in the updating equation and the step size stability conditions are derived explicitly in addition the analytical expressions for the mean square error mse and mean square deviation msd learning curves of the lms algorithm are clarified the effectiveness of the abest algorithm is verified and the theoretical analyses are corroborated through numerical examples simulation results show that the abest algorithm is computationally simple and has good adaptation and tracking abilities

a time domain feedback analysis of filtered error adaptive gradient algorithms this paper provides a time domain feedback analysis of gradient based adaptive schemes a key emphasis is on the robustness performance of the adaptive filters in the presence of disturbances and modeling uncertainties along the lines of h sup spl infin theory and robust filtering the analysis is carried out in a purely deterministic framework and assumes no prior statistical information or independence conditions it is shown that an intrinsic feedback structure can be associated with the varied adaptive schemes the feedback structure is motivated via energy arguments and is shown to consist of two major blocks a time variant lossless i e energy preserving feedforward path and a time variant feedback path the configuration is further shown to lend itself to analysis via a so called small gain theorem thus leading to stability and robustness conditions that require the contractivity of certain operators choices for the step size parameter in order to guarantee faster rates of convergence are also derived and simulation results are included to demonstrate the theoretical findings in addition the time domain analysis provided in this paper is shown to extend the so called transfer function approach to a general time variant scenario without any approximations

speech enhancement in discontinuous transmission systems using the constrained stability least mean squares algorithm in this paper a novel constrained stability least mean squares lms algorithm for filtering speech sounds is proposed in the adaptive noise cancellation anc problem it is based on the minimization of the squared euclidean norm of the weight vector change under a stability constraint over the a posteriori estimation errors to this purpose the lagrangian methodology has been used in order to propose a nonlinear adaptation in terms of the product of differential input and error convergence analysis is also studied in terms of the evolution of the natural modes to the optimal wiener hopf solution so that the stability performance depends exclusively on the adaptation parameter and the eigenvalues of the difference matrix r1 the algorithm shows superior performance over the referenced algorithms in the anc problem of speech discontinuous transmission systems which are characterized by rapid transitions of the desired signal the experimental analysis carried out on the aurora 3 speech databases provides an extensive performance evaluation together with an exhaustive comparison to the standard lms algorithms i e the normalized lms nlms and other recently reported lms algorithms such as the modified nlms the error nonlinearity lms or the normalized data nonlinearity lms adaptation

convergence and performance analysis of the normalized lms algorithm with uncorrelated gaussian data it is demonstrated that the normalized least mean square nlms algorithm can be viewed as a modification of the widely used lms algorithm the nlms is shown to have an important advantage over the lms which is that its convergence is independent of environmental changes in addition the authors present a comprehensive study of the first and second order behavior in the nlms algorithm they show that the nlms algorithm exhibits significant improvement over the lms algorithm in convergence rate while its steady state performance is considerably worse

optimal orthogonal carriers and sum sinr sum capacity of the multiple access vector channel for the general correlated multiple access vector channel we derive algebraically the sum sinr signal to interference plus noise ratio optimal orthonormal set of carriers for any given set of carrier amplitudes if in addition the correlated multiple access channel is assumed gaussian the sum capacity optimal orthonormal carrier set design and the jointly sum capacity optimal orthonormal carrier and power assignment for a given power budget follow readily

the limiting behavior of lms a realization oriented analysis is given of the gradient noise misadjustment and lag misadjustment performance of the lms least mean square algorithm new formulas are given for both of these components of excess mean square error it is shown that the traditional formula for lag misadjustment needs to be modified by adding further terms due to gradient noise and noise variance to perform the analysis it is necessary to study the convergence with probability one of the noise free fixed parameter lms algorithm convergence is found under simple conditions that improve on those previously obtained

transient and steady state analysis of the affine combination of two adaptive filters in this paper we propose an approach to the transient and steady state analysis of the affine combination of one fast and one slow adaptive filters the theoretical models are based on expressions for the excess mean square error emse and cross emse of the component filters which allows their application to different combinations of algorithms such as least mean squares lms normalized lms nlms and constant modulus algorithm cma considering white or colored inputs and stationary or nonstationary environments since the desired universal behavior of the combination depends on the correct estimation of the mixing parameter at every instant its adaptation is also taken into account in the transient analysis furthermore we propose normalized algorithms for the adaptation of the mixing parameter that exhibit good performance good agreement between analysis and simulation results is always observed

region based wavelet packet adaptive algorithm for identification of sparse impulse responses identification of systems with sparse impulse response encounters large applicability numerous techniques have been proposed to identify such systems efficiently one strategy that leads to specially good results is to detect the active nonzero response samples and update only the corresponding adaptive coefficients wavelet based approaches have been shown to be specially effective to this end this paper proposes a new region based wavelet packet rbwp algorithm for efficient identification of systems with sparse impulse responses with arbitrary frequency spectra and with any delay of the effective response the discrete wavelet packet transform dwpt is adaptively tailored to the energy distribution of the unknown system s response spectrum the new algorithm leads to a reduced number of active weights and to a reduced computational complexity when compared with competing wavelet based algorithms monte carlo simulation results show good performances regarding convergence speed and robustness to design parameter choice

on the convergence rate performance of the normalized least mean square adaptation this paper compares the convergence rate performance of the normalized least mean square nlms algorithm to that of the standard least mean square lms algorithm which is based on a well known interpretation of the nlms algorithm as a form of the lms via input normalization with this interpretation the analysis is considerably simplified and the difference in rate of parameter convergence can be compared directly by evaluating both the condition number of the normalized and unnormalized input correlation matrix this paper derives the condition number expressions for the normalized input correlation matrix of which the arbitrary length filter model is linear with respect to its adaptable parameters and contain only two distinct unnormalized eigenvalues these expressions which require that the input samples be statistically stationary and zero mean gaussian distributed provide an important insight into the relative convergence performance of the nlms algorithm to that of the lms as a function of filter length this paper also provides a conjecture which set bounds on the nlms condition number for any arbitrary number of distinct unnormalized eigenvalues

a family of adaptive filter algorithms with decorrelating properties although the normalized least mean square nlms algorithm is robust it suffers from low convergence speed if driven by highly correlated input signals one method presented to overcome this problem is the ozeki umeda 1984 affine projection ap algorithm the algorithm applies update directions that are orthogonal to the last p input vectors and thus allows decorrelation of an ar p input process speeding up the convergence this article presents a simple approach to show this property which furthermore leads to the construction of new algorithms that can handle other kinds of correlations such as ma and arma processes a statistical analysis is presented for this family of algorithms similar to the ap algorithm these algorithms also suffer a possible increase in the noise energy caused by their pre whitening filters

convergence behavior of affine projection algorithms a class of equivalent algorithms that accelerate the convergence of the normalized lms nlms algorithm especially for colored inputs has previously been discovered independently the affine projection algorithm apa is the earliest and most popular algorithm in this class that inherits its name the usual apa algorithms update weight estimates on the basis of multiple unit delayed input signal vectors we analyze the convergence behavior of the generalized apa class of algorithms allowing for arbitrary delay between input vectors using a simple model for the input signal vectors conditions for convergence of the apa class are derived it is shown that the convergence rate is exponential and that it improves as the number of input signal vectors used for adaptation is increased however the rate of improvement in performance time to steady state diminishes as the number of input signal vectors increases for a given convergence rate apa algorithms are shown to exhibit less misadjustment steady state error than nlms simulation results are provided to corroborate the analytical results

variable length stochastic gradient algorithm the transversal variable length stochastic gradient algorithm is described it is a modification of the stochastic gradient algorithm that allows dynamic allocation of coefficients of an adaptive filter the order of the filter and the adaptation step size are changed automatically when an appropriate level of performance is reached during the course of the adaptation process in this way the algorithm results in fast convergence typical of low order filters and good steady state performance typical of high order filters

tracking properties and steady state performance of rls adaptive filter algorithms adaptive signal processing algorithms derived from ls least squares cost functions are known to converge extremely fast and have excellent capabilities to track an unknown parameter vector this paper treats analytically and experimentally the steady state operation of rls recursive least squares adaptive filters with exponential windows for stationary and nonstationary inputs a new formula for the estimation noise has been derived involving second and fourth order statistics of the filter input as well as the exponential windowing factor and filter length furthermore it is shown that the adaptation process associated with lag effects depends solely on the exponential weighting parameter in addition the calculation of the excess mean square error due to the lag for an assumed markov channel provides the necessary information about tradeoffs between speed of adaptation and steady state error it is also the basis for comparison to the simple lms algorithm in a simple case of channel identification it is shown that the lms and rls adaptive filters have the same tracking behavior finally in the last part we present new rls restart procedures applied to transversal structures for mitigating the disastrous results of the third source of noise namely finite precision arithmetic

gaussian multiaccess channels with isi capacity region and multiuser water filling the capacity region of a two user gaussian multiaccess channel with intersymbol interference isi in which the inputs pass through respective linear systems and are superimposed before being corrupted by an additive gaussian noise process is discussed a geometrical method for obtaining the optimal input power spectral densities and the capacity region is presented this method can be viewed as a nontrivial generalization of the single user water filling argument it is shown that as in the traditional memoryless multiaccess channel frequency division multiaccess fdma with optimally selected frequency bands for each user achieves the total capacity of the multiuser gaussian multiaccess channel with isi however the capacity region of the two user channel with memory is in general not a pentagon unless the channel transfer functions for both users are identical

user capacity of asynchronous cdma systems with matched filter receivers and optimum signature sequences for a symbol asynchronous but chip synchronous single cell code division multiple access cdma system we define a system wide quantity called the total squared asynchronous correlation tsac which for arbitrary signature sets depends on the users delay profile we develop a lower bound for tsac that is independent of the users delays we show that if the signature set achieves this tsac lower bound then the user capacity of the asynchronous cdma system using matched filters becomes the same as that of a single cell synchronous cdma system in this case there is no loss in user capacity due to asynchronism we present iterative signature adaptation algorithms which when executed sequentially by the users appear to converge to these optimum signature sequences however the existence for all user delay profiles of signature sequences achieving this lower bound remains a significant open problem

behavior of the e normalized lms algorithm with gaussian inputs the first and second moment behavior of one variation of the normalized lms e nlms algorithm is investigated for a white covariance matrix and gaussian statistics for the data for this model it is shown that the e nlms algorithm has neither behavior independent of the input data power nor a performance significantly better than the lms algorithm for which the input power level also must be known a priori hence based upon the results of the analysis it is recommended that the algorithm not be used in place of the lms for known input power levels or in place of the nlms for unknown input power levels

set membership binormalized data reusing lms algorithms this paper presents and analyzes novel data selective normalized adaptive filtering algorithms with two data reuses the algorithms the set membership binormalized lms sm bn drlms algorithms are derived using the concept of set membership filtering smf these algorithms can be regarded as generalizations of the previously proposed set membership nlms sm nlms algorithm they include two constraint sets in order to construct a space of feasible solutions for the coefficient updates the algorithms include data dependent step sizes that provide fast convergence and low excess mean squared error mse convergence analyzes in the mean squared sense are presented and closed form expressions are given for both white and colored input signals simulation results show good performance of the algorithms in terms of convergence speed final misadjustment and reduced computational complexity

low complexity data reusing methods in adaptive filtering most adaptive filtering algorithms couple performance with complexity over the last 15 years a class of algorithms termed affine projection algorithms have given system designers the capability to tradeoff performance with complexity by changing parameters and the size scale of data used to update the coefficients of an adaptive filter but without fundamentally changing the algorithm structure a system designer can radically change the performance of the adaptive algorithm this paper discusses low complexity data reusing algorithms that are closely related to affine projection algorithms this paper presents various low complexity and highly flexible schemes for improving convergence rates of adaptive algorithms that utilize data reusing strategies all of these schemes are unified by a row projection framework in existence for more than 65 years this framework leads to the classification of all data reusing and affine projection methods for adaptive filtering into two categories the kaczmarz and cimmino methods simulation and convergence analysis results are presented for these methods under a number of conditions they are compared in terms of convergence rate performance and computational complexity

randomness arrays differences and duality random variables that take on values in the finite field of q elements are considered it is shown that joint distributions of such random variables are equivalently described by the individual distributions of their linear combinations random vectors x that are equally likely to take on any row of an arbitrary q ary rectangular array as their value are treated extensively together with the random vector spl delta x defined as the difference between two independent versions of such a random vector it is shown that linear combinations of exactly spl tau of the components of x are always biased toward 0 a quantitative measure spl beta sub spl tau of this bias is introduced and shown to be given by a sum of krawtchouk polynomials the vanishing of spl beta sub spl tau is shown to be equivalent to the maximal randomness of linear combinations of exactly spl tau of the components of x as well as of spl delta x when the rows of the original array are the codewords of a q ary linear code then the bias spl beta sub spl tau coincides with the number of codewords of hamming weight spl tau in the dual code the results of this article generalize certain well known results such as the macwilliams 1977 identities and delsarte s 1973 theorem on the significance of the dual distance of nonlinear codes

convergence analysis of the binormalized data reusing lms algorithm normalized least mean squares algorithms for fir adaptive filtering with or without the reuse of past information are known to converge often faster than the conventional least mean squares lms algorithm this correspondence analyzes an lms like algorithm the binormalized data reusing least mean squares bndr lms algorithm this algorithm which corresponds to the affine projection algorithm for the case of two projections compares favorably with other normalized lms like algorithms when the input signal is correlated convergence analyses in the mean and in the mean squared are presented and a closed form formula for the mean squared error is provided for white input signals as well as its extension to the case of a colored input signal a simple model for the input signal vector that imparts simplicity and tractability to the analysis of second order statistics is fully described the methodology is readily applicable to other adaptation algorithms of difficult analysis simulation results validate the analysis and ensuing assumptions

a note on the convergence analysis of lms adaptive filters with gaussian data necessary and sufficient conditions for the convergence of lms least mean squares adaptive filters with gaussian data have been established by l l horowitz and k d senne 1981 with the recent support of a feuer and e weinstein 1985 a feature of both of these studies is the necessity to investigate bounds on the roots of rather unwieldy characteristic equations the authors show how such an investigation can be avoided through the use of a theorem of f r gantmacher 1959 in formally applying this theorem similar results to those of the above studies are obtained in a precise and straightforward manner

statistical convergence behavior of affine projection algorithms class of algorithms referring to the affine projection algorithms apa applies updates to the weights in a direction that is orthogonal to the most recent input vectors this speeds up the convergence of the algorithm over that of the normalized least mean square nlms algorithm especially for highly colored input processes in this paper a new statistical analysis model is used to analyze the apa class of algorithms with unity step size four assumptions are made which are based on the direction vector for the apa class under these assumptions deterministic recursive equations for the weight error and for the mean square error are derived we also analyze the steady state behavior of the apa class the new model is applicable to input processes that are autoregressive as well as autoregressive moving average and therefore is useful under more general conditions than previous models for prediction of the mean square error of the apa class simulation results are provided to corroborate the analytical results

fast transversal filters with data sequence weighting a fast o n algorithm is presented for the adaptive recursive least squares design of transversal filters data sequence weighting is introduced to allow for arbitrarily time varying weighting strategies facilitating the tracking of arbitrary nonstationary phenomena a novel weighting adaptation mechanism is presented that has various desirable features and optimal performance under certain conditions the algorithm is derived for the prewindowed given data case and the underlying principles of the derivation are clearly exposed exact and soft constraint initialization are discussed and an improved restart procedure is proposed

optimization of adaptive identification for time varying filters adaptive identification in a time varying context is studied when controlled by the lms algorithm with constant gain under the assumption of correlated successive input vectors it is well known by experience that the tracking mean square error mse epsilon mu results from the tradeoff between the gradient part which is increasing and the lag contribution which is decreasing in this note we clarify the relative roles of the gradient and lag errors by proving their decoupled character this property relies upon independence between the additive noise at the output of the plant to be identified and the information vector at the plant input convergence of the mse is established rather than assumed quantitative evaluations of upper and lower bounds allow an approximate optimization of the gain in two important cases the optimum is exact one of these cases is slow variations it is defined in a quantitative manner thanks to the ratio of the variation noise to the output additive noise

exact performance analysis of the epsilon nlms algorithm for colored circular gaussian inputs this paper presents exact mean square analysis of the nlms algorithm for circular complex correlated gaussian input the analysis is based on the derivation of a closed form expression for the cumulative distribution function cdf of random variables of the form ui d12 ui d12 and using that to derive the first and second moments of such variables these moments in turn completely characterize the mean square ms behavior of the nlms in explicit closed form expressions both transient and steady state behavior are analyzed consequently new explicit closed form expressions for the mean square error mse behavior are derived our simulations of the transient and steady state behavior of the filter match the expressions obtained theoretically for various degrees of input correlation and for various values of

on the learning mechanism of adaptive filters this paper highlights both analytically and by simulations some interesting phenomena regarding the behavior of ensemble average learning curves of adaptive filters that may have gone unnoticed among other results the paper shows that even ensemble average learning curves of single tap lms filters actually exhibit two distinct rates of convergence one for the initial time instants and another faster one for later time instants in addition such curves tend to converge faster than predicted by mean square theory and can converge even when a mean square stability analysis predicts divergence these effects tend to be magnified by increasing the step size two of the conclusions that follow from this work are 1 the mean square stability alone may not be the most appropriate performance measure especially for larger step sizes a combination of mean square stability and almost sure a s stability seems to be more appropriate 2 care is needed while interpreting ensemble average curves for larger step sizes the curves can lead to erroneous conclusions unless a large number of experiments are averaged at times of the order of tens of thousands or higher

mean square performance of a family of affine projection algorithms affine projection algorithms are useful adaptive filters whose main purpose is to speed the convergence of lms type filters most analytical results on affine projection algorithms assume special regression models or gaussian regression data the available analysis also treat different affine projection filters separately this paper provides a unified treatment of the mean square error tracking and transient performances of a family of affine projection algorithms the treatment relies on energy conservation arguments and does not restrict the regressors to specific models or to a gaussian distribution simulation results illustrate the analysis and the derived performance expressions

quantifying the effects of dimension on the convergence rate of the lms adaptive fir estimator the convergence rate of an lms adaptive fir filter to an unknown stationary channel may be influenced by the filter parameter dimension as well as by the input signal s characteristics this dimension influence may be of importance in applications such as adaptive acoustic echo cancellation in which the unknown channel is typically modeled as a long fir filter the paper includes the development and proposal of a novel measure of the expected convergence rate of the lms fir filter followed by analysis of this convergence rate measure the analysis indicates that unless the input signal is white the expected convergence rate decreases with increasing dimension down to a limiting value which is determined by the input signal s autocorrelation level

optimal step size of pseudo affine projection algorithm a new statistical analysis model is used to analyze the pseudo affine projection pap algorithm with the optimal step size in this paper by setting the weight error to be zero in the direction of the adaptive weight update the optimal step size is obtained two assumptions for the direction vector are proposed which simplify the analysis of the convergence behavior for the optimal step size pap algorithm under this condition deterministic recursive equations for the weight error and for the mean square error are derived in the direction of the adaptive weight update simulation results are provided to corroborate the analytical results

a stochastic model for a pseudo affine projection algorithm this paper presents a statistical analysis of a pseudo affine projection pap algorithm obtained from the affine projection algorithm ap for a step size alpha 1 and a scalar error signal in the weight update deterministic recursive equations are derived for the mean weight and for the mean square error mse for a large number of adaptive taps n compared to the order p of the algorithm simulations are presented which show good to excellent agreement with the theory in the transient and steady states the pap learning behavior is of special interest in applications where tradeoffs are necessary between convergence speed and steady state misadjustment

fixed point error analysis of the normalized ladder algorithm an attempt is made to analyze the fixed point error performance of the normalized ladder algorithm for autoregressive system identification assuming rounding arithmetic a preliminary simulation study of this algorithm has shown that the bias in the estimated reflection coefficients is much more predominant than the variance of the error in the estimate the study therefore is directed to find a model for predicting the bias in the estimated reflection coefficients the analysis shows that the roundoff errors associated with the square root operations in one of the algorithm equations are mainly responsible for the bias in the estimated reflection coefficients these errors arise because of the normalization procedure that makes the quantities under the square root operations very close to one two main results are presented in the paper 1 a simplified theoretical expression for predicting the average bias in the estimated reflection coefficients at any stage is derived 2 a recursive relation for the average error arising from the finite precision arithmetic in the squared residuals is derived this relation illustrates how the errors made in one stage affect the errors in the succeeding stages simulations are performed to check the theoretical models the experimental results agree very closely with the theoretical predictions

analysis of the normalized lms algorithm with gaussian inputs the lms adaptive filter algorithm requires a priori knowledge of the input power level to select the algorithm gain parameter for stability and convergence since the input power level is usually one of the statistical unknowns it is normally estimated from the data prior to beginning the adaptation process it is then assumed that the estimate is perfect in any subsequent analysis of the lms algorithm behavior in this paper the effects of the power level estimate are incorporated in a data dependent that appears explicitly within the algorithm the transient mean and second moment behavior of the modified lms nlms algorithm are evaluated taking into account the explicit statistical dependence of upon the input data the mean behavior of the algorithm is shown to converge to the wiener weight a constant coefficient matrix difference equation is derived for the weight fluctuations about the wiener weight the equation is solved for a white data covariance matrix and for the adaptive line enhancer with a single frequency input in steady state for small expressions for the misadjustment error are also presented it is shown for the white data covariance matrix case that the averaging of about ten data samples causes negligible degradation as compared to the lms algorithm in the ale application the steady state weight fluctuations are shown to be mode dependent being largest at the frequency of the input

convergence analysis of lms filters with uncorrelated gaussian data statistical analysis of the least mean squares lms adaptive algorithm with uncorrelated gaussian data is presented exact analytical expressions for the steady state mean square error mse and the performance degradation due to weight vector misadjustment are derived necessary and sufficient conditions for the convergence of the algorithm to the optimal wiener solution within a finite variance are derived it is found that the adaptive coefficient which controls the rate of convergence of the algorithm must be restricted to an interval significantly smaller than the domain commonly stated in the literature the outcome of this paper therefore places fundamental limitations on the mse performance and rate of convergence of the lms adaptive scheme

the capacity region of the totally asynchronous multiple access channel it is shown that the capacity region of the asynchronous multiple access channel differs from that of the synchronous channel only by the lack of a convex hull operation

fast recursive least squares transversal filters for adaptive filtering fast transversal filter ftf implementations of recursive least squares rls adaptive filtering algorithms are presented in this paper substantial improvements in transient behavior in comparison to stochastic gradient or lms adaptive algorithms are efficiently achieved by the presented algorithms the true not approximate solution of the rls problem is always obtained by the ftf algorithms even during the critical initialization period first n iterations of the adaptive filter this true solution is recursively calculated at a relatively modest increase in computational requirements in comparison to stochastic gradient algorithms factor of 1 6 to 3 5 depending upon application additionally the fast transversal filter algorithms are shown to offer substantial reductions in computational requirements relative to existing fast rls algorithms such as the fast kalman algorithms of morf ljung and falconer 1976 and the fast ladder lattice algorithms of morf and lee 1977 1981 they are further shown to attain steady state unnormalized or improve upon first n initialization steps the very low computational requirements of the efficient rls solutions of carayannis manolakis and kalouptsidis 1983 finally several efficient procedures are presented by which to ensure the numerical stability of the transversal filter algorithms including the incorporation of soft constraints into the performance criteria internal bounding and rescuing procedures and dynamic range increasing square root normalized variations of the transversal filters

a general formula for channel capacity a formula for the capacity of arbitrary single user channels without feedback not necessarily information stable stationary etc is proved capacity is shown to equal the supremum over all input processes of the input output inf information rate defined as the liminf in probability of the normalized information density the key to this result is a new converse approach based on a simple new lower bound on the error probability of m ary hypothesis tests among equiprobable hypotheses a necessary and sufficient condition for the validity of the strong converse is given as well as general expressions for spl epsiv capacity

almost sure convergence of adaptive algorithms by projections a convergence proof is discussed for the normalized least mean square lms algorithm for ergodic inputs the approach is based on interpreting the algorithm as a sequence of relaxed projection operators by which the key contraction property is derived the proof technique is strongly motivated by physical intuition and provides additional insight into lms type algorithms under ergodic inputs embedded in the development is a slight generalization to a random time varying gain parameter this allows the incorporation of variations such as the lms and signed lms algorithms

transient analysis of data normalized adaptive filters this paper develops an approach to the transient analysis of adaptive filters with data normalization among other results the derivation characterizes the transient behavior of such filters in terms of a linear time invariant state space model the stability of the model then translates into the mean square stability of the adaptive filters likewise the steady state operation of the model provides information about the mean square deviation and mean square error performance of the filters in addition to deriving earlier results in a unified manner the approach leads to stability and performance results without restricting the regression data to being gaussian or white the framework is based on energy conservation arguments and does not require an explicit recursion for the covariance matrix of the weight error vector

asymptotic spectral efficiency of multiuser multisignature cdma in frequency selective channels this paper presents an asymptotic analysis of multisignature code division multiple access cdma in the presence of frequency selective channels we characterize the sum spectral efficiency and spectral efficiency regions for both the optimal and linear minimum mean squared error mmse multiuser receivers both independent and identically distributed i i d signatures and isometric signatures which are orthogonal at each transmitter are considered our results are asymptotic as the number of signatures per user and processing gain both tend to infinity with fixed ratio the spectral efficiency of the mmse receiver is determined from the asymptotic output signal to interference plus noise ratio sinr for isometric signatures our results rely on approximating certain covariance matrices with unitarily invariant matrices that are asymptotically free this approximation is shown to be very accurate through comparison with both simulation and an incremental signature analysis which can be used to compute asymptotic moments also a novel proof of the convergence of the empirical spectral distribution of the signal correlation matrix is presented from these results we derive the optimal coding spreading tradeoff which maximizes the mmse spectral efficiency for the case of a single user with multiple i i d signatures simulation studies demonstrate that the asymptotic results accurately predict the performance of finite size systems of interest the resulting expressions are used to highlight and infer properties of the multisignature cdma system including the benefit of orthogonal relative to i i d signatures and the tradeoff between spectral efficiency and the versatility of providing a variable data rate service through multiple signatures

a unified approach to the steady state and tracking analyses of adaptive filters most adaptive filters are inherently nonlinear and time variant systems the nonlinearities in the update equations tend to lead to difficulties in the study of their steady state performance as a limiting case of their transient performance this paper develops a unified approach to the steady state and tracking analyses of adaptive algorithms that bypasses many of these difficulties the approach is based on the study of the energy flow through each iteration of an adaptive filter and it relies on a fundamental error variance relation

a full mean square analysis of clms for second order noncircular inputs a full mean square transient and steady state analysis of the complex least mean square clms algorithm is provided for strictly linear estimation of general second order noncircular improper gaussian inputs to this end we also consider the performance assessment in terms of the evolution of the complementary mean square error cmse and the complementary covariance pseudocovariance matrix of the weight error vector of clms this makes it possible to measure the degrees of noncircularity of the output error and the weight error vector which arise due to second order noncircularity improperness of the system input and system noise the recently introduced approximate uncorrelating transform which allows for joint direct diagonalization of both the input covariance and complementary covariance matrices with a single singular value decomposition is then employed in order to derive a unified bound on the step size which guarantees the convergence of both the standard mse and the proposed cmse a joint consideration of the standard mean square performance analysis and the proposed complementary performance analysis is shown to provide full second order closed form statistical descriptions of both the transient and steady state performances of clms for second order noncircular improper gaussian input data simulations in the system identification setting support the analysis

error energy bounds for adaptive gradient algorithms the paper establishes robustness optimality and convergence properties of the widely used class of instantaneous gradient adaptive algorithms the analysis is carried out in a purely deterministic framework and assumes no a priori statistical information it employs the cauchy schwarz inequality for vectors in an euclidean space and derives local and global error energy bounds that are shown to highlight as well as explain relevant aspects of the robust performance of adaptive gradient filters along the lines of h sup spl infin theory

adaptive recovery of a chirped sinusoid in noise ii performance of the lms algorithm the authors study the ability of the exponentially weighted recursive least square rls algorithm to track a complex chirped exponential signal buried in additive white gaussian noise power p sub n the signal is a sinusoid whose frequency is drifting at a constant rate psi lt is recovered using an m tap adaptive predictor five principal aspects of the study are presented the methodology of the analysis proof of the quasi deterministic nature of the data covariance estimate r k a new analysis of rls for an inverse system modeling problem a new analysis of rls for a deterministic time varying model for the optimum filter and an evaluation of the residual output mean square error mse resulting from the nonoptimality of the adaptive predictor the misadjustment in terms of the forgetting rate beta of the rls algorithm it is shown that the misadjustment is dominated by a lag term of order beta sup 2 and a noise term of order beta thus a value beta sub opt exists which yields a minimum misadjustment it is proved that beta sub opt m 1 rho psi sup 2 sup 1 3 and the minimum misadjustment is equal to 3 4 p sub n m 1 beta sub opt where rho is the input signal to noise ratio snr

the behavior of lms and nlms algorithms in the presence of spherically invariant processes the behavior of least mean square lms and normalized least mean square nlms algorithms with spherically invariant random processes sirps as excitations is shown many random processes fall into this category and sirps closely resemble speech signals the most pertinent properties of these random processes are summarized the lms algorithm is introduced and the first and second order moments of the weight error vector between the wiener solution and the estimated solution are shown the behavior of the nlms algorithm is obtained and the first and second order moments of the weight error vector are calculated the results are verified by comparison with known results when a white noise process and a colored gaussian process are used as input sequences some simulation results for a k sub 0 process are then shown

a theory on the convergence behavior of the affine projection algorithm in this paper we present a theoretical convergence analysis of the affine projection algorithm apa based on the arguments of energy conservation although the apa and its convergence analysis have been widely studied the dependency of weight error vector on past noise is usually neglected for simplicity to obtain accurate theoretical results for the apa we here consider the dependency between the weight error vector and past noise in the mean square analysis presented by shin and sayed in mean square performance of a family of affine projection algorithms ieee transactions on signal processing vol 52 no 1 pp 90 102 january 2004 through this work we can also theoretically analyze the behavior of the periodic apa which updates its weights periodically simulation results show that our theoretical results coincide closely with simulations

a global gradient noise covariance expression for stationary real gaussian inputs supervised parameter adaptation in many artificial neural networks is largely based on an instantaneous version of gradient descent called the least mean square lms algorithm this paper considers only neural models which are linear with respect to their adaptable parameters and has two major contributions first it derives an expression for the gradient noise covariance under the assumption that the input samples are real stationary gaussian distributed but can be partially correlated this expression relates the gradient correlation and input correlation matrices to the gradient noise covariance and explains why the gradient noise generally correlates maximally with the steepest principal axis and minimally with the one of the smallest curvature regardless of the magnitude of the weight error second a recursive expression for the weight error correlation matrix is derived in a straightforward manner using the gradient noise covariance and comparisons are drawn with the complex lms algorithm

on the convergence behavior of the lms and the normalized lms algorithms it is shown that the normalized least mean square nlms algorithm is a potentially faster converging algorithm compared to the lms algorithm where the design of the adaptive filter is based on the usually quite limited knowledge of its input signal statistics a very simple model for the input signal vectors that greatly simplifies analysis of the convergence behavior of the lms and nlms algorithms is proposed using this model answers can be obtained to questions for which no answers are currently available using other perhaps more realistic models examples are given to illustrate that even quantitatively the answers obtained can be good approximations it is emphasized that the convergence of the nlms algorithm can be speeded up significantly by employing a time varying step size the optimal step size sequence can be specified a priori for the case of a white input signal with arbitrary distribution

multiple access channels with memory with and without frame synchronism the capacity region of frame synchronous and frame asynchronous discrete two user multiple access channels with finite memory is obtained frame synchronism refers to the ability of the transmitters to send their code words in unison the absence of frame synchronism in memoryless multiple access channels is known to result in the removal of the convex hull operation from the expression of the capacity region it is shown that when the channel has memory frame asynchronism rules out nonstationary inputs to achieve any point in the capacity region thereby allowing only coding strategies that involve cooperation in the frequency domain but not in the time domain this restriction drastically reduces the capacity region of some multiple access channels with memory and in particular the total capacity of the channel which is invariant to frame asynchronism for memoryless channels

normalized data nonlinearities for lms adaptation properly designed nonlinearly modified lms algorithms in which various quantities in the stochastic gradient estimate are operated upon by memoryless nonlinearities have been shown to perform better than the lms algorithm in system identification type problems the authors investigate one such algorithm given by w sub k l w sub k spl mu d sub k w sub k sup t x sub k x sub k f x sub k in which the function f x sub k is a scalar function of the sum of the squares of the n elements of the input data vector x sub k this form of algorithm generalizes the so called normalized lms nlms algorithm they evaluate the expected behavior of this nonlinear algorithm for both independent input vectors and correlated gaussian input vectors assuming the system identification model by comparing the nonlinear algorithm s behavior with that of the lms algorithm they then provide a method of optimizing the form of the nonlinearity for the given input statistics in the independent input case they show that the optimum nonlinearity is a single parameter version of the nlms algorithm with an additional constant in the denominator and show that this algorithm achieves a lower excess mean square error mse than the lms algorithm with an equivalent convergence rate additionally they examine the optimum step size sequence for the optimum nonlinear algorithm and show that the resulting algorithm performs better and is less complex to implement than the optimum step size algorithm derived for another form of the nlms algorithm simulations verify the theory and the predicted performance improvements of the optimum normalized data nonlinearity algorithm

mean square deviation analysis of affine projection algorithm this paper presents an improved mean square deviation msd analysis of the standard affine projection algorithm apa based on two distinctive features first the propagation model of the error covariance includes the cross correlation between the current weight error vector and the prior measurement noises associated with the reused inputs such a cross correlation has merely been considered previously second the analysis based on n most recent accumulated iterations rather than a typical analysis based on a current single iteration is suggested to reveal a previously unseen phenomenon where n denotes the tap length of the filter simulation results are in better agreement with the proposed theoretical results than the previous theoretical ones over a wide range of parameters such as tap length projection order and step size

capacity region of the asynchronous gaussian vector multiple access channel in this paper we derive explicit expressions for the capacity region of the two user symbol asynchronous gaussian vector multiple access channel verdu considered the case where each user linearly modulates a fixed waveform in each symbol period where the symbol periods for the users are not perfectly aligned at the receiver he derived explicit capacity region expressions for the case where the transmitters have knowledge of the mutual offset and also for the case where the transmitters have no knowledge of the mutual offset in this paper we extend verdu s results to allow each user to linearly modulate a set of orthonormal waveforms instead of a single waveform in each symbol period and with no restrictions imposed on the waveforms we consider group power constraints which include individual sum power constraints as orthonormal waveforms assigned to each user may come from different frequency bands with different power constraints similar to the case where each user is allowed to linearly modulate only a single waveform our results hold regardless of whether or not the transmitters are frame synchronous in addition we present some results that are necessary to numerically compute the capacity region expressions with general purpose convex optimization algorithms next we simplify the capacity region expression when there are only individual sum power constraints and when the transmitters know the mutual offset we also prove a sufficient condition for a similar simplification to hold when the transmitters have no knowledge of the mutual offset finally we consider a specialized algorithm to numerically compute the simplified capacity region expression when the transmitters know the mutual offset