energy weighted linear predictive spectral estimation a new method combining robustness and high resolution a new method for estimating the ar process coefficients for spectral estimation is introduced the m selected coefficients achieve the minimum square error in fitting a recursion among the estimated covariance elements of the data which would be satisfied exactly if the statistics were known exactly and the data process fit the model assumptions mth order ar this minimization is shown to be identical to minimizing the average one step prediction error with adaptive weights determined by the energy of the measured data as in the burg algorithm forward and backward sweeps are averaged and the levinson recursion is employed spectra computed from short deterministic and noisy data are compared with computed burg spectra and show improvement in bias resolution and robustness of peak detection

an approximation theoretic approach to maximum entropy spectral analysis it is shown that the theory of best approximation in weighted mean square can be used to provide a mathematical foundation for the analysis and extension of the maximum entropy method of spectral analysis

an optimum tapered burg algorithm for linear prediction and spectral analysis several problems associated with the burg algorithm are for the most part alleviated by the use of an appropriate energy taper used in the derivation of the algorithm the tapered burg technique uses a weighted forward backward least squares fit to the parameters of the all pole model subject to levinson s recursion constraint based on a minimum average variance of the estimated frequency of a real sinusoid an optimum taper is derived finally the performance of the optimally tapered burg technique is compared with the least squares yule walker and the usual burg techniques

an experimental comparison between maximum entropy and minimum relative entropy spectral analysis it is known that the minimum relative entropy spectral analysis mresa of j e shore and r w johnson 1980 and j e shore 1981 provides an even better power spectrum estimation than the maximum entropy spectral analysis mesa by its explicit inclusion of a prior estimate that catches certain important shape features of the unknown true power spectrum since the kinds of prior estimates that catch certain important shape features of the unknown true power spectrum are generally hard to determine a comparison between mesa and mresa is undertaken in order to justify mresa the results indicate that unless the shape information of the true spectrum is correctly caught by the prior spectrum it is safer to use mesa

iterative image reconstruction algorithms based on cross entropy minimization the related problems of minimizing the functionals f x alpha kl y px 1 alpha kl p x and g x alpha kl px y 1 alpha kl x p respectively over the set of vectors x or 0 are considered kl a b is the cross entropy or kullback leibler distance between two nonnegative vectors a and b iterative algorithms for minimizing both functionals using the method of alternating projections are derived a simultaneous version of the multiplicative algebraic reconstruction technique mart algorithm called smart is introduced and its convergence is proved

least squares identification of the dynamic relation between the electromyogram and joint moment the dynamic relation between surface emg of an agonist and isometric joint moment is important in models for the prediction of dynamic joint moment from emg an efficient approach for determining the best fitting linear transfer function relating emg and isometric moment using a least squares approach is presented and illustrated for a second order model

nonextensive triangle equality and other properties of tsallis relative entropy minimization kullback leibler relative entropy has unique properties in cases involving distributions resulting from relative entropy minimization tsallis relative entropy is a one parameter generalization of kullback leibier relative entropy in the nonextensive thermostatistics in this paper we present the properties of tsallis relative entropy minimization and present some differences with the classical case in the representation of such a minimum relative entropy distribution we highlight the use of the q product an operator that has been recently introduced to derive the mathematical structure behind the tsallis statistics one of our main results is the generalization of triangle equality of relative entropy minimization to the nonextensive case

multichannel relative entropy spectrum analysis a new relative entropy method is presented for estimating the power spectral density matrix for multichannel data given correlation values for linear combinations of the channels and given an initial estimate of the spectral density matrix a derivation of the method from the relative entropy principle is given the basic approach is similar in spirit to the multisignal relative entropy spectrum analysis of johnson and shore but the results differ significantly because the present method does not arbitrarily require the final distributions of the various channels to be independent for the special case of separately estimating the spectra of a signal and noise given the correlations of their sum multichannel relative entropy spectrum analysis turns into a two stage procedure first a smooth power spectrum model is fitted to the correlations of the signal plus noise then final estimates of the spectra and cross spectra are obtained through linear filtering for the special case where p uniformly spaced correlations are known and where the initial estimate of the signal plus noise spectrum is all pole with order p or less this method fits a standard maximum entropy autoregressive spectrum to the noisy correlations then linearly filters to calculate the signal and noise spectra and cross spectra consideration is given to the case where only an initial estimate of the noise power spectrum is available an illustrative example is given

minimum relative entropy forward probabilistic modeling the pioneering work of jaynes in bayesian maximum entropy methods has been successfully explored in many disciplines the principle of maximum entropy pme is a powerful and versatile tool of inferring a probability distribution from constraints that do not completely characterize that distribution minimum relative entropy mre is a method which has all the important attributes of the maximum entropy approach with the advantage that prior information may be easily included in this paper we use mre to determine the prior probability density function pdf of a set of model parameters based on limited information the resulting pdf is used in monte carlo simulations to provide expected values in field variables such as concentration and confidence limits we compare the probabilistic results from a traditional advection dispersion ade model based on volumetric averaging concepts with that of a model based on the assumption that the hydraulic conductivity is a stationary stochastic process the results suggest that although naff s 1990 model satisfies the observed data to a better degree than ade model the upper and lower confidence bands about the mean value are larger than the ade results this result we attribute to the fact that naff s 1990 model simply contains more parameters each of which is unknown and has to be estimated there is no statistical difference between the expected values of second spatial moments for the two models the examples presented in this paper illustrate problems associated with assigning gaussian pdfs as priors in a probabilistic model first such an assumption for the input parameters actually injects more information into the problem than may actually exist whether consciously or unconsciously this fact is borne out by comparison with minimum relative entropy theory second the output parameters as suggested from the monte carlo analysis cannot be assumed to be gaussian distributed even when the prior pdf is gaussian in form in a practical setting the significance of this result and the approximation of gaussian form would depend on the toxicity and environmental standards that apply to the problem

maximum entropy and spectral estimation a review the method of maximum entropy is reviewed with emphasis on its relationship to entropy rate wiener filters autoregressive processes extrapolation the levinson algorithm lattice all pole and all pass filters and stability

multisignal minimum cross entropy spectrum analysis with weighted initial estimates this paper presents a generalization of multisignal minimum cross entropy spectrum analysis multisignal relative entropy spectrum analysis multisignal resa a method for simultaneously estimating a number of power spectra when an initial prior estimate of each is available and new information is obtained in the form of values of the sum of their autocorrelation functions the generalization involves attaching to each initial spectrum estimate a possibly frequency dependent weighting parameter that indicates its relative reliability i e the relative degree of belief associated with the initial estimate mathematical properties of the generalized method are discussed and illustrative numerical examples are given

a general method of minimum cross entropy spectral estimation a new general approach to estimating power spectra is presented based on minimizing a cross entropy functional involving spectral densities examples are worked out using this approach as well as shore s 5 minimum cross entropy spectral analysis mcesa and burg s 7 maximum entropy spectral analysis mesa the results show that aside from its wider applicability ease of implementation and pedagogical advantage the method performs better than mesa and equally well or better than mcesa

general entropy criteria for inverse problems with applications to data compression pattern classification and cluster analysis minimum distance approaches are considered for the reconstruction of a real function from finitely many linear functional values an optimal class of distances satisfying an orthogonality condition analogous to that enjoyed by linear projections in hilbert space is derived these optimal distances are related to measures of distances between probability distributions recently introduced by c r rao and t k nayak 1985 and possess the geometric properties of cross entropy useful in speech and image compression pattern classification and cluster analysis several examples from spectrum estimation and image processing are discussed

mismatched estimation and relative entropy a random variable with distribution p is observed in gaussian noise and is estimated by a mismatched minimum mean square estimator that assumes that the distribution is q instead of p this paper shows that the integral over all signal to noise ratios snrs of the excess mean square estimation error incurred by the mismatched estimator is twice the relative entropy d p q in nats this representation of relative entropy can be generalized to nonreal valued random variables and can be particularized to give new general representations of mutual information in terms of conditional means inspired by the new representation we also propose a definition of free relative entropy which fills a gap in and is consistent with the literature on free probability

multichannel ar spectrum estimation the optimum approach in the reflection coefficient domain a new approach for multichannel autoregressive ar spectrum estimation is introduced based on an iterative gradient algorithm in the context of dynamic programming the approach is shown to be optimum in the reflection coefficient domain whereas the multichannel ar nuttall strand algorithm is shown to be suboptimum the approach guarantees a stable multichannel ar filter and a nonnegative spectrum matrix estimate it is demonstrated via simulation examples that such drawbacks of the nuttall strand method as frequency bias and line splitting may be alleviated by the optimum approach however the improved performance is achieved at the expense of considerably more computational effort and time the single channel fougere descent method may be seen as a special case of the proposed multichannel ar approach

spectral estimation of quasi periodic data a computationally efficient algorithm for estimating the autocorrelation function and or the power spectrum of periodic quasi periodic time series when information about its period is available is described the autocorrelation matrix obtained from the sample covariance matrix using this method is a good approximation of the maximum likelihood estimate of the autocorrelation matrix simulation results involving sinusoids in noise and speech data are presented

on mutual information likelihood ratios and estimation error for the additive gaussian channel this paper considers the model of an arbitrarily distributed signal x observed through an added independent white gaussian noise w y x w new relations between the minimal mean square error of the noncausal estimator and the likelihood ratio between y and w are derived this is followed by an extended version of a recently derived relation between the mutual information i x y and the minimal mean square error these results are applied to derive infinite dimensional versions of the fisher information and the de bruijn identity a comparison between the causal and noncausal estimation errors yields a restricted form of the logarithmic sobolev inequality the derivation of the results is based on the malliavin calculus

quadratically constrained minimum cross entropy analysis quadratically constrained minimum cross entropy problem has recently been studied by zhang and brockett through an elaborately constructed dual in this paper we take a geometric programming approach to analyze this problem unlike zhang and brockett we separate the probability constraint from general quadratic constraints and use two simple geometric inequalities to derive its dual problem furthermore by using the dual perturbation method we directly prove the strong duality theorem and derive a dual to primal conversion formula as a by product the perturbation proof gives us insights to develop a computation procedure that avoids dual non differentiability and allows us to use a general purpose optimizer to find ane optimal solution for the quadratically constrained minimum cross entropy analysis

entropy optimization models with convex constraints in this paper we study the minimum cross entropy optimization problem subject to a general class of convex constraints using a simple geometric inequality and the conjugate inequality we demonstrate how to directly construct a partial geometric dual program which allows us to apply the dual perturbation method to derive the strong duality theorem and a dual to primal conversion formula this approach generalizes the previous results of linearly quadratically and entropically constrained cross entropy optimization problems and provides a platform for using general purpose optimizers to generate optimal solution pair to the problem

minimum relative entropy inversion theory and application to recovering the release history of a groundwater contaminant in this paper we show that given prior information in terms of a lower and upper bound a prior bias and constraints in terms of measured data minimum relative entropy mre yields exact expressions for the posterior probability density function pdf and the expected value of the linear inverse problem in addition we are able to produce any desired confidence intervals in numerical simulations we use the mre approach to recover the release and evolution histories of plume in a one dimensional constant known velocity and dispersivity system for noise free data we find that the reconstructed plume evolution history is indistinguishable from the true history an exact match to the observed data is evident two methods are chosen for dissociating signal from a noisy data set the first uses a modification of mre for uncertain data the second method uses presmoothing by fast fourier transforms and butterworth filters to attempt to remove noise from the signal before the noise free variant of mre inversion is used both methods appear to work very well in recovering the true signal and qualitatively appear superior to that of skaggs and kabala 1994 we also solve for a degenerate case with a very high standard deviation in the noise the recovered model indicates that the mre inverse method did manage to recover the salient features of the source history once the plume source history has been developed future behavior of a plume can then be cast in a probabilistic framework for an example simulation the mre approach not only was able to resolve the source function from noisy data but also was able to correctly predict future behavior

minimum cross entropy spectral analysis the principle of minimum cross entropy minimum directed divergence minimum discrimination information minimum relative entropy is summarized discussed and applied to the classical problem of estimating power spectra given values of the autocorrelation function this new method differs from previous methods in its explicit inclusion of a prior estimate of the power spectrum and it reduces to maximum entropy spectral analysis as a special case the prior estimate can be viewed as a means of shaping the spectral estimator cross entropy minimization yields a family of shaped spectral estimators consistent with known autocorrelations results are derived in two equivalent ways once by minimizing the cross entropy of underlying probability densities and once by arguments concerning the cross entropy between the input and output of linear filters several example minimum cross entropy spectra are included

sensory contributions to stabilization of trunk posture in the sagittal plane trunk stabilization is required to control posture and movement during daily activities various sensory modalities such as muscle spindles golgi tendon organs and the vestibular system might contribute to trunk stabilization and our aim was to assess the contribution of these modalities to trunk stabilization in 35 healthy subjects upper body sway was evoked by continuous unpredictable force controlled perturbations to the trunk in the anterior direction subjects were instructed to either maximally resist the perturbation or to relax but remain upright with eyes closed frequency response functions frfs of admittance the amount of movement per unit of force applied and reflexes the modulation of trunk extensor activity per unit of trunk displacement were obtained to these frfs we fitted physiological models to estimate intrinsic trunk stiffness and damping as well as feedback gains and delays the different model versions were compared to assess which feedback loops contribute to trunk stabilization intrinsic stiffness and damping and muscle spindle short delay feedback alone were sufficient to accurately describe trunk stabilization but only with unrealistically low reflex delays addition of muscle spindle acceleration feedback or inhibitory golgi tendon organ feedback yielded realistic delays and improved the model fit with a significantly better model fit with acceleration feedback addition of vestibular feedback did not improve the model fit in conclusion muscle spindle feedback and intrinsic mechanical properties are sufficient to describe trunk stabilization in the sagittal plane under small mechanical perturbations provided that muscle spindles encode acceleration in addition to velocity and position information

a moment based variational approach to tomographic reconstruction we describe a variational framework for the tomographic reconstruction of an image from the maximum likelihood ml estimates of its orthogonal moments we show how these estimated moments and their correlated error statistics can be computed directly and in a linear fashion from given noisy and possibly sparse projection data moreover thanks to the consistency properties of the radon transform this two step approach moment estimation followed by image reconstruction can be viewed as a statistically optimal procedure furthermore by focusing on the important role played by the moments of projection data we immediately see the close connection between tomographic reconstruction of nonnegative valued images and the problem of nonparametric estimation of probability densities given estimates of their moments taking advantage of this connection our proposed variational algorithm is based on the minimization of a cost functional composed of a term measuring the divergence between a given prior estimate of the image and the current estimate of the image and a second quadratic term based on the error incurred in the estimation of the moments of the underlying image from the noisy projection data we show that an iterative refinement of this algorithm leads to a practical algorithm for the solution of the highly complex equality constrained divergence minimization problem we show that this iterative refinement results in superior reconstructions of images from very noisy data as compared with the classical filtered back projection fbp algorithm

directed information causal estimation and communication in continuous time a notion of directed information between two continuous time processes is proposed a key component in the definition is taking an infimum over all possible partitions of the time interval which plays a role no less significant than the supremum over space partitions inherent in the definition of mutual information properties and operational interpretations in estimation and communication are then established for the proposed notion of directed information for the continuous time additive white gaussian noise channel it is shown that duncan s classical relationship between causal estimation error and mutual information continues to hold in the presence of feedback upon replacing mutual information by directed information a parallel result is established for the poisson channel the utility of this relationship is demonstrated in computing the directed information rate between the input and output processes of a continuous time poisson channel with feedback where the channel input process is constrained to be constant between events at the channel output finally the capacity of a wide class of continuous time channels with feedback is established via directed information characterizing the fundamental limit on reliable communication

cross entropy approximation of structured gaussian covariance matrices we apply two variations of the principle of minimum cross entropy the kullback information measure to fit parameterized probability density models to observed data densities for an array beamforming problem with p incident narrowband point sources sensors and colored noise both approaches yield eigenvector fitting methods similar to that of the music algorithm and of the oblique transformation in factor analysis furthermore the corresponding cross entropies ce are related to the mdl model order selection criterion

a new look at entropy for solving linear inverse problems entropy based methods are widely used for solving inverse problems particularly when the solution is known to be positive here we address linear ill posed and noisy inverse problems of the form z ax n with a general convex constraint x spl isin x where x is a convex set although projective methods are well adapted to this context we study alternative methods which rely highly on some information based criteria our goal is to clarify the role played by entropy in this field and to present a new point of view on entropy using general tools and results coming from convex analysis we present then a new and broad scheme for entropic based inversion of linear noisy inverse problems this scheme was introduced by navaza in 1985 in connection with a physical modeling for crystallographic applications and further studied by dacunha castelle and gamboa 1990 important features of this paper are i a unified presentation of many well known reconstruction criteria ii proposal of new criteria for reconstruction under various prior knowledge and with various noise statistics iii a description of practical inversion of data using the aforementioned criteria and iv a presentation of some reconstruction results

arma models prewhitening and minimum cross entropy the problem of spectral estimation on the basis of observations from a finite stretch of a stationary time series is considered in connection with knowledge of a prior estimate of the spectral density a reasonable posterior spectral density estimate would be the density that is closest to the prior according to some measure of divergence while at the same time being compatible with the data the cross entropy has often been proposed to serve as such a measure of divergence a correction of the original minimum cross entropy spectral analysis mcesa method of j e shore see ieee trans acoust speech signal process vol 29 p 230 7 1981 to traditional prewhitening techniques and to autoregressive moving average arma models is pointed out and a fast approximate solution of the minimum cross entropy problem is proposed the solution is in a standard multiplicative form that is the posterior is equal to the prior multiplied by a correction factor

improved spectrum performance via a data adaptive weighted burg technique a new method for generating autoregressive ar parameters for spectral estimation is introduced and shown to be a data adaptive weighted burg dab technique the respective ar coefficient estimates minimize a weighted measure of the squared forward and backward linear predictive errors with weights that are proportional to the common energy of the forward and backward process realizations and result in a stable filter representation spectra are computed comparing the new method to other weighted burg techniques e g rectangular hamming and optimum parabolic weighting and demonstrate performance improvements in frequency bias and resolution no evidence of a line splitting tendency is exhibited however somewhat greater spectral variability is exhibited by the dab method in processing nonsinusoidal signals the provision of asymptotically unbiased ar parameter estimates is presented

cross entropy minimization given fully decomposable subset and aggregate constraints corresp the principle of maximum entropy and the principle of minimum cross entropy minimum directed divergence minimum discrimination information have been applied recently to problems in queuing theory and computer system performance modeling these information theoretic principles estimate probability distributions based on information in the form of known expected values in the case of queuing theory and computer system modeling the known expected values arise from rate balance equations this correspondence concerns situations in which the system state probabilities decompose into disjoint subsets and in which the known expected values are either expectations conditional on a specific subset or expectations involving aggregate subset probabilities new properties of minimum cross entropy distributions are derived and an efficient method of computing these distributions is derived computational examples are included in the case of queuing theory and computer system modeling the disjoint subsets correspond to internal device states and the aggregate probabilities correspond to overall device states the results here apply when one has both rate balance equations for device equilibrium involving internal device state probabilities as well as rate balance equations for system equilibrium involving aggregate device state probabilities

rate distortion bounds for quotient based distortions with application to itakura saito distortion measures in many linear predictive coding lpc speech compression systems the encoding of the lpc parameters is performed using a product code book scheme one possible approach consists in organizing the set of lpc reproduction models as the cartesian product of a vector code book describing the shape of each reproduction lpc model and a scalar codebook describing the gain or energy we first present a formal development of rate distortion theoretic results for distortion functions based on quotients of input and reproduction symbols we also obtain theoretical bounds for the rate distortion function of the gain term of lpc systems which depending on the quantization scheme can use about 10 25 percent of the transmission rate as well as asymptotic small distortions approximations to the performance of the optimal scalar quantizer for these gain terms when the overall fidelity criterion is the itakura saito distortion measure which is a member of the class of quotient distortion measures the approximations and bounds are compared finally with experimental results

myoelectric activity detection during a sit to stand movement using threshold methods myoelectric signals recorded via surface electrodes contain rich muscle activity information that is beneficial for both clinical diagnosis and biomedical research when synchronized with the kinematic data these signals provide investigators an insight into muscle activation sequence onset levels and periods a primary difficulty with the analysis and interpretation of electromyography emg signals lies in the inherent stochastic nature of the emg process which arises from its biological variability as well as noise added during the collection process various techniques for muscle onset and activity detection from the myoelectric signal have been proposed in the literature our focus in this study is myoelectric activity detection from emg signals collected during sit to stand sts and stand to sit stst movements we explore a previously established double threshold detection method and compare its results with a novel detection scheme based on the energy of the signal accordingly emg signals from four lower extremity muscles and synchronized kinematic data were collected for 180 trials of sts and stst movements performed in the laboratory detection thresholds above baseline in the case of both algorithms were computed and analyzed using a 2 detectors x 4 activity thresholds repeated measures analysis of variance our statistical analysis revealed that the energy detection method performed similarly to the double threshold method while both methods required a considerably higher threshold above baseline for detection

a computationally compact divergence measure for speech processing the directed divergence which is a measure based on the discrimination information between two signal classes is investigated a simplified expression for computing the directed divergence is derived for comparing two gaussian autoregressive processes such as those found in speech this expression alleviates both the computational cost reduced by two thirds and the numerical problems encountered in computing the directed divergence in addition the simplified expression is compared with the itakura saito distance which asymptotically approaches the directed divergence although the expressions for these two distances closely resemble each other only moderate correlations between the two were found on a set of actual speech data

mutual information relative entropy and estimation in the poisson channel let be a nonnegative random variable and let the conditional distribution of a random variable given be poisson for a parameter we identify a natural loss function such that 1 the derivative of the mutual information between and with respect to is equal to the minimum mean loss in estimating based on regardless of the distribution of 2 when is estimated based on by a mismatched estimator that would have minimized the expected loss had the integral over all values of of the excess mean loss is equal to the relative entropy between and for a continuous time setting where is a nonnegative stochastic process and the conditional law of given is that of a non homogeneous poisson process with intensity function under the same loss function 1 the minimum mean loss in causal filtering when is equal to the expected value of the minimum mean loss in noncausal filtering smoothing achieved with a channel whose parameter is uniformly distributed between 0 and bridging the two quantities is the mutual information between and 2 this relationship between the mean losses in causal and noncausal filtering holds also in the case where the filters employed are mismatched i e optimized assuming a law on which is not the true one bridging the two quantities in this case is the sum of the mutual information and the relative entropy between the true and the mismatched distribution of thus relative entropy quantifies the excess estimation loss due to mismatch in this setting these results are parallel to those recently found for the gaussian channel the i mmse relationship of guo the relative entropy and mismatched estimation relationship of verdu and the relationship between causal and noncasual mismatched estimation of weissman

minimum cross entropy pattern classification and cluster analysis this paper considers the problem of classifying an input vector of measurements by a nearest neighbor rule applied to a fixed set of vectors the fixed vectors are sometimes called characteristic feature vectors codewords cluster centers models reproductions etc the nearest neighbor rule considered uses a non euclidean information theoretic distortion measure that is not a metric but that nevertheless leads to a classification method that is optimal in a well defined sense and is also computationally attractive furthermore the distortion measure results in a simple method of computing cluster centroids our approach is based on the minimization of cross entropy also called discrimination information directed divergence k l number and can be viewed as a refinement of a general classification method due to kullback the refinement exploits special properties of cross entropy that hold when the probability densities involved happen to be minimum cross entropy densities the approach is a generalization of a recently developed speech coding technique called speech coding by vector quantization

spectral expressions of information measures of gaussian time series and their relation to aic and cat interrelations among the spectral expressions of the information measures of kullback leibler 1959 and renyi 1961 for discrimination between two stationary gaussian time series are discussed the spectral expression of fisher s information rate matrix is also treated as well as two intuitively acceptable discrimination functions it is shown that all of them are equivalent except for scalar multiplication and are expressed by fisher s information rate matrix in the sense of their second order taylor series approximation finally a relation between two criteria for order determination of models for time series data namely h akaike s 1974 information criterion aic and the criterion of autoregressive transfer functions cat is discussed in connection with these spectral expressions

comments on a general method of minimum cross entropy spectral estimation in a recent correspondence tzannes et al 1 introduced a new form of minimum cross entropy mce spectral analysis based on the observation that normalized spectra and symmetric probability density functions pdf s are axiomatically indistinguishable thus the form of the minimum cross entropy posterior pdf can be used as a model for spectral estimation in this correspondence we derive the same result without using a lagrange multiplier methodology we also establish the connection of this estimate to the cepstral representation of the signal and propose an efficient algorithm not requiring explicit solution of nonlinear equations

identifying intrinsic and reflexive contributions to low back stabilization motor control deficits have been suggested as potential cause and or effect of a specific chronic low back pain and its recurrent behavior therefore the goal of this study is to identify motor control in low back stabilization by simultaneously quantifying the intrinsic and reflexive contributions upper body sway was evoked using continuous force perturbations at the trunk while subjects performed a resist or relax task frequency response functions frfs and coherences of the admittance kinematics and reflexes semg were obtained in comparison with the relax task the resist task resulted in a 61 decrease in admittance and a 73 increase in reflex gain below 1 1 hz intrinsic and reflexive contributions were captured by a physiologically based neuromuscular model including proprioceptive feedback from muscle spindles position and velocity and golgi tendon organs force this model described on average 90 of the variance in kinematics and 39 of the variance in semg while resulting parameter values were consistent over subjects

comparison of multi objective optimization methodologies for engineering applications computational models describing the behavior of complex physical systems are often used in the engineering design field to identify better or optimal solutions with respect to previously defined performance criteria multi objective optimization problems arise and the set of optimal compromise solutions pareto front has to be identified by an effective and complete search procedure in order to let the decision maker the designer to carry out the best choice four multi objective optimization techniques are analyzed by describing their formulation advantages and disadvantages the effectiveness of the selected techniques for engineering design purposes is verified by comparing the results obtained by solving a few benchmarks and a real structural engineering problem concerning an engine bracket of a car

which is the better entropy expression for speech processing s log s or log s in maximum entropy spectral analysis mesa one maximizes the integral of logs f where s f is a power spectrum the resulting spectral estimate which is equivalent to that obtained by linear prediction and other methods is popular in speech processing applications an alternative expression s f logs f is used in optical processing and elsewhere this paper considers whether the alternative expression leads to spectral estimates useful in speech processing we investigate the question both theoretically and empirically the theoretical investigation is based on generalizations of file two estimates the generalizations take into account prior estimates of the unknown power spectrum it is shown that both estimates result from applying a generalized version of the principle of maximum entropy but they differ concerning the quantities that are treated as random variables the empirical investigation is based on speech synthesized using the different spectral estimates although both estimates lead to intelligible speech speech based on the mesa estimate is qualitatively superior

a convex projections approach to spectral analysis the projection onto convex sets pocs method is applied to the classical problem of estimating a power spectrum given values of the autocorrelation function acf this new method can use any number of linear and or nonlinear constraints on the spectrum to be estimated the measured acf constraint the amplitude limit constraint and the reference spectrum constraint are introduced and the corresponding projection operators are established computer simulation results indicate that the pocs spectral analysis performs well

pointwise relations between information and estimation in gaussian noise many of the classical and recent relations between information and estimation in the presence of gaussian noise can be viewed as identities between expectations of random quantities these include the relationship between mutual information and minimum mean square error i mmse of guo the relative entropy and mismatched estimation relationship of verdu the relationship between causal estimation and mutual information of duncan and its extension to the presence of feedback by kadota the relationship between causal and non casual estimation of guo and its mismatched version of weissman we dispense with the expectations and explore the nature of the pointwise relations between the respective random quantities the pointwise relations that we find are as succinctly stated as and give considerable insight into the original expectation identities as an illustration of our results consider duncan s 1970 discovery that the mutual information is equal to the causal mmse in the additive white gaussian noise channel which can equivalently be expressed saying that the difference between the input output information density and half the causal estimation error is a zero mean random variable regardless of the distribution of the channel input we characterize this random variable explicitly rather than merely its expectation classical estimation and information theoretic quantities emerge with new and surprising roles for example the variance of this random variable turns out to be given by the causal mmse which in turn is equal to twice the mutual information by duncan s result

identification of dynamic myoelectric signal to force models during isometric lumbar muscle contractions a 14 muscle myoelectric signal mes driven muscle force prediction model of the l3 l4 cross section is developed which includes a dynamic mes force relationship and allows for cocontraction model parameters are estimated from mes and moments data recorded during rapid exertions in trunk flexion extension lateral bending and axial twist nine young healthy males participated in the experimental testing the model used in the parameter estimation is of the output error type consistent and physically feasible parameter estimates were obtained by normalizing the rms mes to maximum exertion levels and using nonlinear constrained optimization to minimize a cost function consisting of the trace of the output error covariance matrix model performance was evaluated by comparing measured and mes predicted moments over a series of slow and rapid exertions moment prediction errors were on the order of 25 30 and 40 during attempted trunk flexion extensions lateral bends and axial twists respectively the model and parameter estimation methods developed provide a means to estimate lumbar muscle and spine loads as well as to empirically investigate the use and effects of cocontraction during physical task performances

iterative convex i projection algorithms for maximum entropy and minimum cross entropy computations general multiple constraint minimum cross entropy and maximum entropy problems can be solved by the iterative solution of single constraint subproblems existing theoretical results justify iterative i projection algorithms that may be computationally expedient in some cases an iterative algorithm for j e shore s ibid vol it 28 no 6 p 956 61 nov 1982 minimum cross entropy spectral analysis is developed as an example

hybrid harmony search and artificial bee colony algorithm for global optimization problems harmony search hs is one of the newest and the easiest to code music inspired heuristics for optimization problems in order to enhance the accuracy and convergence rate of harmony search a hybrid harmony search is proposed by incorporating the artificial bee colony algorithm abc the artificial bee colony algorithm is a new swarm intelligence technique inspired by intelligent foraging behavior of honey bees the abc and its variants are used to improve harmony memory hm to compare and analyze the performance of our proposed hybrid algorithms a number of experiments are carried out on a set of well known benchmark global optimization problems the effects of the parameters about the hybrid algorithms are discussed by a uniform design experiment numerical results show that the proposed algorithms can find better solutions when compared to hs and other heuristic algorithms and are powerful search algorithms for various global optimization problems

estimating the entropy of a signal with applications we present a new estimator of the entropy of continuous signals we model the unknown probability density of data in the form of an ar spectrum density and use regularized long ar models to identify the ar parameters we then derive both an analytical expression and a practical procedure for estimating the entropy from sample data we indicate how to incorporate recursive and adaptive features in the procedure we evaluate and compare the new estimator with other estimators based on histograms kernel density models and order statistics finally we give several examples of applications an adaptive version of our entropy estimator is applied to detection of law changes blind deconvolution and source separation

a minimum discrimination information approach for hidden markov modeling an iterative approach for minimum discrimination information mdi hidden markov modeling of information sources is proposed the approach is developed for sources characterized by a given set of partial covariance matrices and for hidden markov models hmms with gaussian autoregressive output probability distributions pds the approach aims at estimating the hmm which yields the mdi with respect to all sources that could have produced the given set of partial covariance matrices each iteration of the mdi algorithm generates a new hmm as follows first a pd for the source is estimated by minimizing the discrimination information measure with respect to the old model over all pds which satisfy the given set of partial covariance matrices then a new model that decreases the discrimination information measure between the estimated pd of the source and the pd of the old model is developed the problem of estimating the pd of the source is formulated as a standard constrained minimization problem in the euclidean space the estimation of a new model given the pd of the source is done by a procedure that generalizes the baum algorithm the mdi approach is shown to be a descent algorithm for the discrimination information measure and its local convergence is proved

modelling of the hamstring muscle group by use of fractional derivatives this paper deals with a viscoelastic model of the hamstring muscle group the model includes fractional derivatives of stretching force and elongation as well as restrictions on the coefficients that follow from the clausius duhem inequality on the basis of a ramp and hold type of experiment four rheological parameters have been calculated by numerical treatment ab initio riemann liouville fractional derivatives were approximated numerically using the grunwald letnikov definition obtained results were verified by use of the laplace transform method the stretching force in time domain involves mittag leffler type function

volume image registration by cross entropy optimization cross entropy ce an information theoretic measure quantifies the difference between two probability density functions this measure is applied to volume image registration when a good prior estimation of the joint distribution of the voxel values of two images in registration is available the ce can be minimized to find an optimal registration if such a prior estimation is not available one seeks the registration which gives a joint distribution different from unlikely ones as much as possible i e the ce is maximized to find an optimal registration when the unlikely distribution is a uniform one ce maximization reduces to joint entropy minimization when the unlikely distribution is proportional to one of the marginal distributions it reduces to conditional entropy minimization when the unlikely distribution is the product of two marginal distributions it degenerates to mutual information maximization these different ces are added together and are used as criteria for image registration the accuracy and robustness of this new approach are tested and compared using a likely joint distribution and various unlikely joint distributions and their combinations

learning algorithm for multimodal optimization we present a new evolutionary algorithm learning algorithm for multimodal optimization the scheme for reproducing a new generation is very simple control parameters of the length of the list of historical best solutions and the learning probability of the current solutions being moved towards the current best solutions and towards the historical ones are used to assign different search intensities to different parts of the feasible area and to direct the updating of the current solutions results of numerical tests on minimization of the 2d schaffer function the 2d shubert function and the 10d ackley function show that this algorithm is effective and efficient in finding multiple global solutions of multimodal optimization problems

approximation theoretic derivation of logarithmic entropy principles for inverse problems and unique extension of the maximum entropy method to incorporate prior knowledge the minimum distance approach for reconstructing a positive function based on knowledge of finitely many linear functional values is examined two important classes of directed distances for signal processing and statistical inference are discussed by imposing conditions analogous to those satisfied by linear projections in hilbert space two logarithmic entropy principles are derived one of these involves the itakura saito distortion measure of communication theory and uniquely extends burg s maximum entropy method to incorporate prior knowledge the other uses the kullback leibler distance of statistics

minimum cross entropy analysis with entropy type constraints abstract in this paper we study the cross entropy optimization problem with entropy type constraints a simple geometric inequality is used to derive its dual problem and to show the strong duality theorem we found this geometric dual is a computationally attractive canonical program that is always consistent a dual to primal conversion formula and a dual perturbation algorithm are also derived for computations

mutual information and minimum mean square error in gaussian channels this paper deals with arbitrarily distributed finite power input signals observed through an additive gaussian noise channel it shows a new formula that connects the input output mutual information and the minimum mean square error mmse achievable by optimal estimation of the input given the output that is the derivative of the mutual information nats with respect to the signal to noise ratio snr is equal to half the mmse regardless of the input statistics this relationship holds for both scalar and vector signals as well as for discrete time and continuous time noncausal mmse estimation this fundamental information theoretic result has an unexpected consequence in continuous time nonlinear estimation for any input signal with finite power the causal filtering mmse achieved at snr is equal to the average value of the noncausal smoothing mmse achieved with a channel whose snr is chosen uniformly distributed between 0 and snr

maximum entropy regularization in inverse synthetic aperture radar imagery the method of maximum entropy is applied to the regularization of inverse synthetic aperture radar isar image reconstructions this is accomplished by considering an ensemble of images with associated allowed probability density functions instead of directly considering the solution to be an image the author takes it to be the a posteriori probability density found by minimizing a regularization functional composed of the usual least squares term and a kullback cross entropy information difference term the desired image is then found as the expectation of this density the basic model of this approach is similar to that used in usual maximum a posteriori analysis and allows for a more general relationship between the image and its configuration entropy than is usually employed in addition it eliminates the need for inappropriate nonnegativity constraints on the generally complex valued image

minimum cross entropy inversion spectrum analysis and classification introduction and discussion of possible geophysical applications abstract the principle of minimum cross entropy mce is a general method of inference about unknown probability densities given information in the form of constraints on expected values the background justification and properties of mce inversion are summarized briefly some recent applications in spectrum analysis and pattern classification are then summarized and some speculations on the application of mce to general inversion problems in geophysics are offered

an efficient computational procedure for solving entropy optimization problems with infinitely many linear constraints a cutting plane type algorithm for solving entropy optimization problems with a finite number of variables but an infinite number of linear constraints is proposed in this paper in each iteration we solve a finite entropy optimization problem and add one more constraint the iterative process ends when an optimal solution is identified a convergence proof under some mild conditions is given an efficient implementation based on a dual approach is also included our preliminary computational experience confirms the efficiency of the proposed method and shows its potential for solving large scale problems

computation of minimum cross entropy spectral estimates an unconstrained dual convex programming method the minimum cross entropy spectral analysis procedure a generalization of maximum entropy spectral analysis is formulated as a convex programming problem and its unconstrained dual convex programming problem is shown in this dual setting the lagrange multipliers are precisely the dual variables and the numerical solution values are easily determined by any of a number of nonlinear programming codes this result vastly simplifies the computation of all such spectral density estimates

a neural net algorithm for multidimensional minimum relative entropy spectral analysis a neural net algorithm is presented to solve the general 1 d or multidimensional minimum relative entropy spectral analysis the problem is formulated as a primal constrained optimization and is reduced to solving an initial value problem of differential equation of lyapunov type the initial value problem of lyapunov system comprises the basis of the neural net algorithm experiments with simulated data convincingly showed that the algorithm did provide the multidimensional minimum relative entropy spectral estimator with the autocorrelation matching property with computational efficiency

properties of cross entropy minimization the principle of minimum cross entropy minimum directed divergence minimum discrimination information is a general method of inference about an unknown probability density when there exists a prior estimate of the density and new information in the form of constraints on expected values various fundamental properties of cross entropy minimization are proven and collected in one place cross entropy s well known properties as an information measure are extended and strengthened when one of the densities involved is the result of cross entropy minimization the interplay between properties of cross entropy minimization as an inference procedure and properties of cross entropy as an information measure is pointed out examples are included and general analytic and computational methods of finding minimum cross entropy probability densities are discussed

spiking band limited traces with a relative entropy algorithm a novel method for the inversion of band limited seismic traces to full bandwidth reflectivity traces is based on a probabilistic spiky model of the reflectivity trace in which position indicators and amplitudes of the spikes occur as random variables and relies on relative entropy inference from information theory first an a priori model for general reflectivity traces in the prospect is derived from nearby wells second the a priori distribution is updated into an a posteriori distribution for the specific trace being studied by the addition of the fourier data of the seismic trace within a passband uncertainty about the fourier coefficients can be accounted for by specification of a noise variance which implicitly is infinite outside the passband the update with relative entropy inference is justified because of its relationship with bayesian inference application of maximum a posteriori map estimation to the a posteriori distribution results in the most likely spiky reflectivity trace of f

rate distortion speech coding with a minimum discrimination information distortion measure an information theory approach to the theory and practice of linear predictive coded lpc speech compression systems is developed it is shown that a traditional lpc system can be viewed as a minimum distortion or nearest neighbor system where the distortion measure is a minimum discrimination information between a speech process model and an observed frame of actual speech this distortion measure is used in an algorithm for computer aided design of block source codes subject to a fidelity criterion to obtain a 750 bits s speech compression system that resembles an lpc system but has a much lower rate a larger memory requirement and requires no on line lpc analysis quantitative and informal subjective comparisons are made among our system and lpc systems

jaynes principle and maximum entropy spectral estimation this paper makes use of jaynes principle to include prior information into a maximum entropy spectral estimate such information is often available but is not included in the usual maximum entropy method the paper develops an algorithm which is particularly simple to implement for equispaced correlation lags a numerical example demonstrates the effectiveness of the technique and its superiority to the usual maximum entropy method

predictions of knee and ankle moments of force in walking from emg and kinematic data abstract a deterministic model was developed and validated to calculate instantaneous ankle and knee moments during walking using processed emg from representative muscles instantaneous joint angle as a correlate of muscle length and angular velocity as a correlate of muscle velocity and having available total instantaneous joint moments for derivation of certain model parameters a linear regression of the moment on specifically processed emg recorded while each subject performed cycled isometric calibration contractions yielded the constants for a basic moment emg relationship using the resultant moment for optimization the predicted moment was proportionally augmented for longer muscle lengths and reduced for shorter lengths similarly the predicted moment was reduced for shortening velocities and increased if the muscle was lengthening the plots of moments predicted using the full model and those calculated from link segment mechanics followed each other quite closely the range of root mean square errors were 3 2 9 5 nm for the ankle and 4 7 13 0 nm for the knee

the relationship between causal and noncausal mismatched estimation in continuous time awgn channels a continuous time finite power process with distribution p is observed through an awgn channel at a given signal to noise ratio snr and is estimated by an estimator that would have minimized the mean square error if the process had distribution q we show that the causal filtering mean square error mse achieved at snr level snr is equal to the average value of the noncausal smoothing mse achieved with a channel whose snr is chosen uniformly distributed between 0 and snr emerging as the bridge for equating these two quantities are mutual information and relative entropy our result generalizes that of guo shamai and verdu 2005 from the nonmismatched case where p q to general p and q among our intermediate results is an extension of duncan s theorem that relates mutual information and causal mmse to the case of mismatched estimation some further extensions and implications are discussed key to our findings is the recent result of verdu on mismatched estimation and relative entropy

on the equivalence of maximum snr and mmse estimation applications to additive non gaussian channels and quantized observations the minimum mean squared error mmse is one of the most popular criteria for bayesian estimation conversely the signal to noise ratio snr is a typical performance criterion in communications radar and generally detection theory in this paper we first formalize an snr criterion to design an estimator and then we prove that there exists an equivalence between mmse and maximum snr estimators for any statistics we also extend this equivalence to specific classes of suboptimal estimators which are expressed by a basis expansion model bem then by exploiting an orthogonal bem for the estimator we derive the mmse estimator constrained to a given quantization resolution of the noisy observations and we prove that this suboptimal mmse estimator tends to the optimal mmse estimator that uses an infinite resolution of the observation besides we derive closed form expressions for the mean squared error mse and for the snr of the proposed suboptimal estimators and we show that these expressions constitute tight asymptotically exact bounds for the optimal mmse and maximum snr

information estimation and lookahead in the gaussian channel we consider mean squared estimation with lookahead of a continuous time signal corrupted by additive white gaussian noise we show that the mutual information rate function i e the mutual information rate as function of the signal to noise ratio snr does not in general determine the minimum mean squared error mmse with fixed finite lookahead in contrast to the special cases with 0 and infinite lookahead filtering and smoothing errors respectively which were previously established in the literature further we investigate the simple class of continuous time stationary gauss markov processes ornstein uhlenbeck processes as channel inputs and explicitly characterize the behavior of the minimum mean squared error mmse with finite lookahead and signal to noise ratio snr we extend our results to mixtures of ornstein uhlenbeck processes and use the insight gained to present lower and upper bounds on the mmse with lookahead for a class of stationary gaussian input processes whose spectrum can be expressed as a mixture of ornstein uhlenbeck spectra

information theory measures with application to model identification the identification of the order of a model which is fitted to data is of central importance we have considered in this paper an information theoretic criterion the aic which has found considerable use in many diverse applications our discussion is presented with various points in mind first of all since the theory of the aic appears in the statistical literature in a complex form we have attempted to present the salient points of the development in a simplified manner coupled with a geometrical interpretation which we hope will illuminate the nature of this criterion furthermore we have illustrated the development with an analytical example and have compared the aic to other measures which have been proposed in the literature diverse applications of the aic which we have investigated include time series modeling parametric inverse problems and spectral analysis we have found the aic to be both a versatile and what is particularly important a robust criterion

minimum cross entropy spectral analysis of multiple signals this paper presents a new information theoretic method for simultaneously estimating a number of power spectra when a prior estimate of each is available and new information is obtained in the form of values of the autocorrelation function of their sum one application of this method is the separate estimation of the spectra of a signal and additive noise based on autocorrelations of the signal plus noise a derivation of the method from the principle of minimum cross entropy is given and the method is compared to minimum cross entropy spectral analysis of which it is a generalization some basic mathematical properties are discussed three numerical examples are included two based oil synthetic spectra and one based on actual speech data

on mittag leffler type functions in fractional evolution processes abstract we review a variety of fractional evolution processes so defined being governed by equations of fractional order whose solutions turn out to be related to mittag leffler type functions the chosen equations are the simplest of the fractional calculus and include the abel integral equations of the second kind which are relevant in typical inverse problems and the fractional differential equations which govern generalized relaxation and oscillation phenomena

using a particle swarm method to optimize the weighting in extension theory for the detection of islanding in photovoltaic systems this paper proposes a two dimensional particle swarm optimization 2d pso method for optimizing the weighting in extension theory for the detection of islanding in photovoltaic pv power generation systems generally using extension theory to implement and analyze a system with a correlation function would involve constructing a weighting determined by trial and error to help judge the problem s performance however the judgment accuracy can be degraded if one uses an inappropriate weighting set hence this paper proposes a weighting determination method for optimizing the performance of the extension method using the 2d pso algorithm some simulation results are obtained to verify the effectiveness of the proposed islanding detection method in addition the simulated results obtained using the proposed 2d pso algorithm are also compared with those obtained using genetic algorithm ga and evolutionary programming ep algorithms in order to reveal the search performance of the proposed method

entropy interpretation of maximum likelihood deconvolution entropy concepts are embodied in the maximum likelihood deconvolution mld method just as they are in minimum entropy med methods mld asymptotically minimizes shannon s entropy of the reflectivity sequence which in mld is modeled as a bernoulli gaussian random sequence study of maximum likelihood detection and estimation of the reflectivity sequence reveals that mld is embodied in the general framework of the adaptive med methods comparisons based on similarities and differences between mld and various existing med techniques show that mld is robust due to explicit inclusion of noise in its statistical model