comparison of maximum entropy and higher order entropy estimators abstract we show that generalized maximum entropy gme is the only estimation method that is consistent with a set of five axioms the gme estimator can be nested using a single parameter into two more general classes of estimators gme estimators each of these gme estimators violates one of the five basic axioms however small sample simulations demonstrate that certain members of these gme classes of estimators may outperform the gme estimation rule

empirical likelihood and general estimating equations for some time so called empirical likelihoods have been used heuristically for purposes of nonparametric estimation owen showed that empirical likelihood ratio statistics for various parameters theta f of an unknown distribution f have limiting chi square distributions and may be used to obtain tests or confidence intervals in a way that is completely analogous to that used with parametric likelihoods our objective in this paper is twofold first to link estimating functions or equations and empirical likelihood second to develop methods of combining information about parameters we do this by assuming that information about f and theta is available in the form of unbiased estimating functions empirical likelihoods for parameters are developed and shown to have properties similar to those for parametric likelihood efficiency results for estimates of both theta and f are obtained the methods are illustrated on several problems and areas for future investigation are noted

expressions for renyi and shannon entropies for bivariate distributions exact forms of renyi and shannon entropies are determined for 27 continuous bivariate distributions including the kotz type distribution truncated normal distribution distributions with normal and centered normal conditionals natural exponential distribution freund s exponential distribution marshall and olkin s exponential distribution exponential mixture distribution arnold and strauss s exponential distribution mckay s gamma distribution distribution with gamma conditionals gamma exponential distribution dirichlet distribution inverted beta distribution distribution with beta conditionals beta stacy distribution cuadras and auge s distribution farlie gumbel morgenstern distribution logistic distribution pearson type vii distribution pearson type ii distribution distribution with cauchy conditionals bilateral pareto distribution muliere and scarsini s pareto distribution distribution with pareto conditionals and the distribution with gumbel conditionals we believe that the results presented here will serve as an important reference for scientists and engineers in many areas

information indices unification and applications the unified framework of information theoretic statistics was established by kullback 1959 since then numerous information indices have been developed in various contexts this paper represents many of these indices in a unified context the unification thread is the discrimination information function information indices are all logarithmic measures of discrepancy between two probability distributions first we present a summary of informational aspects of the basic information functions a unification of various information theoretic modeling approaches and some explication in terms of traditional measures we then tabulate a unified representation of assortments of information indices developed in the literature for maximum entropy modeling covariate information and influence diagnostics the subjects of these indices include parametric model fitting nonparametric entropy estimation categorical data analysis the linear and exponential family regression and time series the coverage however is not exhaustive the tabulation includes sampling theory and bayesian indices but the focus is on interpretation as descriptive measures and inferential properties are noted tangentially finally applications of some information indices are illustrated through modeling duration data for sprint s churned customer and choice of long distance provider

bayes estimators of generalized entropies the order q tsallis and renyi entropy receive broad applications in the statistical analysis of complex phenomena a generic problem arises however when these entropies need to be estimated from observed data the finite size of data sets can lead to serious systematic and statistical errors in numerical estimates in this paper we focus upon the problem of estimating generalized entropies from finite samples and derive the bayes estimator of the order q tsallis entropy including the order 1 i e the shannon entropy under the assumption of a uniform prior probability density the bayes estimator yields in general the smallest mean quadratic deviation from the true parameter as compared with any other estimator exploiting the functional relationship between and we use the bayes estimator of to estimate the renyi entropy we compare these novel estimators with the frequency count estimators for and we find by numerical simulations that the bayes estimator reduces statistical errors of order q entropy estimates for bernoulli as well as for higher order markov processes derived from the complete genome of the prokaryote haemophilus influenzae

information and entropy econometrics editor s view amos golan department of economics american university roper 200 4400 massachusetts ave nw washington dc 20016 usa

online data driven fuzzy clustering with applications to real time robotic tracking robotic target tracking has been used in a variety of applications due to limited sampling rate sensory characteristics and processing delays an important issue in such systems is to extrapolate ahead the trajectory position orientation velocity and or acceleration of moving targets from past observations this paper introduces a novel online data driven fuzzy clustering algorithm that is based on the maximum entropy principle for this particular task in this algorithm the fuzzy inference mechanism is extracted automatically from observed data without human help which thus eliminates the necessity of expert s knowledge and a priori information on moving targets as required by most traditional techniques this algorithm does not require training which enables it to work in a completely online fashion another important and distinct advantage of the algorithm exists in the fact that it is very fast and efficient in terms of computational cost and thus can be implemented in real time in the meantime the introduced algorithm is able to adapt quickly to the dynamics of moving targets all these desired features make it especially suitable for the task to predict the trajectory of moving targets in robotic tracking simulation results show the effectiveness and efficiency of the presented algorithm

on some entropy functionals derived from renyi information divergence we consider the maximum entropy problems associated with renyi q entropy subject to two kinds of constraints on expected values the constraints considered are a constraint on the standard expectation and a constraint on the generalized expectation as encountered in nonextensive statistics the optimum maximum entropy probability distributions which can exhibit a power law behaviour are derived and characterized the renyi entropy of the optimum distributions can be viewed as a function of the constraint this defines two families of entropy functionals in the space of possible expected values general properties of these functionals including nonnegativity minimum convexity are documented their relationships as well as numerical aspects are also discussed finally we work out some specific cases for the reference measure q x and recover in a limit case some well known entropies

possible generalization of boltzmann gibbs statistics with the use of a quantity normally scaled in multifractals a generalized form is postulated for entropy namelys q k 1 i 1 w p i q q 1 whereq characterizes the generalization andp i are the probabilities associated withw microscopic configurations w the main properties associated with this entropy are established particularly those corresponding to the microcanonical and canonical ensembles the boltzmann gibbs statistics is recovered as theq 1 limit

empirical likelihood for linear models empirical likelihood is a nonparametric method of inference it has sampling properties similar to the bootstrap but where the bootstrap uses resampling it profiles a multinomial likelihood supported on the sample its properties in i i d settings have been investigated in works by owen by hall and by diciccio hall and romano this article extends the method to regression problems fixed and random regressors are considered as are robust and heteroscedastic regressions to make the extension three variations on the original idea are considered it is shown that when some functionals of the distribution of the data are known one can get sharper inferences on other functionals by imposing the known values as constraints on the optimization the result is first order equivalent to conditioning on a sample value of the known functional the use of a euclidean alternative to the likelihood function is investigated a triangular array version of the empirical likelihood theorem is given the one way anova and heteroscedastic regression models are considered in detail an example is given in which inferences are drawn on the parameters of both the regression function and the conditional variance model

invariance properties of density ratio priors density ratio neighborhoods are classes of probabilities that are used in robust bayesian inference these classes are invariant under bayesian updating and marginalization this makes them computationally convenient in robust bayesian inference we show that this is the unique class of probabilities that has these invariance properties aside from its theoretical value this result has computational implications as well

an information theoretic alternative to generalized method of moments estimation while optimally weighted gmm estimation has desirable large sample properties its small sample performance is poor in some applications we propose a computationally simple alternative for weakly dependent data generating mechanisms based on minimization of the kullback leibler information criterion conditions are derived under which the large sample properties of this estimator are similar to gmm i e the estimator will be consistent and asymptotically normal with the same asymptotic covariance matrix as gmm in addition we propose overidentifying and parametric restrictions tests as alternatives to analogous gmm procedures

on the complexity of boolean matrix ranks abstract we present a reduction which shows that the fooling set number tropical and determinantal ranks of a boolean matrix are np hard to compute

some aspects of the history of bayesian information processing for many years traditional bayesian tb and information theoretic it procedures for learning from data were viewed as distinctly different approaches derivations of the tb and it learning models are reviewed and compared then the synthesis by zellner in 1988 of the tb and it learning models and generalizations of them are described along with descriptions of selected applications included are learning procedures that do not require use of likelihood functions and or priors works by leading bayesians and information theorists are cited and related to tb it issues

a class of locally and globally robust regression estimates abstract we present a new class of regression estimates called generalized estimates these estimates are defined by minimizing the scale of the weighted residuals with weights that penalize high leverage observations like the estimates the generalized estimates utilize for their definition two loss functions 1 and 2 which together with the weights can be chosen to achieve simultaneously high breakdown point finite gross error sensitivity and high efficiency we recommend however choosing these functions so as to control the bias behavior of the estimate for a large range of possible contaminations and then boosting the efficiency by a simple least squares reweighting step the generalized estimate with loss functions 1 and 2 is related to the hill ryan gm estimate with a loss function which is a linear combination of 1 and r in fact both estimates have the same influence function and asymptotic distribution under the central model we show that a certain generalized estima

a simultaneous estimation and variable selection rule a new data based method of estimation and variable selection in linear statistical models is proposed this method is based on a generalized maximum entropy formalism and makes use of both sample and non sample information in determining a basis for coefficient shrinkage and extraneous variable identification in contrast to tradition shrinkage and variable selection are achieved on a coordinate by coordinate basis and the procedure works well for both ill and well posed statistical models analytical asymptotic results are presented and sampling experiments are used as a basis for determining finite sample behavior and comparing the sampling performance of the new estimation rule with traditional competitors solution algorithms for the non linear inversion problem that results are simple to implement

information theoretic approaches to inference in moment condition models one step efficient gmm estimation has been developed in the recent papers of back and brown 1990 imbens 1993 and qin and lawless 1994 these papers emphasized methods that correspond to using owen s 1988 method of empirical likelihood to reweight the data so that the reweighted sample obeys all the moment restrictions at the parameter estimates in this paper we consider an alternative klic motivated weighting and show how it and similar discrete reweightings define a class of unconstrained optimization problems which includes gmm as a special case such klic motivated reweightings introduce m auxiliary tilting parameters where m is the number of moments parameter and overidentification hypotheses can be recast in terms of these tilting parameters such tests when appropriately conditioned on the estimates of the original parameters are often startlingly more effective than their conventional counterparts this is apparently due to the local ancillarity of the original parameters for the tilting parameters

a versatile and robust metric entropy test of time reversibility and other hypotheses abstract we examine the performance of a metric entropy statistic as a robust test for time reversibility tr symmetry and serial dependence it also serves as a measure of goodness of fit the statistic provides a consistent and unified basis in model search and is a powerful diagnostic measure with surprising ability to pinpoint areas of model failure we provide empirical evidence comparing the performance of the proposed procedure with some of the modern competitors in nonlinear time series analysis such as robust implementations of the bds and characteristic function based tests of tr along with correlation based competitors such as the ljung box q statistic unlike our procedure each of its competitors is motivated for a different specific context and hypothesis our evidence is based on monte carlo simulations along with an application to several stock indices for the us equity market

generalized statistical mechanics connection with thermodynamics the authors show the manner through which the recent generalization by tsallis 1988 of the boltzmann gibbs statistics becomes consistent with a generalized thermodynamics preserving the legendre transformation framework of standard thermodynamics in addition to that they generalize the shannon additivity