information theoretic determination of minimax rates of convergence we present some general results determining minimax bounds on statistical risk for density estimation based on certain information theoretic considerations these bounds depend only on metric entropy conditions and are used to identify the minimax rates of convergence

probability product kernels the advantages of discriminative learning algorithms and kernel machines are combined with generative modeling using a novel kernel between distributions in the probability product kernel data points in the input space are mapped to distributions over the sample space and a general inner product is then evaluated as the integral of the product of pairs of distributions the kernel is straightforward to evaluate for all exponential family models such as multinomials and gaussians and yields interesting nonlinear kernels furthermore the kernel is computable in closed form for latent distributions such as mixture models hidden markov models and linear dynamical systems for intractable models such as switching linear dynamical systems structured mean field approximations can be brought to bear on the kernel evaluation for general distributions even if an analytic expression for the kernel is not feasible we show a straightforward sampling method to evaluate it thus the kernel permits discriminative learning methods including support vector machines to exploit the properties metrics and invariances of the generative models we infer from each datum experiments are shown using multinomial models for text hidden markov models for biological data sets and linear dynamical systems for time series data

feature extraction by non parametric mutual information maximization we present a method for learning discriminative feature transforms using as criterion the mutual information between class labels and transformed features instead of a commonly used mutual information measure based on kullback leibler divergence we use a quadratic divergence measure which allows us to make an efficient non parametric implementation and requires no prior assumptions about class densities in addition to linear transforms we also discuss nonlinear transforms that are implemented as radial basis function networks extensions to reduce the computational complexity are also presented and a comparison to greedy feature selection is made

on a 2 generated infinite 3 group the presentation problem the author and narain gupta constructed in 3 a family of residually finite 2generated infinite p groups for every odd prime p as groups of automorphisms of regular p trees this was inspired by grigorchuk s construction in z of a 3 generated infinite 2 group as a permutation group of subintervals of the unit interval in 6 merzlyakov relates grigorchuk s group to p groups constructed by aleshin in 1972 l by using automatas the regularity of our definitions has allowed us to generalize our earlier constructions see 4 and gave us ready access to some of their special properties see 51 the prototype of all our constructions has been the infinite 3 group

discrete time h controllers satisfying a minimum entropy criterion a discrete time minimum entropy h control problem is solved and state space formulae are derived the solution is obtained from first principles and so avoids the reformulation of the problem to a continuous time problem via a bilinear transformation

type based random access for distributed detection over multiaccess fading channels the problem of distributed detection in a sensor network over multiaccess fading channels is considered a random access transmission scheme referred to as the type based random access tbra is proposed and analyzed error exponents of tbra under noncoherent detection are characterized with respect to the mean transmission rate and the channel coherence index for the zero mean multiaccess fading channels it is shown that there exists an optimal mean transmission rate that maximizes the detection error exponents the optimal mean transmission rate can be calculated numerically or estimated using the gaussian approximation and it gives a sensor activation strategy that achieves an optimal allocation of transmission energy to spatial and temporal domains numerical examples and simulations are used to compare tbra with the conventional centralized time division multiple access tdma scheme it is shown that for the zero mean multiaccess fading channels tbra gives substantial improvement in the low signal to noise ratio snr regime whereas for the nonzero mean fading channels tbra performs better over a wide range of snr

learning on dynamic statistical manifolds hyperbolic balance laws with uncertain random parameters and inputs are ubiquitous in science and engineering quantification of uncertainty in predictions derived from such laws and reduction of predictive uncertainty via data assimilation remain an open challenge that is due to nonlinearity of governing equations whose solutions are highly non gaussian and often discontinuous to ameliorate these issues in a computationally efficient way we use the method of distributions which here takes the form of a deterministic equation for spatio temporal evolution of the cumulative distribution function cdf of the random system state as a means of forward uncertainty propagation uncertainty reduction is achieved by recasting the standard loss function i e discrepancy between observations and model predictions in distributional terms this step exploits the equivalence between minimization of the square error discrepancy and the kullback leibler divergence the loss function is regularized by adding a lagrangian constraint enforcing fulfilment of the cdf equation minimization is performed sequentially progressively updating the parameters of the cdf equation as more measurements are assimilated

comments on a source matching approach to finding minimax codes by davisson l d and leon garcia a in the above paper 1 davisson and leon garcia prove a result equating the minimax redundancy for universal codes to a channel capacity that is computable by standard techniques they point out that the result had been developed earlier in unpublished work by gallager i would like to point out an alternative development of this result and to compare this work with that of davisson and leon garcia

bounds for the loss in probability of correct classification under model based approximation in many pattern recognition classification problem the true class conditional model and class probabilities are approximated for reasons of reducing complexity and or of statistical estimation the approximated classifier is expected to have worse performance here measured by the probability of correct classification we present an analysis valid in general and easily computable formulas for estimating the degradation in probability of correct classification when compared to the optimal classifier an example of an approximation is the naive bayes classifier we show that the performance of the naive bayes depends on the degree of functional dependence between the features and labels we provide a sufficient condition for zero loss of performance too

rates of convergence for minimum contrast estimators we shall present here a general study of minimum contrast estimators in a nonparametric setting although our results are also valid in the classical parametric case for independent observations these estimators include many of the most popular estimators in various situations such as maximum likelihood estimators least squares and other estimators of the regression function estimators for mixture models or deconvolution the main theorem relates the rate of convergence of those estimators to the entropy structure of the space of parameters optimal rates depending on entropy conditions are already known at least for some of the models involved and they agree with what we get for minimum contrast estimators as long as the entropy counts are not too large but under some circumstances large entropies or changes in the entropy structure due to local perturbations the resulting the rates are only suboptimal counterexamples are constructed which show that the phenomenon is real for non parametric maximum likelihood or regression this proves that under purely metric assumptions our theorem is optimal and that minimum contrast estimators happen to be suboptimal

some inequalities for information divergence and related measures of discrimination inequalities which connect information divergence with other measures of discrimination or distance between probability distributions are used in information theory and its applications to mathematical statistics ergodic theory and other scientific fields we suggest new inequalities of this type often based on underlying identities as a consequence we obtain certain improvements of the well known pinsker inequality our study depends on two measures of discrimination called capacitory discrimination and triangular discrimination the discussion contains references to related research and comparison with other measures of discrimination e g ali silvey csiszar 1996 1966 divergences and in particular the hellinger distance

classification of homogeneous data with large alphabets given training sequences generated by two distinct but unknown distributions on a common alphabet we study the problem of determining whether a third sequence was generated according to the first or second distribution to model sources such as natural language for which the underlying distributions are difficult to learn from realistic amounts of data we allow the alphabet size to grow and therefore the probability distributions to change with the block length our primary focus is the situation in which the underlying probabilities are all of the same order and in this regime we show that consistent classification is possible if and only if the alphabet grows subquadratically with the block length we also show that some commonly used statistical tests are suboptimal in that they are consistent only if the alphabet grows sublinearly

symmetrizing the kullback leibler distance we define a new distance measure the resistor average distance between two probability distributions that is closely related to the kullback leibler distance while the kullbackleibler distance is asymmetric in the two distributions the resistor average distance is not it arises from geometric considerations similar to those used to derive the chernoff distance determining its relation to well known distance measures reveals a new way to depict how commonly used distance measures relate to each other

sliding mode control design for nonlinear systems using probability density function shaping in this paper we propose a sliding mode based stochastic distribution control algorithm for nonlinear systems where the sliding mode controller is designed to stabilize the stochastic system and stochastic distribution control tries to shape the sliding surface as close as possible to the desired probability density function kullback leibler divergence is introduced to the stochastic distribution control and the parameter of the stochastic distribution controller is updated at each sample interval rather than using a batch mode it is shown that the estimated weight vector will converge to its ideal value and the system will be asymptotically stable under the rank condition which is much weaker than the persistent excitation condition the effectiveness of the proposed algorithm is illustrated by simulation

entropy and distance of random graphs with application to structural pattern recognition the notion of a random graph is formally defined it deals with both the probabilistic and the structural aspects of relational data by interpreting an ensemble of attributed graphs as the outcomes of a random graph we can use its lower order distribution to characterize the ensemble to reflect the variability of a random graph shannon s entropy measure is used to synthesize an ensemble of attributed graphs into the distribution of a random graph or a set of distributions we propose a distance measure between random graphs based on the minimum change of entropy before and after their merging when the ensemble contains more than one class of pattern graphs the synthesis process yields distributions corresponding to various classes this process corresponds to unsupervised learning in pattern classification using the maximum likelihood rule and the probability computed for the pattern graph based on its matching with the random graph distributions of different classes we can classify the pattern graph to a class

a just nonsolvable torsion free group defined on the binary tree abstract a two generator torsion free subgroup of the group of finite state automorphisms of the binary tree is constructed having the properties of being just nonsolvable and residually torsion free solvable a presentation is produced for this subgroup in two generators and two relations together with their images under the iterated application of a certain simple substitution

jensen divergence based on fisher s information the measure of jensen fisher divergence between probability distributions is introduced and its theoretical grounds set up this quantity in contrast to the remaining jensen divergences grasps the fluctuations of the probability distributions because it is controlled by the local fisher information which is a gradient functional of the distribution so it is appropriate and informative when studying the similarity of distributions mainly for those having oscillatory character the new jensen fisher divergence shares with the jensen shannon divergence the following properties non negativity additivity when applied to an arbitrary number of probability densities symmetry under exchange of these densities vanishing under certain conditions and definiteness even when these densities present non common zeros moreover the jensen fisher divergence is shown to be expressed in terms of the relative fisher information as the jensen shannon divergence does in terms of the kullback leibler or relative shannon entropy finally the jensen shannon and jensen fisher divergences are compared for the following three large non trivial and qualitatively different families of probability distributions the sinusoidal generalized gamma like and rakhmanov hermite distributions which are closely related to the quantum mechanical probability densities of numerous physical systems

on divergences and informations in statistics and information theory the paper deals with the f divergences of csiszar generalizing the discrimination information of kullback the total variation distance the hellinger divergence and the pearson divergence all basic properties of f divergences including relations to the decision errors are proved in a new manner replacing the classical jensen inequality by a new generalized taylor expansion of convex functions some new properties are proved too e g relations to the statistical sufficiency and deficiency the generalized taylor expansion also shows very easily that all f divergences are average statistical informations differences between prior and posterior bayes errors mutually differing only in the weights imposed on various prior distributions the statistical information introduced by de groot and the classical information of shannon are shown to be extremal cases corresponding to alpha 0 and alpha 1 in the class of the so called arimoto alpha informations introduced in this paper for 0 alpha 1 by means of the arimoto alpha entropies some new examples of f divergences are introduced as well namely the shannon divergences and the arimoto alpha divergences leading for alphauarr1 to the shannon divergences square roots of all these divergences are shown to be metrics satisfying the triangle inequality the last section introduces statistical tests and estimators based on the minimal f divergence with the empirical distribution achieved in the families of hypothetic distributions for the kullback divergence this leads to the classical likelihood ratio test and estimator

information divergence and risk for binary experiments we unify f divergences bregman divergences surrogate regret bounds proper scoring rules cost curves roc curves and statistical information we do this by systematically studying integral and variational representations of these objects and in so doing identify their representation primitives which all are related to cost sensitive binary classification as well as developing relationships between generative and discriminative views of learning the new machinery leads to tight and more general surrogate regret bounds and generalised pinsker inequalities relating f divergences to variational divergence the new viewpoint also illuminates existing algorithms it provides a new derivation of support vector machines in terms of divergences and relates maximum mean discrepancy to fisher linear discriminants

state space approach to discrete time h control abstract state space solutions to a discrete time problem are given for a given number 0we give a characterization of all controllers such that the norm of the closed loop transfer function is less than the approach taken is based on the solution of two riccati equations using the stable deflating subspace of a symplectic pencil this ensures that no unnecessary assumptions on the a matrix on the realization of the transfer function are needed while the results are the direct analogues of those in the continuous time the state space derivation is considerably more difficult

all optimal hankel norm approximations of linear multivariable systems and their l error bounds the problem of approximating a multivariable transfer function g s of mcmillan degree n by s of mcmillan degree k is considered a complete characterization of all approximations that minimize the hankel norm is derived the solution involves a characterization of all rational functions s f s that minimize where s has mcmillan degree k and f s is anticavisal the solution to the latter problem is via results on balanced realizations all pass functions and the inertia of matrices all in terms of the solutions to lyapunov equations it is then shown that where k 1 g s is the k l st hankel singular value of g s and for one class of optimal hankel norm approximations the method is not computationally demanding and is applied to a 12 state model

a descriptor solution to a class of discrete distance problems hankel norm and nehari type approximation problems arise in model reduction and spl hscr sub spl infin control theory existing solutions to the discrete time version of these problems may be derived using a standard state space framework but the resulting solution formulas require an invertible a matrix as a further complication the d matrix in the representation formula for all solutions becomes unbounded in the optimal case the aim of this paper is to show that both these complications may be removed by analyzing these problems in a descriptor framework

bounding bar d distance by informational divergence a method to prove measure concentration there is a simple inequality by pinsker between variational distance and informational divergence of probability measures defined on arbitrary probability spaces we shall consider probability measures on sequences taken from countable alphabets and derive from pinsker s inequality bounds on the bar d distance by informational divergence such bounds can be used to prove the concentration of measure phenomenon for some nonproduct distributions

estimating divergence functionals and the likelihood ratio by convex risk minimization we develop and analyze m estimation methods for divergence functionals and the likelihood ratios of two probability distributions our method is based on a nonasymptotic variational characterization of f divergences which allows the problem of estimating divergences to be tackled via convex empirical risk optimization the resulting estimators are simple to implement requiring only the solution of standard convex programs we present an analysis of consistency and convergence for these estimators given conditions only on the ratios of densities we show that our estimators can achieve optimal minimax rates for the likelihood ratio and the divergence functionals in certain regimes we derive an efficient optimization algorithm for computing our estimates and illustrate their convergence behavior and practical viability by simulations

principal component analysis in linear systems controllability observability and model reduction kalman s minimal realization theory involves geometric objects controllable unobservable subspaces which are subject to structural instability specifically arbitrarily small perturbations in a model may cause a change in the dimensions of the associated subspaces this situation is manifested in computational difficulties which arise in attempts to apply textbook algorithms for computing a minimal realization structural instability associated with geometric theories is not unique to control it arises in the theory of linear equations as well in this setting the computational problems have been studied for decades and excellent tools have been developed for coping with the situation one of the main goals of this paper is to call attention to principal component analysis hotelling 1933 and an algorithm golub and reinsch 1970 for computing the singular value decompositon of a matrix together they form a powerful tool for coping with structural instability in dynamic systems as developed in this paper principal component analysis is a technique for analyzing signals singular value decomposition provides the computational machinery for this reason kalman s minimal realization theory is recast in terms of responses to injected signals application of the signal analysis to controllability and observability leads to a coordinate system in which the internally balanced model has special properties for asymptotically stable systems this yields working approximations of x c x bar o the controllable and unobservable subspaces it is proposed that a natural first step in model reduction is to apply the mechanics of minimal realization using these working subspaces

brief paper positivity preserving h model reduction for positive systems this paper is concerned with the model reduction of positive systems for a given stable positive system our attention is focused on the construction of a reduced order model in such a way that the positivity of the original system is preserved and the error system is stable with a prescribed h performance based upon a system augmentation approach a novel characterization on the stability with h performance of the error system is first obtained in terms of linear matrix inequality lmi then a necessary and sufficient condition for the existence of a desired reduced order model is derived accordingly furthermore iterative lmi approaches with primal and dual forms are developed to solve the positivity preserving h model reduction problem finally a compartmental network is provided to show the effectiveness of the proposed techniques

approximation dans les espaces metriques et theorie de l estimation we investigate the relations between the speed of estimation and the metric structure of the parameter space especially in the case when its metric dimension is infinite given some distance d on generally hellinger distance in the case of n i i d variables we consider the minimax risk for n observations r n q mathop inf sup limits t n theta in theta text ie theta d q theta t n t n being any estimate of we shall look for functions r such that for positive constants c 1 q and c 2 q c 1 r q n r n q c2 r q n r n is the speed of estimation and we shall show under fairly general conditions including i i d variables and regular cases of markov chains and stationnary gaussian processes that r n is determined up to multiplicative constants by the metric structure of we shall also give a construction for some sort of universal estimates the risk of which is bounded by c 2 r q n in all cases where the preceding theory applies

quantization of prior probabilities for hypothesis testing in this paper bayesian hypothesis testing is investigated when the prior probabilities of the hypotheses taken as a random vector are quantized nearest neighbor and centroid conditions are derived using mean bayes risk error mbre as a distortion measure for quantization a high resolution approximation to the distortion rate function is also obtained human decision making in segregated populations is studied assuming bayesian hypothesis testing with quantized priors

nonextensive information theoretic kernels on measures positive definite kernels on probability measures have been recently applied to classification problems involving text images and other types of structured data some of these kernels are related to classic information theoretic quantities such as shannon s mutual information and the jensen shannon js divergence meanwhile there have been recent advances in nonextensive generalizations of shannon s information theory this paper bridges these two trends by introducing nonextensive information theoretic kernels on probability measures based on new js type divergences these new divergences result from extending the the two building blocks of the classical js divergence convexity and shannon s entropy the notion of convexity is extended to the wider concept of q convexity for which we prove a jensen q inequality based on this inequality we introduce jensen tsallis jt q differences a nonextensive generalization of the js divergence and define a k th order jt q difference between stochastic processes we then define a new family of nonextensive mutual information kernels which allow weights to be assigned to their arguments and which includes the boolean js and linear kernels as particular cases nonextensive string kernels are also defined that generalize the p spectrum kernel we illustrate the performance of these kernels on text categorization tasks in which documents are modeled both as bags of words and as sequences of characters

tight bounds for symmetric divergence measures and a refined bound for lossless source coding tight bounds for several symmetric divergence measures are derived in terms of the total variation distance it is shown that each of these bounds is attained by a pair of two or three element probability distributions an application of these bounds for lossless source coding is provided refining and improving a certain bound by csiszar another application of these bounds has been recently introduced by yardi et al for channel code detection

on the selection of measures of distance between probability distributions abstract a notion of relative strength and weakness is defined for the comparison of measures of distance between probability distributions and is shown to be appropriate for use in the selection of distances as quasioptimal design criteria in problems such as signal selection and detector design when the preferred criteria probability of error and asymptotic relative efficiency are intractable csiszar s work on the topological properties of divergences a class of distances also investigated by ali and silvey enables one to identify strong distances and provides a justification for the experimental findings of previous investigators

entropy bounds for discrete random variables via maximal coupling this paper derives new bounds on the difference of the entropies of two discrete random variables in terms of the local and total variation distances between their probability mass functions the derivation of the bounds relies on maximal coupling and they apply to discrete random variables which are defined over finite or countably infinite alphabets loosened versions of these bounds are demonstrated to reproduce some previously reported results the use of the new bounds is exemplified for the poisson approximation where bounds on the local and total variation distances follow from stein s method

on amenability of automata groups we show that the group of bounded automatic automorphisms of a rooted tree is amenable which implies amenability of numerous classes of groups generated by finite automata the proof is based on reducing the problem to showing amenability just of a certain explicit family of groups mother groups which is done by analyzing the asymptotic properties of random walks on these groups

lower bounds for the minimax risk using f divergences and applications lower bounds involving f divergences between the underlying probability measures are proved for the minimax risk in estimation problems our proofs just use simple convexity facts special cases and straightforward corollaries of our bounds include well known inequalities for establishing minimax lower bounds such as fano s inequality pinsker s inequality and inequalities based on global entropy conditions two applications are provided a new minimax lower bound for the reconstruction of convex bodies from noisy support function measurements and a different proof of a recent minimax lower bound for the estimation of a covariance matrix

variance reduction techniques for gradient estimates in reinforcement learning policy gradient methods for reinforcement learning avoid some of the undesirable properties of the value function approaches such as policy degradation baxter and bartlett 2001 however the variance of the performance gradient estimates obtained from the simulation is sometimes excessive in this paper we consider variance reduction methods that were developed for monte carlo estimates of integrals we study two commonly used policy gradient techniques the baseline and actor critic methods from this perspective both can be interpreted as additive control variate variance reduction methods we consider the expected average reward performance measure and we focus on the gpomdp algorithm for estimating performance gradients in partially observable markov decision processes controlled by stochastic reactive policies we give bounds for the estimation error of the gradient estimates for both baseline and actor critic algorithms in terms of the sample size and mixing properties of the controlled system for the baseline technique we compute the optimal baseline and show that the popular approach of using the average reward to define the baseline can be suboptimal for actor critic algorithms we show that using the true value function as the critic can be suboptimal we also discuss algorithms for estimating the optimal baseline and approximate value function

state space solution of the discrete time minimum entropy control problem via separation a discrete time minimum entropy h sub infinity control problem is solved the general solution is obtained by separating the problem into two special cases which are solved directly in state space the analysis is performed wholly in discrete time and so avoids the difficulties in obtaining the solution from the analogous continuous time problem

state space formulae for all the stabilizing discrete controllers that satisfy an h norm bound part 1 main results and the full information problem given a linear discrete time system necessary and sufficient conditions for the existence of a linear or nonlinear controller which satisfies a specified closed loop h norm bound are stated in terms of the solutions to two uncoupled algebraic discrete riccati equations of the order of the system state space formulae for all the stabilizing controllers some of the order of the system which satisfy an achievable bound are also given in this first part of the paper new growth and convergence properties of the solution to a time varying discrete riccati equation are used to derive the necessary and sufficient conditions for the existence of a solution in the full information case these properties generalize those of the corresponding discrete riccati equation related to the lqr problem in the second part of the paper a non standard estimation problem is posed and solved this problem arises as a novel reinterpretation of the h norm objective its solution can be used to reduce a partial informa

robust output feedback stabilization via risk sensitive control we consider a problem of robust linear quadratic gaussian lqg control for discrete time stochastic uncertain systems with partial state measurements for a finite horizon case the problem was recently introduced by petersen et al ieee trans automat control 45 2000 398 in this paper an infinite horizon extension of the results of petersen et al ieee trans automat control 45 2000 398 is discussed we show that for a broad class of uncertain systems under consideration a controller constructed in terms of the solution to a specially parameterized risk sensitive stochastic control problem absolutely stabilizes the stochastic uncertain system

information inequalities and concentration of measure we derive inequalities of the form p q h p r h q r which hold for every choice of probability measures p q r where h p r denotes the relative entropy of p with respect to r and p q stands for a coupling type distance between p and q using the chain rule for relative entropies and then specializing to q with a given support we recover some of talagrand s concentration of measure inequalities for product spaces

a redundancy based measure of dissimilarity among probability distributions for hierarchical clustering criteria we introduce novel dissimilarity into a probabilistic clustering task to properly measure dissimilarity among multiple clusters when each cluster is characterized by a subpopulation in the mixture model this measure of dissimilarity is called redundancy based dissimilarity among probability distributions from aspects of both source coding and a statistical hypothesis test we shed light on several of the theoretical reasons for the redundancy based dissimilarity among probability distributions being a reasonable measure of dissimilarity among clusters we also elucidate a principle in common for the measures of redundancy based dissimilarity and ward s method in terms of hierarchical clustering criteria moreover we show several related theorems that are significant for clustering tasks in the experiments properties of the measure of redundancy based dissimilarity are examined in comparison with several other measures

multirate sampled data systems all h sub spl infin suboptimal controllers and the minimum entropy controller for a general multirate sampled data sd system the authors characterize explicitly the set of all causal stabilizing controllers that achieve a certain h sub spl infin norm bound moreover they give explicitly a particular controller that further minimizes an entropy function for the sd system the characterization lays the groundwork for synthesizing multirate control systems with multiple mixed control specifications

inequalities between entropy and index of coincidence derived from information diagrams to any discrete probability distribution p we can associate its entropy h p spl sigma p sub i ln p sub i and its index of coincidence ic p spl sigma p sub i sup 2 the main result of the paper is the determination of the precise range of the map p spl rarr ic p h p the range looks much like that of the map p spl rarr p sub max h p where p sub max is the maximal point probability cf research from 1965 kovalevskij 1965 to 1994 feder and merhav 1994 the earlier results which actually focus on the probability of error 1 p sub max rather than p sub max can be conceived as limiting cases of results obtained by methods presented here ranges of maps as those indicated are called information diagrams the main result gives rise to precise lower as well as upper bounds for the entropy function some of these bounds are essential for the exact solution of certain problems of universal coding and prediction for bernoulli sources other applications concern shannon theory relations between various measures of divergence statistical decision theory and rate distortion theory two methods are developed one is topological the other involves convex analysis and is based on a lemma of replacement which is of independent interest in relation to problems of optimization of mixed type concave convex optimization

sharp inequalities for f divergences f divergences are a general class of divergences between probability measures which include as special cases many commonly used divergences in probability mathematical statistics and information theory such as kullback leibler divergence chi squared divergence squared hellinger distance total variation distance and so on in this paper we study the problem of maximizing or minimizing an f divergence between two probability measures subject to a finite number of constraints on other f divergences we show that these infinite dimensional optimization problems can all be reduced to optimization problems over small finite dimensional spaces which are tractable our results lead to a comprehensive and unified treatment of the problem of obtaining sharp inequalities between f divergences we demonstrate that many of the existing results on inequalities between f divergences can be obtained as special cases of our results we also improve on some existing non sharp inequalities

on the jensen shannon divergence and variational distance we study the distance measures between two probability distributions via two different distance metrics a new metric induced from jensen shannon divergence and the well known l sub 1 metric we show that several important results and constructions in computational complexity under the l sub 1 metric carry over to the new metric such as yao s next bit predictor the existence of extractors the leftover hash lemma and the construction of expander graph based extractor finally we show that the useful parity lemma in studying pseudorandomness does not hold in the new metric

model reduction via balanced state space representations a model reduction procedure based on balanced state space representations is studied in this paper the reduced order model is examined from the point of view of stability controllability and observability both continuous time and discrete time systems are considered

endomorphic presentations of branch groups we introduce endomorphic presentations or l presentations group presentations whose relations are iterated under a set of substitutions on the generating set and show that a broad class of groups acting on rooted trees admit explicitly constructible finite l presentations generalising results by igor lysionok and said sidki

a simple proof of the blowing up lemma corresp the blowing up lemma says that if the probability with respect to a product measure of a set a subseteq cal x n cal x finite n large is not exponentially small then its l n neighborhood has probability almost one for some l n o n here an information theoretic proof of the blowing up lemma generalizing it to continuous alphabets is given

ℋ control in discrete time state feedback control and norm bounds h norm bounds are useful in control design for disturbance rejection for discrete systems h norm optimizing controls can be characterized via the solution of the discrete algebraic riccati equation dare in addition the dare can be used to characterize controls that guarantee a uniform h norm bound in this paper one identifies an entire class of state feedback controls that guarantee an h norm bound and provides new lower bounds for the h norm of stable systems as well as for the best achievable h norm using state feedback controls

a distribution dependent refinement of pinsker s inequality given two probability distributions q and p let spl par q p spl par sub 1 and d q spl par p respectively denote the l sub 1 distance and divergence between q and p we derive a refinement of pinsker s inequality of the form d q spl par p spl ges c p spl par q p spl par sub 1 sup 2 and characterize the best p dependent factor c p we apply the refined inequality to large deviations and measure concentration

learning decision trees with taxonomy of propositionalized attributes we consider the problem of exploiting a taxonomy of propositionalized attributes in order to learn compact and robust classifiers we introduce propositionalized attribute taxonomy guided decision tree learner pat dtl an inductive learning algorithm that exploits a taxonomy of propositionalized attributes as prior knowledge to generate compact decision trees since taxonomies are unavailable in most domains we also introduce propositionalized attribute taxonomy learner pat learner that automatically constructs taxonomy from data pat dtl uses top down and bottom up search to find a locally optimal cut that corresponds to the literals of decision rules from data and propositionalized attribute taxonomy pat learner propositionalizes attributes and hierarchically clusters the propositionalized attributes based on the distribution of class labels that co occur with them to generate a taxonomy our experimental results on uci repository data sets show that the proposed algorithms can generate a decision tree that is generally more compact than and is sometimes comparably accurate to those produced by standard decision tree learners

on surrogate loss functions and f divergences the goal of binary classification is to estimate a discriminant function y from observations of covariate vectors and corresponding binary labels we consider an elaboration of this problem in which the covariates are not available directly but are transformed by a dimensionality reducing quantizer q we present conditions on loss functions such that empirical risk minimization yields bayes consistency when both the discriminant function and the quantizer are estimated these conditions are stated in terms of a general correspondence between loss functions and a class of functionals known as ali silvey or divergence functionals whereas this correspondence was established by blackwell proc 2nd berkeley symp probab statist 1 1951 93 102 univ california press berkeley for the 0 1 loss we extend the correspondence to the broader class of surrogate loss functions that play a key role in the general theory of bayes consistency for binary classification our result makes it possible to pick out the strict subset of surrogate loss functions that yield bayes consistency for joint estimation of the discriminant function and the quantizer

derivation of the maximum entropy h controller and a state space formula for its entropy the problem of maximizing the entropy of a stabilized closed loop system is solved the solution exploits the parametrization of all closed loop systems that meet an h norm bound the central solution of this set is shown to maximize the entropy at infinity and for this case a particularly simple formula is derived for the maximum value of the entropy

escort evolutionary game theory abstract a family of replicator like dynamics called the escort replicator dynamic is constructed using information geometric concepts and generalized information divergences from information geometry and statistical thermodynamics a single formula lyapunov function is given that covers the entire class of dynamics which includes the replicator dynamic and the projection dynamic as well as several new dynamics a further class is discussed that allows for more variation such as variable intensities of selection

universal lower bound on the free energy cost of molecular measurements the living cell uses a variety of molecular receptors to read and process chemical signals that vary in space and time we model the dynamics of such molecular level measurements as markov processes in steady state with a coupling between the receptor and the signal we prove exactly that when the signal dynamics is not perturbed by the receptors the free energy consumed by the measurement process is lower bounded by a quantity proportional to the mutual information our result is completely independent of the receptor architecture and dependent on signal properties alone and therefore holds as a general principle for molecular information processing

f divergence inequalities this paper develops systematic approaches to obtain f divergence inequalities dealing with pairs of probability measures defined on arbitrary alphabets functional domination is one such approach where special emphasis is placed on finding the best possible constant upper bounding a ratio of f divergences another approach used for the derivation of bounds among f divergences relies on moment inequalities and the logarithmic convexity property which results in tight bounds on the relative entropy and bhattacharyya distance in terms of chi 2 divergences a rich variety of bounds are shown to hold under boundedness assumptions on the relative information special attention is devoted to the total variation distance and its relation to the relative information and relative entropy including reverse pinsker inequalities as well as on the e gamma divergence which generalizes the total variation distance pinsker s inequality is extended for this type of f divergence a result which leads to an inequality linking the relative entropy and relative information spectrum integral expressions of the renyi divergence in terms of the relative information spectrum are derived leading to bounds on the renyi divergence in terms of either the variational distance or relative entropy

on the burnside problem for periodic groups narain gupta 1 and said sidki 2 1 university of manitoba department of mathematics winnipeg manitoba r3t 2n2 canada 2 university of brasilia department of mathematics brasilia d f brazil introduction the generalized burnside problem refers to the question are finitely generated periodic groups finite this was answered in the negative by golod 1 who proved that for each prime p there exists a finitely generated infinite p group golod s construction in not however direct and is based on his celebrated work with safarevi recently grigor uk 2 has given a direct and elegant construction of an infinite 2 group which is generated by three elements of order 2 in this paper we give for each odd prime p a direct construction of an infinite p group on two generators each of order p our group is a subgroup of the automorphism group of a regular tree of degree p and as might be expected it is residually finite and has infinite exponent preliminaries let p be an odd prime and let t 0 be the infinite regular tree of degree p with vertex 0 so that through each vertex u of t 0 there are p regular subtrees

a transfer function approach to the problems of discrete time systems h sub infinity optimal linear control and filtering a solution is derived to the problems of h sub infinity optimal linear state regulation and filtering the solution method for both problems is based on a transfer function approach which applies standard spectral factorization return difference relations are given which are extensions of the relations associated with the linear quadratic problem

statistical information and discrimination in analogy with the definition of shannon information m h de groot 1962 defined statistical information as the difference between prior and posterior risk of a statistical decision problem relations are studied between the statistical information and the discrimination functions of information theory known as f divergences using previous results it is shown that every f divergence i sub f p q is an average statistical information or decision problem with dichotomic parameter 0 1 loss function and corresponding observation distributions p and q the average is taken over a distribution on the parameter s prior probability this distribution is uniquely determined by the function f the main result is that every f divergence is statistical information in some properly chosen statistical decision problem and conversely that every piece of statistical information is an f divergence this provides a new representation of discrimination functions figuring in signal detection data compression coding pattern classification cluster analysis etc

an entropy formula for time varying discrete time control systems the results of this paper generalize the formula for the entropy of a transfer function to time varying systems this is done through the use of some results on spectral factorizations due to arveson and properties of the cw transform which generalizes the usual cz transform for time varying systems using the formula defined it is shown that for linear fractional transformations like those that arise in time varying f control there exists a unique bounded contraction which maximizes the entropy this generalizes known results in the time invariant case possible extensions are discussed along with state space formulae

divergence measures based on the shannon entropy a novel class of information theoretic divergence measures based on the shannon entropy is introduced unlike the well known kullback divergences the new measures do not require the condition of absolute continuity to be satisfied by the probability distributions involved more importantly their close relationship with the variational distance and the probability of misclassification error are established in terms of bounds these bounds are crucial in many applications of divergence measures the measures are also well characterized by the properties of nonnegativity finiteness semiboundedness and boundedness

the relationship between minimum entropy control and risk sensitive control for time varying systems the connection between minimum entropy control and risk sensitive control for linear time varying systems is investigated for time invariant systems the entropy functional and the linear exponential quadratic gaussian cost are the same in this paper it is shown that this is not true for general time varying systems it does hold however when the system admits a state space representation

on pinsker s and vajda s type inequalities for csiszar s f divergences let d and v denote respectively information divergence and total variation distance pinsker s and vajda s inequalities are respectively d 1 2 v2 and d log 2 v 2 v 2v 2 v in this paper several generalizations and improvements of these inequalities are established for wide classes of f divergences first conditions on f are determined under which an f divergence df will satisfy df cf v2 or df c2 f v2 c4 f v4 where the constants cf c2 f and c4 f are best possible as a consequence lower bounds in terms of v are obtained for many well known distance and divergence measures including the 2 and hellinger s discrimination and the families of tsallis and renyi s divergences for instance if d p q 1 1 p q1 d 1 and p q 1 1 log p q1 d are respectively the relative information of type and the renyi s information gain of order it is shown that d 1 2 v2 1 72 1 2 v4 whenever 1 2 0 1 and that 2 v2 1 36 1 5 5 2 v4 for 0 1 in a somewhat different direction and motivated by the fact that these pinsker s type lower bounds are accurate only for small variation v close to zero lower bounds for df which are accurate for both small and large variation v close to two are also obtained in the special case of the information divergence they imply that d log 2 2 v 2 v 2 log 2 v 2 which uniformly improves vajda s inequality

information recovery from pairwise measurements this paper is concerned with jointly recovering n node variables left x i right 1 leq i leq n from a collection of pairwise difference measurements imagine we acquire a few observations taking the form of x i x j the observation pattern is represented by a measurement graph mathcal g with an edge set mathcal e such that x i x j is observed if and only if i j in mathcal e to account for noisy measurements in a general manner we model the data acquisition process by a set of channels with given input output transition measures employing information theoretic tools applied to channel decoding problems we develop a unified framework to characterize the fundamental recovery criterion which accommodates general graph structures alphabet sizes and channel transition measures in particular our results isolate a family of minimum channel divergence measures to characterize the degree of measurement corruption which together with the size of the minimum cut of mathcal g dictates the feasibility of exact information recovery for various homogeneous graphs the recovery condition depends almost only on the edge sparsity of the measurement graph irrespective of other graphical metrics alternatively the minimum sample complexity required for these graphs scales like n log n mathsf hel 1 2 min for certain information metric mathsf hel 1 2 min defined in the main text as long as the alphabet size is not super polynomial in n we apply our general theory to three concrete applications including the stochastic block model the random corruption model and the haplotype assembly problem our theory leads to orderwise tight recovery conditions for all these scenarios

a new metric for probability distributions we introduce a metric for probability distributions which is bounded information theoretically motivated and has a natural bayesian interpretation the square root of the well known spl chi sup 2 distance is an asymptotic approximation to it moreover it is a close relative of the capacitory discrimination and jensen shannon divergence

relations between maximum entropy h control and combined h lqg control abstract the maximum entropy h control problem and a combined h lqg control problem are presented in a unified way and shown to be equivalent in the case of a full order controller and equalized h lqg weights the result links two separate approaches which have distinctly different methods of solution only two decoupled algebraic riccati equations are needed to state the controller maximum value of the entropy and minimum value of the auxiliary cost

h sub infinity optimal one step ahead output feedback control of discrete time systems the discrete time one step ahead standard four blocks h sub infinity optimal control problem is considered the output feedback problem is transformed into a one step ahead h sub infinity optimal state estimation problem and the existing solution is used to derive the controller it is assumed that only previous measurements are available for control two results are obtained the first requires the solutions of two coupled riccati equations where the necessary and sufficient condition for the existence of the h sub infinity optimal result requires only the positive semidefiniteness of these solutions the second result applies two decoupled riccati equations with a coupling constraint as in the continuous time case and two additional conditions the derivation procedure is most simple and may thus provide considerable insight into the solution of the discrete time standard problem a similar procedure can also be used to obtain a solution to problems where the current measurement is available

wreath operations in the group of automorphisms of the binary tree abstract a new operation called tree wreathing is defined on groups of automorphisms of the binary tree given a countable residually finite 2 group h and a free abelian group k of finite rank r this operation produces uniformly copies of these as automorphism groups of the binary tree such that the group generated by them is an over group of the restricted wreath product h k indeed g contains a normal subgroup n which is an infinite direct sum of copies of the derived group h and the quotient group g n is isomorphic to h k the tree wreathing construction preserves the properties of solvability torsion freeness and of having finite state i e generated by finite automata a faithful representation of any free metabelian group of finite rank is obtained as a finite state group of automorphisms of the binary tree

direct state space solution to the discrete time h sub infinity one block problem a general state space representation is used to allow a complete formulation of the h sub infinity optimization problem without any invertibility condition on the system matrix unlike existing solutions a straightforward approach is used to solve the one block h sub infinity optimization problem the parameterization of all solutions to the discrete time h sub infinity suboptimal one block problem is first given in transfer function form in terms of a set of functions in h sub infinity that satisfy a norm bound the parameterization of all solutions is also given as a linear fractional representation

optimal hankel norm model reductions and weiner hopf factorization i the canonical case we consider the problem of approximating a given stable rational matrix function g s of mcmillan degree n by a function s f s where has mcmillan degree l n and f is antistable it is known that the minimum possible l norm of the error g f l or equivalently the minimum possible hankel norm of the stable part of the error g h is equal to the l 1 st hankel singular value l 1 g s of g we give an explicit linear fractional map parametrization for the class of all functions s f s as above which satisfy g f l l 1 g s the coefficients of the linear fractional map are completely determined by the matrices a b and c in a realization g s c si a 1b for g s and the unique solutions of a pair of lyapunov equations involving these matrices note that without loss of generality g 0 the basic idea is to use the approach of ball and helton to reduce the problem to a symmetric wiener hopf factorization problem which in turn can be solved by applying a result of kaashoek and ran the results obtained here are equivalent to the results of glover but our analysis gives an alternative more geometric approach

a characterization of all solutions to the four block general distance problem all solutions to the four block general distance problem which arises in h infty optimal control are characterized the procedure is to embed the original problem in an all pass matrix which is constructed it is then shown that part of this all pass matrix acts as a generator of all solutions special attention is given to the characterization of all optimal solutions by invoking a new descriptor characterization of all pass transfer functions as an application necessary and sufficient conditions are found for the existence of an h infty optimal controller following that a descriptor representation of all solutions is derived

on the automorphism group of the one rooted binary tree abstract let a be the automorphism group of the one rooted regular binary tree t 2 and g the subgroup of a consisting of those automorphisms admitting a finite description in their action on t 2 let n a g be the normaliser of g in a let aut g be the group of automorphisms of g and let end a g be the semigroup of endomorphisms of g induced by conjugation by elements of a then g is the infinite iterated wreath product c 2 c 2 and a is the topological limit of g we study in some detail the structure of g aut g and end a g in particular we prove n a g is isomorphic to aut g contains a copy of a itself and is a proper subgroup of end a g furthermore we discuss connections with automata and introduce the notion of functionally recursive automorphisms of t 2

lautum information a popular way to measure the degree of dependence between two random objects is by their mutual information defined as the divergence between the joint and product of marginal distributions we investigate an alternative measure of dependence the lautum information defined as the divergence between the product of marginal and joint distributions i e swapping the arguments in the definition of mutual information some operational characterizations and properties are provided for this alternative measure of information