feedback capacity of a class of symmetric finite state markov channels we consider the feedback capacity of a class of symmetric finite state markov channels here symmetry termed quasi symmetry is defined as a generalized version of the symmetry defined for discrete memoryless channels the symmetry yields the existence of a hidden markov noise process that depends on the channel s state process and facilitates the channel description as a function of input and noise where the function satisfies a desirable invertibility property we show that feedback does not increase capacity for such class of finite state channels and that both their nonfeedback and feedback capacities are achieved by an independent and uniformly distributed i u d input as a result the channel capacity is explicitly given as a difference of output and noise entropy rates where the output is driven by the i u d input

domain of definition of levermore s five moment system the simplest system in levermore s moment hierarchy involving moments higher than second order is the five moment closure it is obtained by taking velocity moments of the one dimensional boltzmann equation under the assumption that the velocity distribution is a maximum entropy function the moment vectors for which a maximum entropy function exists consequently make up the domain of definition of the system the aim of this article is a complete characterization of the structure of the domain of definition and the connected maximum entropy problem the space homogeneous case of the equation and numerical aspects are also addressed

links between functions and subdifferentials abstract a function in a class f x is said to be subdifferentially determined in f x if it is equal up to an additive constant to any function in f x with the same subdifferential a function is said to be subdifferentially representable if it can be recovered from a subdifferential we identify large classes of lower semicontinuous functions that possess these properties

lyapunov exponents and the natural invariant density determination of chaotic maps an iterative maximum entropy ansatz we apply the maximum entropy principle to construct the natural invariant density and the lyapunov exponent of one dimensional chaotic maps using a novel function reconstruction technique that is based on the solution of the hausdorff moment problem via maximizing shannon entropy we estimate the invariant density and the lyapunov exponent of nonlinear maps in one dimension from a knowledge of finite number of moments the accuracy and the stability of the algorithm are illustrated by comparing our results to a number of nonlinear maps for which the exact analytical results are available furthermore we also consider a very complex example for which no exact analytical result for the invariant density is available a comparison of our results to those available in the literature is also discussed

hausdorff moment problem and maximum entropy a unified approach hausdorff moment problem in the approach of maximum entropy is reconsidered some aspects as existence condition entropy convergence entropy decreasing stability are studied through a unified approach results known in literature and some new ones are illustrated

a piecewise quadratic maximum entropy method for the statistical study of chaos abstract let the frobenius perron operator p s l 1 0 1 l 1 0 1 related to a nonsingular transformation s 0 1 0 1 have an invariant density f we propose a piecewise quadratic maximum entropy method for the numerical approximation of f the role of the partition of unity property of the quadratic functions for the numerical recovery of f has been depicted the proposed algorithm overcomes the ill conditioning shortage of the traditional maximum entropy method which only employs polynomials so that any number of moments can be used to increase the accuracy of the computed invariant density the convergence rate of the method is shown to be of order 3 numerical results are presented to justify the theoretical analysis of the method

jointly optimal lqg quantization and control policies for multi dimensional systems for controlled bbr n valued linear systems driven by gaussian noise under quadratic cost criteria we investigate the existence and the structure of optimal quantization and control policies for fully observed and partially observed systems we establish the global optimality of a class of predictive encoders and show that an optimal quantization policy exists provided that the quantizers allowed are ones which have convex codecells furthermore optimal control policies are linear in the conditional estimate of the state and a form of separation of estimation and control holds

source coding with limited look ahead side information at the decoder we characterize the rate distortion function for the source coding with decoder side information setting when the ith reconstruction symbol is allowed to depend only on the first i lscr side information symbols for some finite look ahead lscr in addition to the index from the encoder for the case of causal side information i e lscr 0 we find that the penalty of causality is the omission of the subtracted mutual information term in the wyner ziv rate distortion function for lscr 0 we derive a computable infinite letter expression for the rate distortion function when specialized to the near lossless case our results characterize the best achievable rate for the slepian wolf source coding problem with finite side information looka head and have some surprising implications we find that side information is useless for any fixed lscr when the joint probability mass function pmf of the source and side information satisfies the positivity condition p x y 0 for all x y more generally the optimal rate depends on the distribution of the pair x y only through the distribution of x and the bipartite graph whose edges represent the pairs x y for which p x y 0 on the other hand if side information look ahead is allowed to grow faster than logarithmic in the block length then h x y is achievable finally we apply our approach to derive a computable expression for channel capacity when state information is available at the encoder with limited look ahead

stochastic nestedness and the belief sharing information pattern solutions to decentralized stochastic optimization problems lead to recursions in which the state space enlarges with the time horizon thus leading to non tractability of classical dynamic programming a common joint information state supplied to each of the agents leads to a tractable recursion as is evident in the one step delayed information sharing structure case or when deterministic nestedness in information holds when there is a causality relationship as in the case of partially nested information structure however communication requirements for such conditions require exchange of very large data noiselessly hence these assumptions are generally impractical in this paper we present a weaker notion of nestedness which we term as stochastic nestedness which is characterized by a sequence of markov chain conditions it is shown that if the information structure is stochastically nested then an optimization problem is tractable and in particular for lqg problems the team optimal solution is linear despite the lack of deterministic nestedness or partial nestedness one other contribution of this paper is that by regarding the multiple decision makers as a single decision maker and using witsenhausen s equivalent model for discrete stochastic control it is shown that the common state required need not consist of observations and it suffices to share beliefs on the state and control actions a pattern we refer to as k stage belief sharing pattern we discuss the minimum amount of information exchange required to achieve such an information pattern for k 1 the information exchange needed is generally strictly less than what is needed for deterministic nestedness and is zero whenever stochastic nestedness applies in view of nestedness we present a discussion on the monotone values of information channels

approximate solution of fredholm integral equations by the maximum entropy method an approximate means of solving fredholm integral equations by the maximum entropy method is developed the fredholm integral equation is converted to a generalized moment problem whose approximate solution by maximum entropy methods has been successfully implemented in a previous paper by mead and papanicolaou l r mead and n papanicolaou j math phys 25 2404 1984 several explicit examples are given of approximate maximum entropy solutions of fredholm integral equations of the first and second kinds and of the wiener hopf type both the weaknesses and strengths of the method are discussed

solving differential equations by a maximum entropy minimum norm method with applications to fokker planck equations the method of maximum entropy minimum norm is utilized to produce a general method of solving differential equations the technique is a generalization and extension of previous work performed by baker jarvis j math phys 30 302 1989 it is found that introducing an additional constraint on the norm of the solution vector produces a probability distribution that is integrable over the entire real axis a number of simplifications occur in this extended method the lagrange multipliers and solution vector can be solved for explicitly thus eliminating the necessity of solving systems of nonlinear equations for the lagrange multipliers as was required in the previous approach it is shown that the solution obtained is equivalent to a minimum norm approximation the maximum entropy solution of differential equations with fourier moments is shown to be identical to a fourier series solution additionally the new method is applied to solving the random walk and fokker planck equations

on the gaarder slepian tracking system conjecture source coding the authors examine under what conditions and with what notion of optimality the coder of an optimal system can be operated with the information which is also accessible to the decoder after a discussion of the difficulties involved a theorem is proved for markov sources which states that the extra memory of the coder can be substituted with independent randomization the result is not tied to any specific notion of optimality the main result is shown to hold for markov sources of order k under weak assumptions on the structure and with respect to any per letter fidelity criterion

function reconstruction as a classical moment problem a maximum entropy approach we present a systematic study of the reconstruction of non negative functions via maximum entropy approach utilizing the information contained in a finite number of moments of the functions for testing the efficacy of the approach we reconstruct a set of functions using an iterative entropy optimization scheme and study the convergence profile as the number of moments is increased a wide variety of functions are considered that include a distribution with a sharp discontinuity an oscillatory function a distribution with singularities and finally a distribution with several spikes and fine structure the last example is important in the context of the determination of the natural density of the logistic map the convergence of the method is studied by comparing the moments of the approximated functions with the exact ones furthermore by varying the number of moments and iterations we examine to what extent the features of the functions such as the divergence behavior at singular points within the interval is reproduced the proximity of the reconstructed maximum entropy solution to the exact solution is examined via kullback leibler divergence and variation measures for different number of moments

high order entropy based closures for linear transport in slab geometry ii a computational study of the optimization problem we present a numerical algorithm to implement entropy based m n moment models in the context of a simple linear kinetic equation for particles moving through a material slab the closure for these models as is the case for all entropy based models is derived through the solution of a constrained convex optimization problem the algorithm has two components the first component is a discretization of the moment equations which preserves the set of realizable moments thereby ensuring that the optimization problem has a solution in exact arithmetic the discretization is a second order kinetic scheme which uses muscl type limiting in space and a strong stability preserving runge kutta time integrator the second component of the algorithm is a newton based solver for the dual optimization problem which uses an adaptive quadrature to evaluate integrals in the dual objective and its derivatives the accuracy of the numerical solution to the dual problem plays a key role in the time step restric

a maximum entropy method for solving frobenius perron operator equations let s 0 1 0 1 be a nonsingular transformation that preserves an absolutely continuous invariant measure with density f in this paper we develop a general algorithm for approximating f using the maximum entropy approach

integration of subdifferentials of nonconvex functions in nonsmooth analysis and optimization subgradients come in many different flavors e g approximate dini proximal clarke generalized see 2 5 6 12 14 191 these subgradients are important and valuable tools however many questions remain unsolved concerning the exact link between the function and its subgradients for instance can two functions not differing by an additive constant have the same subgradients in this paper we study the fundamental problem of determining functions that can be recovered up to an additive constant from the knowledge of their subgradients this integration problem is not very well understood and very few functions or classes of functions are known to be recoverable from their subgradients in section 4 we show that if the basic constraint qualification holds at r then the composition of a closed i e lowersemicontinuous proper convex function with a twice continuously differentiable mapping is determined up to an additive constant by its generalized subgradients actually in this case all above mentioned subgradients are the same beside the obvious theoretical interest of this integration problem it is our hope or perhaps our long term goal that once this problem is better understood we can then tackle the question of uniqueness of solutions to generalized differential equations involving subgradients in place of partial derivatives an example of such an equation that well deserves study is the extended hamilton jacobi equation used in optimal control see clarke 2 let us also mention that a problem similar to the integration problem is the one of determining the set valued mappings that are in fact subgradient set valued mappings uniqueness is not mandatory for a contribution to this problem see janin 8 before we look at some of the known cases where the function can be recovered from its subgradients let us look at some negative examples it is clear that not every function can be recovered up to an additive constant from its subgradients we only need to look at the following two functions 0 x50 0 x10 f x 1 x o g x 2 x o

on the application of maximum entropy to the moments problem the maximum entropy approach to the solution of classical inverse problem of moments in which one seeks to reconstruct a function p x where x 0 from the values of a finite set n 1 of its moments is studied it is shown that for n 4 such a function always exists while for n 2 and n 3 the acceptable values of the moments are singled out analytically the paper extends to the general case where the results were previously bounded to the case n 2

on the structure of real time encoding and decoding functions in a multiterminal communication system a real time communication system with two encoders communicating with a single receiver over separate noisy channels is considered the two encoders make distinct partial observations of a markov source each encoder must encode its observations into a sequence of discrete symbols the symbols are transmitted over noisy channels to a finite memory receiver that attempts to reconstruct some function of the state of the markov source encoding and decoding must be done in real time that is the distortion measure does not tolerate delays under the assumption that the encoders observations are conditionally independent markov chains given an unobserved time invariant random variable results on the structure of optimal real time encoding and decoding functions are obtained it is shown that there exist finite dimensional sufficient statistics for the encoders the problem with noiseless channels and perfect memory at the receiver is then considered a new methodology to find the structure of optimal real time encoders is employed a sufficient statistic with a time invariant domain is found for this problem this methodology exploits the presence of common information between the encoders and the receiver when communication is over noiseless channels

quasidifferentials and maximal normal operators quasidifferentials are studied with the theory of maximal normal operators the quasidifferential of a normally quasidifferentiable function is a pair of upper and lower semicontinuous operators which are special maximal normal operators the function sum of the upper and lower semicontinuous operators is the clarke subdifferential of this function basic calculus and minimal forms of quasidifferentials are also discussed

semismoothness and decomposition of maximal normal operators abstract a function is semismooth if and only if its subdifferential is semicontinuous all the semicontinuous operators form a big linear subspace of the maximal normal operator space by big we mean that this subspace includes all the continuous operators and all the maximal monotone operators all the decomposable maximal normal operators form another big subspace

universal zero delay joint source channel coding we consider zero delay joint source channel coding of individual source sequences for a general known channel given an arbitrary finite set of schemes with finite memory not necessarily time invariant decoders a scheme is devised that does essentially as well as the best in the set on all individual source sequences using this scheme we construct a universal zero delay joint source channel coding scheme that is guaranteed to achieve asymptotically the performance of the best zero delay encoding decoding scheme with a finite state encoder and a markov decoder on all individual sequences for the case where the channel is a discrete memoryless channel dmc we construct an implementable zero delay joint source channel coding scheme that is based on the follow the perturbed leader scheme of gyoumlrgy for lossy source coding of individual sequences our scheme is guaranteed to attain asymptotically the performance of the best in the set of all encoding decoding schemes with a symbol by symbol decoder and arbitrary encoder on all individual sequences

new developments of the extended quadrature method of moments to solve population balance equations abstract population balance models have a wide range of applications in many industrial fields as they allow accounting for heterogeneity among properties which are crucial for some system modelling they actually describe the evolution of a number density function ndf using a population balance equation pbe for instance they are applied to gas liquid columns or stirred reactors aerosol technology crystallisation processes fine particles or biological systems there is a significant interest for fast stable and accurate numerical methods in order to solve for pbes a class of such methods actually does not solve directly the ndf but resolves their moments these methods of moments and in particular quadrature based methods of moments have been successfully applied to a variety of systems point wise values of the ndf are sometimes required but are not directly accessible from the moments to address these issues the extended quadrature method of moments eqmom has been developed in the past few years and approximates the ndf from its moments as a convex mixture of kernel density functions kdfs of the same parametric family in the present work eqmom is further developed on two aspects the main one is a significant improvement of the core iterative procedure of that method the corresponding reduction of its computational cost is estimated to range from 60 up to 95 the second aspect is an extension of eqmom to two new kdfs used for the approximation the weibull and the laplace kernels all matlab source codes used for this article are provided with this article

optimal causal coding decoding problems the symbols produced by a finite markov source are causally encoded so as to be transmitted through a noisy memoryless channel the encoder is assumed to have channel feedback information and the decoder to be causal the feedback information is shown to be useful in general separation results are derived and used to prove that encoding is useless for a class of symmetric channels

maximum entropy estimation of density and regression functions abstract this paper presents an information efficient technique to determine the functional forms of density and regression functions by maximizing entropy subject to given side conditions a flexible functional form approach is established as special cases of this approach exponential polynomial cobb douglas translog generalized cobb douglas generalized leontief fourier flexible form and minflex laurent series etc can be derived in a monte carlo experiment the maximum entropy regression method shows good performance in terms of goodness of fit boxplots and isoquants produced by the maximum entropy method are compared with those produced by other well known functions

second order necessary conditions in constrained semismooth optimization first and second order conditions are given which must necessarily be satisfied by local minimizers for certain finite dimensional nonsmooth nonlinear programming problems the problems considered are of standard form having a finite number of equality and inequality constraints the principal result does not require a constraint qualification but does require that the functions be semismooth at the minimizer the necessary conditions are stated in terms of the generalized gradients of nonsmooth analysis and certain second order directional derivatives

fourth order pseudo maximum likelihood methods we extend pml theory to account for information on the conditional moments up to order four but without assuming a parametric model to avoid a risk of misspecification of the conditional distribution the key statistical tool is the quartic exponential family which allows us to generalize the pml2 and qgpml1 methods proposed in gourieroux et al 1984 to pml4 and qgpml2 methods respectively an asymptotic theory is developed the key numerical tool that we use is the gauss freud integration scheme that solves a computational problem that has previously been raised in several fields simulation exercises demonstrate the feasibility and robustness of the methods

hamburger moment problem and maximum entropy on the existence conditions the existence of maximum entropy solution for the reduced hamburger moment problem is reconsidered existence conditions previously appeared in literature are revisited allowing an easy way to identifying the existence of maximum entropy solution the obtained results suggest that except for special sequences of moments unknown a priori the maximum entropy solution for the non symmetric reduced hamburger moment problem exists for practical purposes the replacing of the support r with a large enough finite interval finds a theoretical warranty the symmetric case may be formulated as follows once assigned the first 2m moments if maxent density does not exist conclusion drawn uniquely from numerical evidence maxent density with the first 2m 2 moments exists in such a case even if the first 2m moments are known we have to settle for a density which carries less information theoretical results are illustrated through some numerical examples

a maximum entropy approach to recovering information from multinomial response data abstract the classical maximum entropy me approach to estimating the unknown parameters of a multinomial discrete choice problem which is equivalent to the maximum likelihood multinomial logit ml estimator is generalized the generalized maximum entropy gme model includes noise terms in the multinomial information constraints each noise term is modeled as the mean of a finite set of a priori known points in the interval 1 1 with unknown probabilities where no parametric assumptions about the error distribution are made a gme model for the multinomial probabilities and for the distributions associated with the noise terms is derived by maximizing the joint entropy of multinomial and noise distributions under the assumption of independence the gme formulation reduces to the me in the limit as the sample grows large or when no noise is included in the entropy maximization further even though the gme and the logit estimators are conceptually different the dual gme model is related to a gener

a practical computational framework for the multidimensional moment constrained maximum entropy principle the maximum entropy principle is a versatile tool for evaluating smooth approximations of probability density functions with a least bias beyond given constraints in particular the moment based constraints are often a common prior information about a statistical state in various areas of science including that of a forecast ensemble or a climate in atmospheric science with that in mind here we present a unified computational framework for an arbitrary number of phase space dimensions and moment constraints for both shannon and relative entropies together with a practical usable convex optimization algorithm based on the newton method with an additional preconditioning and robust numerical integration routine this optimization algorithm has already been used in three studies of predictability and so far was found to be capable of producing reliable results in one and two dimensional phase spaces with moment constraints of up to order 4 the current work extensively references those earlier studies as practical examples of the applicability of the algorithm developed below

integration of multivalued operators and cyclic submonotonicity we introduce a notion of cyclic submonotonicity for multivalued operators from a banach space x to its dual we show that if the clarke subdifferential of a locally lipschitz function is strictly submonotone on an open subset u of x then it is also maximal cyclically submonotone on u and conversely that every maximal cyclically submonotone operator on u is the clarke subdifferential of a locally lipschitz function which is unique up to a constant if u is connected in finite dimensions these functions are exactly the lower c 1 functions considered by spingarn and rockafellar

coarse grained computation for particle coagulation and sintering processes by linking quadrature method of moments with monte carlo the study of particle coagulation and sintering processes is important in a variety of research studies ranging from cell fusion and dust motion to aerosol formation applications these processes are traditionally simulated using either monte carlo methods or integro differential equations for particle number density functions in this paper we present a computational technique for cases where we believe that accurate closed evolution equations for a finite number of moments of the density function exist in principle but are not explicitly available the so called equation free computational framework is then employed to numerically obtain the solution of these unavailable closed moment equations by exploiting through intelligent design of computational experiments the corresponding fine scale here monte carlo simulation we illustrate the use of this method by accelerating the computation of evolving moments of uni and bivariate particle coagulation and sintering through short simulation bursts of a constant number monte carlo scheme

integration of fenchel subdifferentials of epi pointed functions it is shown that in finite dimensions rockafellar s technique of integrating cyclically monotone operators applied to the fenchel subdifferential of an epi pointed function yields the closed convex hull of that function

real time coding with limited lookahead a real time coding system with lookahead consists of a memoryless source a memoryless channel an encoder which encodes the source symbols sequentially with knowledge of future source symbols up to a fixed finite lookahead d with or without feedback of the past channel output symbols and a decoder which sequentially constructs the source symbols using the channel output the objective is to minimize the expected per symbol distortion for a fixed finite lookahead d 1 we invoke the theory of controlled markov chains to obtain an average cost optimality equation acoe the solution of which denoted by d d is the minimum expected per symbol distortion with increasing d d d bridges the gap between causal encoding d 0 where symbol by symbol encoding decoding is optimal and the infinite lookahead case d where shannon theoretic arguments show that separation is optimal we extend the analysis to a system with finite state decoders with or without noise free feedback for a bernoulli source and binary symmetric channel under hamming loss we compute the optimal distortion for various source and channel parameters and thus obtain computable bounds on d d we also identify regions of source and channel parameters where symbol by symbol encoding decoding is suboptimal finally we demonstrate the wide applicability of our approach by applying it in additional coding scenarios such as the case where the sequential decoder can take cost constrained actions affecting the quality or availability of side information about the source

on optimal causal coding of partially observed markov sources in single and multiterminal settings the optimal causal zero delay coding of a partially observed markov process is studied where the cost to be minimized is a bounded nonnegative additive measurable single letter function of the source and the receiver output a structural result is obtained extending witsenhausen s and walrand varaiya s structural results on optimal causal coders to more general state spaces and to a partially observed setting the decentralized multiterminal setup is also considered for the case where the source is an i i d process it is shown that an optimal solution to the decentralized causal coding of correlated observations problem is memoryless for markov sources a counterexample to a natural separation conjecture is presented

essentially smooth lipschitz functions in this paper we address some of the most fundamental questions regarding the differentiability structure of locally lipschitz functions defined on separable banach spaces for example we examine the relationship between integrability d representability and strict differentiability in addition to this we show that on any separable banach space there is a significant family of locally lipschitz functions that are integrable d representable and possess desirable differentiability properties we also present some striking applications of our results to distance functions

structure theorems for real time variable rate coding with and without side information the output of a discrete markov source is to be encoded instantaneously by a variable rate encoder and decoded by a finite state decoder our performance measure is a linear combination of the distortion and the instantaneous rate structure theorems pertaining to the encoder and next state functions are derived for every given finite state decoder which can have access to side information

maximum entropy autoregressive conditional heteroskedasticity model abstract in many applications it has been found that the autoregressive conditional heteroskedasticity arch model under the conditional normal or student s t distributions are not general enough to account for the excess kurtosis in the data moreover asymmetry in the financial data is rarely modeled in a systematic way in this paper we suggest a general density function based on the maximum entropy me approach that takes account of asymmetry excess kurtosis and also of high peakedness the me principle is based on the efficient use of available information and as is well known many of the standard family of distributions can be derived from the me approach we demonstrate how we can extract information functional from the data in the form of moment functions we also propose a test procedure for selecting appropriate moment functions our procedure is illustrated with an application to the nyse stock returns the empirical results reveal that the me approach with a fewer moment functions leads to a model that captures the stylized facts quite effectively

the structure of us food demand abstract an exactly aggregable system of gorman engel curves for us food consumption is developed and implemented box cox transformations on prices and income nest functional form the model nests rank up to rank three the model is estimated by nonlinear three stage least squares with annual time series data on 21 foods 17 nutrients age and race demographics and the distribution of income for 1919 1941 and 1947 2000 results are consistent with full rank three point estimates for the box cox parameters on income and prices are 0 86 and 1 09 respectively strongly rejecting zero and one in both cases no statistical evidence of serial correlation specification errors or parameter instability is found

two dimensional distribution of streamwise velocity in open channel flow using maximum entropy principle incorporation of additional constraints based on conservation laws abstract this study derived the vertical and transverse distribution of streamwise velocity in open channels using the information theoretic concept involving the maximum entropy maxent principle moment constraints expressed by the conservation of mass momentum and energy and a hypothesis invoking the connection between probability and space domains through a generalized coordinate system explicit analytical solutions were obtained for velocity profiles using a non perturbation approach along with pade approximation the convergence of the series solution was shown both theoretically and numerically the lagrange multipliers arising through the application of maxent principle were obtained by minimizing a convex potential and the quasi newton method along with bfgs scheme was applied for solving the unconstrained optimization problem for applying the conservation laws the momentum and energy coefficients were found to influence the velocity profile in a channel cross section with the expressions of these coefficients the derived velocity equations were validated for a wide range of experimental and field data

maximum entropy approach to classical hard sphere and hard disk equations of state the maximum entropy procedure for extrapolating series is applied to sum the known virial series for the hard disk and hard sphere fluids this procedure is found to produce numerical results more accurate than pade approximants and it agrees with monte carlo simulations of these systems the maximum entropy results extrapolate the series smoothly to densities arbitrarily near to maximum close packing and predict no singularities whatsoever

optimal zero delay coding of markov sources stationary and finite memory codes the optimal zero delay coding of a finite state markov source is considered the existence and structure of optimal codes are studied using a stochastic control formulation prior results in the literature established the optimality of deterministic markov walrand varaiya type coding policies for the finite time horizon problem and the optimality of both deterministic nonstationary and randomized stationary policies for the infinite time horizon problem our main result here shows that for any irreducible and aperiodic markov source with a finite alphabet deterministic and stationary markov coding policies are optimal for the infinite horizon problem in addition the finite block length time horizon performance of an optimal stationary and markov coding policy is shown to approach the infinite time horizon optimum at a rate o 1 t the results are extended to systems where zero delay communication takes place across a noisy channel with noiseless feedback

a maximum entropy method for solving the boundary value problem of second order ordinary differential equations we propose a new method to solve the boundary value problem for a class of second order linear ordinary differential equations which has a non negative solution the method applies the maximum entropy principle to approximating the solution numerically the theoretical analysis and numerical examples show that our method is convergent we propose a new method to solve the boundary value problem for a class of second order linear ordinary differential equations which has a non negative solution the method applies the maximum entropy principle to approximating the solution numerically the theoretical analysis and numerical examples show that our method is convergent

on causal source codes with side information we study the effect of the introduction of side information into the causal source coding setting of neuhoff and gilbert we find that the spirit of their result namely the sufficiency of time sharing scalar quantizers followed by appropriate lossless coding for attaining optimum performance within the family of causal source codes extends to many scenarios involving availability of side information at both encoder and decoder or only on one side for example in the case where side information is available at both encoder and decoder we find that time sharing side information dependent scalar quantizers at most two for each side information symbol attains optimum performance this remains true even when the reproduction sequence is allowed noncausal dependence on the side information and even for the case where the source and the side information rather than consisting of independent and identically distributed i i d pairs form respectively the output of a memoryless channel and its stationary ergodic input

the maximal normal operator space and integration of subdifferentials on nonconvex functions on definit des operateurs normaux maximaux et des fonctions primales sur des ensembles normaux on trouve que la normalite maximale est conservee dans plusieurs operations et que les operateurs normaux maximaux ont de bonnes proprietes de fermeture et de limitation

maximum entropy in the problem of moments the maximum entropy approach to the solution of underdetermined inverse problems is studied in detail in the context of the classical moment problem in important special cases such as the hausdorff moment problem we establish necessary and sufficient conditions for the existence of a maximum entropy solution and examine the convergence of the resulting sequence of approximations a number of explicit illustrations are presented in addition to some elementary examples we analyze the maximum entropy reconstruction of the density of states in harmonic solids and of dynamic correlation functions in quantum spin systems we also briefly indicate possible applications to the lee yang theory of ising models to the summation of divergent series and so on the general conclusion is that maximum entropy provides a valuable approximation scheme a serious competitor of traditional pade like procedures

optimal linear joint source channel coding with delay constraint the problem of joint source channel coding is considered for a stationary remote noisy gaussian source and a gaussian channel the encoder and decoder are assumed to be causal and their combined operations are subject to a delay constraint it is shown that under the mean square error distortion metric an optimal encoder decoder pair from the linear and time invariant lti class can be found by minimization of a convex functional and a spectral factorization the functional to be minimized is the sum of the well known cost in a corresponding wiener filter problem and a new term which is induced by the channel noise and whose coefficient is the inverse of the channel s signal to noise ratio this result is shown to also hold in the case of vector valued signals assuming parallel additive white gaussian noise channels it is also shown that optimal lti encoders and decoders generally require infinite memory which implies that approximations are necessary a numerical example is provided which compares the performance to the lower bound provided by rate distortion theory less

on optimal zero delay coding of vector markov sources optimal zero delay coding quantization of a vector valued markov source driven by a noise process is consid ered using a stochastic control problem formulation the exis tence and structure of optimal quantization policies are studied for a finite horizon problem with bounded per stage distortion measure the existence of an optimal zero delay quantization policy is shown provided that the quantizers allowed are ones with convex codecells the bounded distortion assumption is relaxed to cover cases that include the linear quadratic gaussian problem for the infinite horizon problem and a stationary markov source the optimality of deterministic markov coding policies is shown the existence of optimal stationary markov quantization policies is also shown provided randomization that is shared by the encoder and the decoder is allowed

maximum entropy approximation for lyapunov exponents of chaotic maps let s 0 1 0 1 be a nonsingular chaotic map that preserves an integrable density f that describes the statistics of the orbits in this article we use the maximum entropy approach to approximate the density f and the corresponding lyapunov exponent

information theoretic measures of the income distribution in food demand a new method to nest estimate and test the rank and functional form of the income terms in an incomplete system of demand equations is developed information theory is employed to infer the u s income distribution from data on quintile and top five percentile income ranges and intra quintile and top five percentile mean incomes maximum entropy income distributions are combined with data on the u s demand for 21 food items to estimate u s food demand over the period 1919 1995 excluding 1942 1946

a new encoder for continuous time gaussian signals with fixed rate and reconstruction delay in this paper we propose a method for encoding continuous time gaussian signals subject to a usual data rate constraint and more importantly a reconstruction delay constraint we first apply a karhunen loeve decomposition to reparameterize the continuous time signal as a discrete sequence of vectors we then study the optimal recursive quantization of this sequence of vectors since the optimal scheme turns out to have a very cumbersome design we consider a simplified method for which a numerical example suggests that the incurred performance loss is negligible in this simplified method we first build a state space model for the vector sequence and then use bayesian tracking to sequentially encode each vector the tracking task is performed using particle filtering numerical experiments show that the proposed approach offers visible advantages over other available approaches especially when the reconstruction delay is small

uniqueness of the maximal extension of a monotone operator theorem benilan under the above hypothesis the graph of a is equal to x x ecldoma x e vy a y x y x y ao 1 2 benilan did not publish his result but s menou quoted it with its proof in lo h attouch 9 also proved these two results for the case when a is the minimal section of a and applied them to the study of measurable dependence for a family of maximal monotone operators in hilbert spaces these results can be interpreted as saying that a monotone operator can be uniquely extended to a maximal monotone operator within the closure of the convex hull of its domain if the closure of the domain of the original operator is convex and the interior of the convex hull of its domain is nonempty in section 2 we give a direct proof of such a theorem without using robert s theorem it is interesting to generalize 1 1 to the whole domain of a the difficulty is the unboundedness of a maximal monotone operator at its boundary points in section 3 we prove that a monotone operator is bounded on any open segment connecting an interior point and a boundary point belonging to its domain as a corollary of this result we also prove that a maximal monotone operator is bounded in a bounded convex set pin the interior of its domain

efficient numerical approximation of maximum entropy estimates a class of algorithms for approximation of the maximum entropy estimate of probability density functions on the basis of a finite number of sampled data is introduced the algorithms are presented as a finite sequence in order of increasing accuracy and decreasing computational efficiency in which the last element of the sequence is the exact maximum entropy estimate numerical and applicative examples are reported

observer participant models of neural processing a model is proposed in which the neuron serves as an information channel channel distortion occurs through the channel since the mapping from input boolean codes to output codes are many to one in that neuron outputs consist of just two distinguished states within the described model the neuron performs a decision making function decisions are made regarding the validity of a question passively posed by the neuron this question becomes defined through learning hence learning is viewed as the process of determining an appropriate question based on supplied input ensembles an application of the shannon information measures of entropy and mutual information taken together in the context of the proposed model lead to the hopfield neuron model with conditionalized hebbian learning rules neural decisions are shown to be based on a sigmoidal transfer characteristic or in the limit as computational temperature tends to zero a maximum likelihood decision rule the described work is contrasted with the information theoretic approach of linsker

complete closedness of maximal monotone operators this paper is dedicated to the memory of professor kwan chao chin former academic member and director of the institute of system science of the academy of chinese sciences a maximal monotone operator in rn is completely closed i e not only closed for points but also closed for directions such a completely closed operator is locally bounded at a point if and only if it is bounded at that point

semiregularity and generalized subdifferentials with applications to optimization the michel penot subdifferential of a locally lipschitzian function is the principal part of the clarke subdifferential it coincides with the g derivative at differentiable points a locally lipschitzian function can be determined by its michel penot subdifferential uniquely up to an additive constant though this cannot be done by its clarke subdifferential if the set of abnormal points is not negligible a set valued operator is the michel penot subdifferential of a locally lipschitzian function if and only if it is a seminormal operator satisfying a cyclical condition various calculus rules hold for the michel penot subdifferential equalities hold for these rules at a point under semiregularity which is weaker than regularity for a locally lipschitzian function in a separable banach space semiregularity holds everywhere except for a haar zero set applications in optimization are discussed

calculation of maximum entropy densities with application to income distribution the maximum entropy approach is a flexible and powerful tool for density approximation this paper proposes a sequential updating method to calculate the maximum entropy density subject to known moment constraints instead of imposing the moment constraints simultaneously the sequential updating method incorporates the moment constraints into the calculation from lower to higher moments and updates the density estimates sequentially the proposed method is employed to approximate the size distribution of u s family income empirical evidence demonstrates the efficiency of this method

higher order moments and the maximum entropy inference the thermodynamic limit approach according to jaynes 1963 statistical mechanics can be interpreted as a special type of statistical inference based on the principle of maximum entropy the result of such an inference depends on the available information about a given physical system but the principle itself does not decide what kind of information is essential and what is not the well known gibbs canonical state results from the principle when the statistical mean value of energy is supposed to be known the maximum entropy inference employing higher order moments of energy is examined from the point of view of the thermodynamic limit it is shown that the contribution to the total entropy arising from the extra information corresponding to the higher order moments is o n when n to infinity n v constant n is the number of particles and v is the volume consequently from a purely thermodynamic point of view this extra information is non essential and can be neglected in the maximum entropy procedure these conclusions also characterise the maximum entropy inference employing higher order moments of almost arbitrary extensive quantities one can say that the maximum entropy inference has a certain stability property with respect to information corresponding to higher order moments of extensive physical quantities

on the structure of optimal real time encoders and decoders in noisy communication the output of a discrete time markov source must be encoded into a sequence of discrete variables the encoded sequence is transmitted through a noisy channel to a receiver that must attempt to reproduce reliably the source sequence encoding and decoding must be done in real time and the distortion measure does not tolerate delays the structure of real time encoding and decoding strategies that jointly minimize an average distortion measure over a finite horizon is determined the results are extended to the real time broadcast problem and a real time variation of the wyner ziv problem

second order necessary conditions in semismooth optimization first and second order conditions are given which are necessary for a functionf to have a local minimal value atx inrn it is assumed thatf is locally lipschitzian nearx and semismooth atx the necessary conditions are expressed in terms of the generalized gradients of nonsmooth analysis and certain second order directional derivatives the method of proof bears no resemblance to standard methods three special cases are discussed here but applications to constrained problems are made elsewhere

optimal design of sequential real time communication systems optimal design of sequential real time communication of a markov source over a noisy channel is investigated in such a system the delay between the source output and its reconstruction at the receiver should equal a fixed prespecified amount an optimal communication strategy must minimize the total expected symbol by symbol distortion between the source output and its reconstruction design techniques or performance bounds for such real time communication systems are unknown in this paper a systematic methodology based on the concepts of information structures and information states to search for an optimal real time communication strategy is presented this methodology trades off complexity in communication length linear in contrast to doubly exponential with complexity in alphabet sizes doubly exponential in contrast to exponential as the communication length is usually order of magnitudes bigger than the alphabet sizes the proposed methodology simplifies the search for an optimal communication strategy in spite of this simplification the resultant optimality equations cannot be solved efficiently using existing algorithmic techniques the main idea is to formulate a zero delay communication problem as a dynamic team with nonclassical information structure then an appropriate choice of information states converts the dynamic team problem into a centralized stochastic control problem in function space thereafter markov decision theory is used to derive nested optimality equations for choosing an optimal design for infinite horizon problems these optimality equations give rise to a fixed point functional equation communication systems with fixed finite delay constraint a higher order markov source and channels with memory are treated in the same manner after an appropriate expansion of the state space thus this paper presents a comprehensive methodology to study different variations of real time communication

the maximum entropy method applied to stationary density computation abstract the maximum entropy method maxent is widely used in the context of the moment problem which appears naturally in many branches of physics and engineering it is used to numerically recover the density with least bias from finitely many known moments we introduce the basic idea behind this method and apply this method to approximating fixed densities of markov operators in stochastic analysis and frobenius perron operators in ergodic theory of chaotic dynamics

entropy densities with an application to autoregressive conditional skewness and kurtosis abstract the entropy principle yields for a given set of moments a density that involves the smallest amount of prior information we first show how entropy densities may be constructed in a numerically efficient way as the minimization of a potential next for the case where the first four moments are given we characterize the skewness kurtosis domain for which densities are defined this domain is found to be much larger than for hermite or edgeworth expansions last we show how this technique can be used to estimate a garch model where skewness and kurtosis are time varying we find that there is little predictability of skewness and kurtosis for weekly data