Differential entropy. Differential entropy (also referred to as continuous entropy) is a concept in information theory that began as an attempt by Claude Shannon to extend the idea of (Shannon) entropy (a measure of average surprisal) of a random variable, to continuous probability distributions. Unfortunately, Shannon did not derive this formula, and rather just assumed it was the correct continuous analogue of discrete entropy, but it is not. The actual continuous version of discrete entropy is the  limiting density of discrete points (LDDP). Differential entropy (described here) is commonly encountered in the literature, but it is a limiting case of the LDDP, and one that loses its fundamental association with discrete entropy.

In terms of measure theory, the differential entropy of a probability measure is the negative relative entropy from that measure to the Lebesgue measure, where the latter is treated as if it were a probability measure, despite being unnormalized.

Definition
Let  be a random variable with a probability density function  whose support is a set . The differential entropy  or  is defined as

For probability distributions which do not have an explicit density function expression, but have an explicit quantile function expression, , then  can be defined in terms of the derivative of  i.e. the quantile density function  as

.

As with its discrete analog, the units of differential entropy depend on the base of the logarithm, which is usually 2 (i.e., the units are bits). See logarithmic units for logarithms taken in different bases. Related concepts such as joint, conditional differential entropy, and relative entropy are defined in a similar fashion. Unlike the discrete analog, the differential entropy has an offset that depends on the units used to measure . For example, the differential entropy of a quantity measured in millimeters will be  more than the same quantity measured in meters; a dimensionless quantity will have differential entropy of  more than the same quantity divided by 1000.

One must take care in trying to apply properties of discrete entropy to differential entropy, since probability density functions can be greater than 1. For example, the uniform distribution  has negative differential entropy; i.e., it is better ordered than  as shown now

being less than that of  which has zero differential entropy.  Thus, differential entropy does not share all properties of discrete entropy.

The continuous mutual information  has the distinction of retaining its fundamental significance as a measure of discrete information since it is actually the limit of the discrete mutual information of partitions of  and  as these partitions become finer and finer.  Thus it is invariant under non-linear homeomorphisms (continuous and uniquely invertible maps), including linear transformations of  and , and still represents the amount of discrete information that can be transmitted over a channel that admits a continuous space of values.

For the direct analogue of discrete entropy extended to the continuous space, see  limiting density of discrete points.

Properties of differential entropy
 For probability densities  and , the Kullback–Leibler divergence  is greater than or equal to 0 with equality only if  almost everywhere. Similarly, for two random variables  and ,  and  with equality if and only if  and  are independent.
 The chain rule for differential entropy holds as in the discrete case
.
 Differential entropy is translation invariant, i.e. for a constant .

 Differential entropy is in general not invariant under arbitrary invertible maps.
 In particular, for a constant 

 For a vector valued random variable  and an invertible (square) matrix 

 In general, for a transformation from a random vector to another random vector with same dimension , the corresponding entropies are related via

where  is the Jacobian of the transformation . The above inequality becomes an equality if the transform is a bijection. Furthermore, when  is a rigid rotation, translation, or combination thereof, the Jacobian determinant is always 1, and .
 If a random vector  has mean zero and covariance matrix ,  with equality if and only if  is jointly gaussian (see below).

However, differential entropy does not have other desirable properties:
 It is not invariant under change of variables, and is therefore most useful with dimensionless variables.
 It can be negative.
A modification of differential entropy that addresses these drawbacks is the relative information entropy, also known as the Kullback–Leibler divergence, which includes an invariant measure factor (see limiting density of discrete points).

Maximization in the normal distribution

Theorem
With a normal distribution, differential entropy is maximized for a given variance.  A Gaussian random variable has the largest entropy amongst all random variables of equal variance, or, alternatively, the maximum entropy distribution under constraints of mean and variance is the Gaussian.

Proof
Let  be a Gaussian PDF with mean μ and variance  and  an arbitrary PDF with the same variance. Since differential entropy is translation invariant we can assume that  has the same mean of  as .

Consider the Kullback–Leibler divergence between the two distributions

Now note that

because the result does not depend on  other than through the variance.  Combining the two results yields

with equality when  following from the properties of Kullback–Leibler divergence.

Alternative proof
This result may also be demonstrated using the calculus of variations. A Lagrangian function with two Lagrangian multipliers may be defined as:

where g(x) is some function with mean μ. When the entropy of g(x) is at a maximum and the constraint equations, which consist of the normalization condition  and the requirement of fixed variance , are both satisfied, then a small variation δg(x) about g(x) will produce a variation δL about L which is equal to zero:

Since this must hold for any small δg(x), the term in brackets must be zero, and solving for g(x) yields:

Using the constraint equations to solve for λ0 and λ yields the normal distribution:

Example: Exponential distribution
Let  be an exponentially distributed random variable with parameter , that is, with probability density function

Its differential entropy is then

Here,  was used rather than  to make it explicit that the logarithm was taken to base e, to simplify the calculation.

Relation to estimator error
The differential entropy yields a lower bound on the expected squared error of an estimator. For any random variable  and estimator  the following holds:

with equality if and only if  is a Gaussian random variable and  is the mean of .

Differential entropies for various distributions
In the table below  is the gamma function,  is the digamma function,  is the beta function, and γE is Euler's constant.

Many of the differential entropies are from.

Variants
As described above, differential entropy does not share all properties of discrete entropy. For example, the differential entropy can be negative; also it is not invariant under continuous coordinate transformations. Edwin Thompson Jaynes showed in fact  that the expression above is not the correct limit of the expression for a finite set of probabilities.

A modification of differential entropy adds an invariant measure factor to correct this, (see limiting density of discrete points). If  is further constrained to be a probability density, the resulting notion is called relative entropy in information theory:

The definition of differential entropy above can be obtained by partitioning the range of  into bins of length  with associated sample points  within the bins, for  Riemann integrable. This gives a quantized version of , defined by  if . Then the entropy of  is

The first term on the right approximates the differential entropy, while the second term is approximately . Note that this procedure suggests that the entropy in the discrete sense of a continuous random variable should be .

See also
Information entropy
Self-information
Entropy estimation

References

External links
 
 

Entropy and information
Information theory
Statistical randomness