From: kv07@IASTATE.EDU (Warren Vonroeschlaub)
Subject: Re: Definition/Information
Reply-To: kv07@IASTATE.EDU (Warren Vonroeschlaub)
Organization: Ministry of Silly Walks
Date: Sun, 22 Nov 1992 19:51:34 GMT

  Sorry for the delay on this, but somebody took out all 30+ books on
introductory information theory from the school library (they really should have
a withdrawl limit).  I had to use one of the advanced books, which is a little
harder to read.

In article <iHsFuB1w165w@kalki33>, kalki33!system@lakes.trenton.sc.us writes:
> DEFINITIONS
> 
> Let S be a system of events E[1],...,E[n] such that
> 
> 1)  P(E[k]) = p[k]  (the probability associated with event E[k])
> 
> 2)  0<=p[k]<=1
> 
>      n
> 3)  SUM p[k] = 1
>     k=1
> 
> The self-information of the event E[k] is defined as
> 
>  I(E[k]) = -log p[k]                            (1.1)
> 
> 
> The entropy of S is defined as
> 
>           n
> H(S) = - SUM (p[k] log p[k])                    (1.2)
>          k=1

  Okay, there are a couple of things I want to make clear.

First (in prediction to where this is going.  I hope I am wrong or this will be
short).

  In "Relative Information: Theories and Applications" by G. Jumarie, Q360.J84
1990 p6, section 1.3.4 "Information and Thermodynamics"

  "To the best of our [researchers] knowledge, there is no counterpart of this
theory [the second law of thermodynamics] in information theory."

  In other words, this definition of entropy does not entail that entropy must
always increase.

Second.

  This is not the only information theory of entropy.  This is known as the
Shannon entropy.  The Renyi entropy is also good, especially since it can be
used to determine the relative information content.  this is very important in
modeling events in a specific language (like the physical laws).  The Renyi
entropy is
                         n
  Hc(alpha)=(1-c)^-1 ln SUM p[k]^c
                        k=1
where c is the efficiency of detection of the observer.  An interesting point is
that if c>1 then this is a measure of information, and if c<1 this is a measure
of uncertainty.  (I assume to determine the probability of evolution you are
going to try to calculate the Shannon uncertainty).

 There is also the cross-entropy, Hartley, Kulmogorov (for fractals), and
trajectory entropies.  And that is just from one book.

Third.

  It might be useful to point out how this is used in information theory.  A few
examples are in order:

  Streams.  If we have a length of symbols, and we wish to find the entropy of
the stream, we can use the Shannon entropy as such:  A completely random stream
would be split evenly among the symbols.  The Shannon entropy of binary strings
would then be 1.  A stream consisting solely of 1s or 0s would have an entropy
of 0 (try it).  But, if our stream was 0111100011101, the shannon entropy would
be .9612 (-5/13 *log(5/13) -8/13 *log(8/13), all logarithms base 2).  If we do
this for alternations between two states however, we get 2.4142.  So the choice
of symbols is important. The choice of logarithms isn't.  CS people prefer log
base 2, but ln is poular too.

  Descision processes.  Let us say we wanted to find the most significant factor
for car color.  Well, then, for each car color we could do the following: 
Separate all the cars according to a factor (say manufacturer), then look at
what percentage of the cars in each catagory are of the particular color, and
make that our p[k] for the Shannon entropy (ie p[mazda] for red would be the
proportion of mazdas that are red).  This gives us a Shannon entropy for each
color car as divided by manufacturer.  Take each Shannon entropy for the color,
and multiply it by the fraction of cars overall that are that color.  Add this
all up to get the entropy of car color by manufacturer.
  If we repeat this for every possible catagory, we will have entropies
calculated for manufacturer, cost, area, use, etc.  The factor that has the
lowest entropy is the most important factor in car color.  Neat huh?
  This has an incredible power as a completely objective way to determine what
is important in any situation.

  Hopefully these examples will be enough to make it clear when Shannon entropy
is being misused.

 |  __L__
-|-  ___  Warren Kurt vonRoeschlaub
 |  | o | kv07@iastate.edu
 |/ `---' Iowa State University
/|   ___  Math Department
 |  |___| 400 Carver Hall
 |  |___| Ames, IA  50011
 J  _____