the expected length of your code we encode a symbol will be always larger than the entropy of the distribution there's a janitor in 1948 entropy is lower bound on how many bits need to encode symbols coming from a certain distribution so let's step through what the key things are to get there this is just what we're starting from the difference between entropy and exit code length entropy is this thing here expected code did this look at all possible symbols and look at the length and take the weighted sum we have px I