Cross-Entropy for Dummies. A simple and intuitive explanation of... | by Viraj Kulkarni | Towards Data Science

Cross-entropy is commonly used as a loss function for classification problems, but due to historical reasons, most explanations of cross-entropy are based on communication theory which data scientists may not be familiar with. You cannot understand cross-entropy without understanding entropy, and you cannot understand entropy without knowing what information is. This article builds the concept of cross-entropy in an easy-to-understand manner without relying on its communication theory background.
Let’s consider three illustrative experiments.
(1) I toss a coin, and I give you a message that the coin came up heads. The message conveyed some information to you. How much information? Let’s say you got one bit of information from this message.
Coin came up heads (probability=0.5) | Information = 1 bit
(2) I woke you up in the morning and told you that the sun rose. You shrugged and angrily went back to sleep, because this message was obvious and gave you no new information.
Sun rose (probability=1) | Information = 0 bits
(3) I woke you up in the morning and told you the sun did not rise. You came out shocked and saw the world had gone crazy. This was breaking news — lots of information.
Sun did not rise (probability=0) | Information = ∞ bits
From the above examples, you might have deduced that the information contained in a message about an event is related to the uncertainty and surprise-value of the event. An occurrence of an unlikely event gives you more information than the occurrence of a likely event. Claude Shannon formalised this intuition behind information in his seminal work on Information Theory. He defined information as:
I(x) = -log2P(x) ..where P(x) is probability of occurrence of x
Information quantifies the uncertainty in one single event.
What if you are interested not in one single event but in a sequence of events? Consider the following example. A bin contains 2 red, 3 green, and 4 blue balls. Now, instead of tossing a coin, I pick out one ball at random and give it to you. What is the expected amount of information you will receive every time I pick one ball? We model picking out a ball as a stochastic process represented by a random variable X. The entropy of X is then defined as the expected value of the information conveyed by an outcome in X. Using our above definition of information, we get:
P(red ball) = 2/9; I(red ball) = -log2(2/9)P(green ball) = 3/9; I(green ball) = -log2(3/9)P(blue ball) = 4/9; I(blue ball) = -log2(4/9)Entropy = E[I(all balls)] = -[(2/9)*log2(2/9) + (3/9)*log2(3/9) + (4/9)*log2(4/9)]= 1.53 bits.
* Expected Value or Expectation of random variable X, written as E[X], is the average of all values of X weighted by the probability of their occurrence.
In other words, you can expect to get 1.53 bits of information on average every time I pick out a ball from the bin.
Formally, the entropy H of a probability distribution of a random variable is defined as:
The x~P in the above equation means that the values x takes are from the distribution P. In our example, P = (2 red, 3 green, 4 blue).
Cross-entropy measures the relative entropy between two probability distributions over the same set of events. Intuitively, to calculate cross-entropy between P and Q, you simply calculate entropy for Q using probability weights from P. Formally:
Let’s consider the same bin example with two bins.
Bin P = {2 red, 3 green, 4 blue}
Bin Q = {4 red, 4 green, 1 blue}
H(P, Q) = -[(2/9)*log2(4/9) + (3/9)*log2(4/9) + (4/9)*log2(1/9)]
Instead of the contrived example above, let’s take a machine learning example where we use cross-entropy as a loss function. Suppose we build a classifier that predicts samples in three classes: A, B, C.
Let P be the true label distribution and Q be the predicted label distribution. Suppose the true label of one particular sample is B and our classifier predicts probabilities for A, B, C as (0.15, 0.60, 0.25)
Cross-entropy H(P, Q) will be:
H(P, Q) = -[0 * log2(0.15) + 1 * log2(0.6) + 0 * log2(0.25)] = 0.736
On the other hand, if our classifier is more confident and predicts probabilities as (0.05, 0.90, 0.05), we would get cross-entropy as 0.15, which is lower than the above example.
For classification problems, using cross-entropy as a loss function is equivalent to maximising log likelihood. Consider the below case of binary classification where a, b, c, d represent probabilities:
H(True, Predicted)= -[a*log(b) + c*log(d)]= -[a*log(b) + (1-a)*log(1-b)]= -[y*log(ŷ) + (1-y)*log(1-ŷ)]..where y is true label and ŷ is predicted label
This is the same equation for maximum likelihood estimation.
Due to historical reasons, you will often find cross-entropy defined in the context of communication theory such as: cross-entropy is the average number of bits needed to encode data coming from a source with distribution p when we use model q. Most data scientists have never studied communication theory. I hope they find the above article useful.
If you liked the article, check out my other pieces on Medium, follow me on LinkedIn or Twitter, view my personal webpage, or email me at viraj@berkeley.edu.