An introduction to Information Theory for Data Science | by Gabriel de Longeaux | Towards Data Science

This article aims to present briefly the concept of quantity of information, within the framework of information theory, and to outline possible uses of this idea in the field of data analysis. From these simple elements, I was led to considerations on a measure of multivariate nonlinear correlations presented in this article.
Information theory is largely based on the works of Claude Shannon published in the late 1940s (see the article A Mathematical Theory of Communication, published in the Bell System Technical Journal in 1948, as well as The Mathematical Theory of Communication published the year after).
Its primary objective is to provide tools to improve signals transmission in telecommunication systems. This implies a peculiar definition of what information means, which nevertheless proves to be plentiful for fields, such as data analysis, which at first sight have little to do with signal encoding techniques in telecommunication systems: the information contained in a message is defined as the minimum amount of bits needed to write that message.
To gain a clearer understanding of this definition, let’s consider that I want to write the word “analyse” using a specific encoding in order to translate each letter into a sequence of 0s and 1s.
I can choose any encoding, such as this one:
                         ╔════════╦══════════╗                         ║ Letter ║ Encoding ║                         ╠════════╬══════════╣                         ║   a    ║    111   ║                         ║   n    ║    110   ║                         ║   l    ║    101   ║                         ║   y    ║    100   ║                         ║   s    ║     01   ║                         ║   e    ║     00   ║                         ╚════════╩══════════╝
Consequently, the word “analyse” will be written: 111–110–111–101–100–01–00 (I write the dashes only to simplify reading, they are not present when a computer writes or decrypts the message). Therefore, the word was encoded using 19 bits: the amount of information contained in the word “analysis” is therefore at most equal to 19.
However, the frequency of occurrence of the letter “a” is higher than that of the other letters, which means that if this letter was for instance encoded by 01 while the letter “s” was encoded by 111, then the encoding of the word would only require 18 bits. This allows us to affirm that the quantity of information contained in the word “analyse” is at most equal to 18.
To define a parsimonious encoding, which signifies encoding a text with a minimum number of bits, it is possible to use a Huffman code, based on this very idea that it is better to encode the frequent signs with short binary numbers, and only use the longest binary numbers for the least frequent signs.
Huffman codes can be defined using Huffman trees (which I do not present in detail here). For the word “analyse” a possible tree could be:
This tree provides the following code:
                         ╔════════╦══════════╗                         ║ Letter ║ Encoding ║                         ╠════════╬══════════╣                         ║   a    ║     00   ║                         ║   n    ║     01   ║                         ║   l    ║    100   ║                         ║   y    ║    101   ║                         ║   s    ║    110   ║                         ║   e    ║    111   ║                         ╚════════╩══════════╝
And we finally get, for the word “analysis”, this binary encoding: 00–01–00–100–101–110–111.
The word “analysis” is encoded by 18 bits again. Is this number of bits the minimal number required to encode the word “analyse”? In general, given any message, can someone easily determine the amount of information contained in the message? The answer to these questions was given by Claude Shannon.
If we note P(1) the frequency of appearance of the letter “a” in the word “analyse”, P(2) the frequency of appearance of the letter “n”, etc., then Claude Shannon teaches us that the minimum number of bits needed on average to encode each letter of the word “analyse” is:
He names this quantity “entropy”. For the word “analyse” we obtain the quantity 2.5.
Since this represents the average number of bits absolutely necessary for encoding each letter of the word “analyse”, this quantity must be multiplied by the number of letters in the word, which ultimately gives 17.5.
Therefore, the minimum number of bits needed to encode the word is 18 bits, since only a whole number of bits can be used, and the number 17.5 is interpreted as the amount of information contained in the word “analyse”.
From here on in, I will focus on the entropy, that is to say the average quantity of bits necessary for the encoding of each letter. Indeed, this gives the intrinsic “variability” of the message (or of the data), regardless of its length (or of the amount of data).
The concept of “variability” is essential for studying correlation between variables, since studying correlation is about studying how the variables vary in relation to each other. For instance, if in meteorology one notices that in one part of the globe precipitation increases when temperatures rise, then these two variables are correlated, regardless of the number of days when temperatures and precipitation levels were recorded.
As a result, entropy characterizes the “variability” of a set: the more the set of elements is diverse, the more the entropy of this set becomes significant. Yet, in the field of statistics, the variability of a set can be represented by the variance between the elements of the set. Why should we not be satisfied with the variance, and use entropy instead?
Variance, although well-known and very useful, requires making a number of assumptions about the data studied.
Firstly, the variance can only be calculated for numeric data.
On the other hand, the determination of the variance is based on the estimate of the mean:
Var(X) = E[(X - E[X])2]
This does not really pose a problem when the data are numerical and available in sufficient numbers: the usual mean and variance estimators give very accurate results.
However, the ultimate goal is to determine not the variance of different variables, but the correlation between several variables. We must hence calculate the variance of a variable, assuming the other is known:
Var(Y|X) = E[(Y|X - E[Y|X])2]
And in that case, any estimation of the variance is possible, if I do not suppose a relation between variables X and Y. Indeed, if I refuse to make any assumption, it is enough for me to consider a polynomial model of sufficient degree for overfitting from the available data and thus assert that the variance of Y when the realizations of X are known is zero.
Therefore, to calculate Pearson correlation between two variables, we assume that the relationship between X and Y is linear, except for measurement errors; and among all the possible variances we choose the minimal variance. Pearson’s correlation is for that reason a linear correlation, and is intrinsically linked to linear regression by the ordinary least squares method, which consists in minimizing:
Var(Y|X) = E[(Y|X - E[Y|X])2]
assuming that:
E[Y|X = x] = αX+β
Unlike the variance, the entropy cannot be calculated from numeric data but requires categorical data.
The greater the number of categories in a set, and the more these categories are as likely as any other to be found, the greater the entropy will be. The entropy, therefore, measures the variability, the “intrinsic disorder” of the data in the set, without making any assumptions about how the data should be organized.
Entropy hence makes it possible to overcome assumptions about relationships between data, in particular the linearity assumption, but then at the cost of discretizing the information. Seeking to define a correlation that is not based on any hypothesis while considering numerical data is in any circumstance doomed to failure. Indeed, if I refuse to make any hypothesis whatsoever on the relationship between the variables, it means that I am not limiting the complexity of the underlying model. Consequently, I will always be able to find a model that will perfectly interpolate the data, which would allow me to say that the correlation between two variables is always perfect. This would of course be a case of overfitting: one must therefore limit the complexity of the underlying model either by expressing an assumption (such as the linearity hypothesis) or by discretizing the data to prevent overfitting.
Lastly, it should be noted that any diversity index can be a substitute to entropy, since they have the same properties crucial for the definition of a nonlinear correlation.
The considerations of the first part, especially the comparison between variance and entropy, deserve to be explained in more details. I henceforth consider that we are studying the relationship between two variables X and Y. How do we measure mutual information between these two variables, or to put it less meticulously, to what extent can one predict the other?
To avoid any confusion between Pearson’s linear correlation and nonlinear correlation, I will talk about correlation in the linear case, and mutual information between two variables in the nonlinear case.
Nevertheless, before we study the theoretical aspects of mutual information, let’s take a look at how entropy can help us compare different variables against each other.
There is a link between Shannon’s measure of information and set theory, which in a very practical way allows us to reason using set theory and make use of Venn diagrams to visually represent formulas.
I do not present here the theoretical justification of the association between information theory and set theory. The interested reader can refer to chapter 6 of A First Course in Information Theory by Raymond T. Weung.
From examining the link between information theory and set theory, we can come to the conclusion that it is possible to represent information theory formulas visually by Venn diagrams. In such a diagram, each disk then represents the “variability” of a variable, “variability” of which Shannon’s entropy is the measure. I, therefore, equate the area of disk X to the associated entropy H(X).
The multivariate entropy H(X,Y) is defined as being the entropy of the variable Z=(X,Y). In graphical terms, H(X,Y) is the area of the union of the X and Y disks.
It therefore appears graphically that:
The “variability” that two variables have in common is the overlapping surface of the disks of the two variables: the area of this intersection is the mutual information between the two variables. In other words, the mutual information I(X,Y) between X and Y is represented by the area of the intersection between the disks X and Y, and corresponds to the entropy “shared” between X and Y.
When the variables X and Y are perfectly « correlated », then the two disks X and Y are entirely overlapping, and so H(X,Y)=H(X)=H(Y). On the contrary, when X and Y have nothing in common (case of no « correlation » between the variables), then the disks X and Y are disjoint and H(X,Y)=H(X)+H(Y).
Now it is possible to introduce Claude Shannon’s definition of the mutual information in a simplified way, keeping in mind practical applications of data analysis.
In the case of discrete variables X and Y, the measure of normalized mutual information between the two variables can be defined by:
This standardized measure of mutual information assumes values between 0 and 1. Indeed, it is possible to deduce from the previous formulas that:
When the variables X and Y are perfectly “correlated”, then H(X,Y)=H(X)=H(Y) , so the standardized mutual information is worth 1. On the contrary, when X and Y have nothing of common, then H(X,Y)=H(X)+H(Y) and the mutual information is consequently also zero.
In the case of continuous X and Y variables, we must:
either discretize the variables to be able to apply the previous section formula (or a similar one)
or use a form of differential entropy (for instance Kullback-Leibler divergence — a measurement of asymmetric mutual information), which assumes that we know the probability distributions of the random variables X and Y.
When the probability distributions of the variables X and Y are known, the second method can give an exact measure of the mutual information between X and Y . For this reason the Kullback-Leibler divergence can be seen as a way to define an exact measurement of mutual information without needing to discretize the variables. Indeed, it is a question of comparing the distributions, possibly continuous, associated to the variables.
However, in practice, distributions of variables are often not identified at first glance. Estimating them forces us either to make assumptions about the shapes of the distributions, or to discretize the variables by creating histograms which, with the number of observations, end up matching the shape of the unknown distribution. Consequently, this last method is equivalent to the act of discretizing the data, since estimating the distributions from histograms amounts to adopting a discretized view of the data.
Broadly speaking, it appears that to calculate any type of “correlation”, it is crucial to either make assumptions about the relationship between the variables studied, or to discretize the data.
Normalized mutual information is very similar to Pearson correlation. A notable difference is that the Pearson correlation is between -1 and 1 while the normalized mutual information is between 0 and 1. This is because in the case of a linear relationship between two variables, they can be correlated in two ways, and only two: positively or negatively. However, in the case of any relationship between variables, there exists an infinite number of ways they can be correlated with each other. Therefore, the normalized mutual information should primarily be compared to the absolute value of the Pearson correlation.
Similarly to the standardized mutual information, the absolute value of the Pearson always stands between 0 and 1, and when X and Y are independent the correlation between them is zero.
In contrast, if Y=αX+β (perfect linear correlation), then the Pearson correlation is equal to sgn(α). Therefore, the correlation equals either 1 or -1 depending on whether α is positive or negative.
I have stated that in order to measure a nonlinear “correlation” between several variables, the data scientist is faced with a dilemma: either to make assumptions about how the variables relate or to discretize the data.
In reality, these two possibilities are equivalent. Indeed, for anyone interested in learning theories (Vapnik-Chervonenkis theory or PAC-Bayesian theory — “PAC” meaning “Probably Approximately Correct”), it is clear that four aspects are linked:
the amount of data available to learn
the distribution of data in the studied space (are the data grouped in together, or quite the opposite very dispersed? In other words, what is the data variance?)
the complexity of the model used to express the relationship between the variables (is the model a linear model or a polynomial of degree 10?)
the generalization error of the model.
The less data we possess, the more we are forced to limit ourselves to simple models in order to be able to reasonably hope that the model gives satisfactory results in the general case: this is why it is useless to use a linear model when only two points are available.
This is more particularly true when considering a large number of variables, which is known as the “dimensionality curse”.
By discretizing the data using a very large number of classes, we allow the computer to observe very sophisticated relationships between variables. This amounts to considering very complex models: the measurement of mutual information can hence be excellent, but the generalization error will in all likelihood be very large. On the contrary, by using only a few classes, the computer can only consider basic relationships between them: consequently, the mutual information score will degrade, on the other hand the generalization error will generally be lower.
The way in which the discretization is performed is therefore directly related to the number of observations available and the number of dimensions of the problem. In general cases, the data scientist can do it several times: he can for instance set up a decision tree on training data, then check on verification data that the generalization error seems reasonable. If the results obtained on the verification samples are unsatisfactory, the data scientist can in this case consider a new model, and repeat his work until he obtains appropriate results (while being careful not to overfit the verification samples themselves).
Unfortunately, this procedure is uninteresting in the present case, where we are looking for a measure of “correlation” between several variables. Let us imagine a data scientist who, after having found a high “correlation” between two variables, decides to establish the linear relation which binds them. He then notices that even on the training sample this linear relation is non-existent (e.g. the linear regression on the learning sample presenting an R2 of 5%) : the “correlation” initially used would be quite useless (the same would apply for any attempt to reduce the dimensionality of the problem, the principal component analysis for example being based on the measurement of the correlations between the variables).
What has no chance of happening in the case of linear correlation may very well happen in the case of mutual information measurement if we are not cautious. In particular, being capable of defining “good discretization” remains essential. Yet this is a stumbling block. Literature on this subject is immense, the results can sometimes be very complex, and to my knowledge there is no perfect solution (at least for the moment).
I hence present here only a few rules of thumb for the practical implementation of mutual information measurement. Suppose we are trying to measure the information provided by a set of variables called descriptors to describe other variables called target variables.
The first rule is that the target variables must be discretized as much as is required and sufficient for the practical case. If, for instance, the target variable consists of temperature data in degrees Celsius, and if for the use case it is both necessary and sufficient to distinguish between “low”, “medium” and “high” temperatures, it is then both necessary and sufficient to discretize the target-variable into three corresponding classes. The greater the number of classes of the target variables (it works in the reverse way for the descriptors), the more the overfitting (that is, unduly obtaining a high mutual information score) is reduced; nonetheless, asking for more than necessary creates a risk of obtaining an artificially low mutual information score for the interest issue (unless you have a very large amount of data).
The second rule is that the descriptors must be discretized according to the different quantiles, to have classes as balanced as possible. To determine which quantiles to take into account, I postulate that the number of classes for each descriptor corresponds to an inequality of this formula:
In this equation, m is the number of data items available. To determine c, let’s assume that we must answer the following question: if I want to establish a possible linear relation between two variables X and Y, from what minimum quantity of data for the training sample can I allow myself to perform a linear regression (hoping that the generalization error will be “quite possibly close enough” to the learning error)? This quantity is equal to c.
Suppose the answer to the previous question is c = 7. Assume that we are trying to predict Z as a function of X and Y (the number of descriptors is therefore 2), and suppose that we have 500 data items. The maximum number of classes for each of the variables X and Y will thus be:
(1/7) ∙ √500 ≃ 3,20.
We can as a result discretize X using the median (to divide the data into two classes) or the tertiles (creation of three classes). It is also the same for Y.
In the event that some of the variables (descriptors or target variables) are already discrete, some classes must eventually be merged in order to meet the preceding requirements.
The minimum number of data that is essential to correctly quantify a linear relationship between two variables X and Y is not obvious, and I shall detail one possible approach in a future article.
In a nutshell, information theory is a particularly broad theory which goes beyond the scope of telecommunications application, and can be used conveniently in data analysis. It is typically used to build certain decision trees, but to my knowledge, only univariate effects are considered. However, from the basic considerations presented here, it is possible to formulate a multivariate nonlinear correlation definition, the subject of my previous text. Nevertheless, complications emerge when moving from a linear correlation to a non-linear correlation, as the complexity of the functions considered as potential models is no longer limited, the topic then opening onto learning theories.