Covariance, Correlation, R-Squared
By: Deepak Khandelwal
Tags: R Squared, Mathematics, Covariance, Correlation, Statistics

This article is dedicated to the explanation of the three closely related but still different concepts namely — covariance, correlation and R-Squared. This also describes about the interpretation of the three.

What is variance?

Random variables, by definition, can take different values. The range of values a random variable takes and the variation among them is determined by the distribution of that random variable. Next, the expected value (aka mean, average, expectation) of a random variable tells us what value we can expect from a random variable on an average. For example, throwing a six sided die is a random variable (D) which can take the following values — [1, 2, 3, 4, 5, 6]. The probability of D taking any of the six values is 1/6 for all the values. Hence D follows a uniform distribution. The expected value of D, denoted as E[D], is 3.5.

But what we are interested in is what maximum and minimum values it can take and what is the variation from the expected value. This is defined by the variance. The variance gives us a measure of how much the random variable X deviates from its expected value E(X). It measures the variability among data, i.e., how much the quantity, the random variable represents, fluctuates.

Furthermore, it is entirely possible that two random variables X and Y, having different distributions, have same mean but different variances. For example

X takes the values — [3, 3, 3, 3] , Y takes the values — [1, 3, 3, 5]

E(X) = 3, E(Y) = 3

VAR(X) = 0, VAR(Y) = 2

We can see that both X and Y have same mean, but different variations — 0 and 2 respectively.

The importance of variance can be understood by the following real life example — suppose an investment is made in real estate sector. Although, the expected amount for the return might be a profit, but we want to actually know how much it can vary, how much low and high it can go so that we can calculate the risk and plan our investment accordingly.

Mathematically, the variance of a random variable X is defined as —

Figure 1: Variance

What is covariance?

Now we extend and generalize the idea of variance. We have seen the variance of one random variable as shown in figure 1. We can rewrite it as —

Now consider instead of one random variable, we have two random variables X and Y, we can exploit the above formula and we can write —

Figure 2: Covariance

This explains how much X varies from its mean when Y varies from its own mean. It is a statistical measure used to analyze how two random variables behave as a pair. To illustrate the concept with examples, I have taken three datasets to explain three different trends represented by covariance.

Positive Trend —

The Figure 3 shows the ice-cream dataset which has two random variables — Temperature and Revenue. It has the information of how income of ice cream sale varies as temperature varies. Using this dataset, we try to find the covariance between Temperature and Revenue.

Figure 3: Ice-cream dataset

These observations have been taken from the same ice-cream shop and have been taken in pairs i.e. we have pairs of (Temperature, Revenue). Now we are interested to know whether observations taken in pair (Temperature, Revenue) possess some information which individual observation (Temperature or Revenue) does not.

We have the following distributions of Temperature and Revenue.

Figure 4: Distribution of Random variables — Temperature and Revenue

Table 1: Ice-cream dataset — Covariance

Generally speaking, the pairs with relatively low values of Temperature also have relatively low values for Revenue and the pairs with relatively high values of Temperature also have relatively high values for Revenue. The blue line in Figure 3 represents this relationship. Note that this line has positive slope and it represents positive trend where values for Revenue increases with the values for Temperature.

Now we will measure the covariance between the two. We can observe in figure 3 that if temperature increase, revenue also increase.

Negative Trend —

Now I have taken fuel economy dataset which has two random variables — HorsePower and FuelEconomy as shown in Figure 4.. Here, we can see that blue line has fitted the data and has negative slope. This means that for relatively higher values for HorsePower, we have relatively lower values for FuelEconomy and vice-versa. This represents negative trend in the data.

Figure 5: Fuel Economy Dataset

We have the following distributions of HorsePower and FuelEconomy.

Figure 6: Distribution of Random variables — HorsePower and FuelEconomy

Table 2: Covariance with Negative Trend

No Trend —

If we have data that has following distribution —