Correlation – Introduction
COMP20008
School of Computing and Information Systems
Chris Ewin & Geela Chee
1
Plan
• Discuss correlations between pairs of features in a dataset
• Why useful and important
• Pitfalls
• Methods for computing correlation
• Pearson correlation
• Mutual information (another method to compute correlation)
2
Correlations in life
As a child grows, so does his/her clothing size.
As you drink more coffee, the number of hours you stay awake increases.
The longer amount of time you spend in the bath, the more wrinkly your skin becomes.
The older a man gets, the less hair that he has.
https://towardsdatascience.com/3-beautiful-real-life-correlations-fed9e855da52,
https://examples.yourdictionary.com/positive-correlation-examples.html,
https://examples.yourdictionary.com/negative-correlation-examples.html
3
What is Correlation?
Correlation is used to detect pairs of variables that
might have some relationship.
AND
How strong is the relationship?
https://www.mathsisfun.com/data/correlation.html
4
What is Correlation?
Visually can be identified via inspecting scatter plots
https://www.mathsisfun.com/data/correlation.html
5
What is Correlation?
Linear relations
https://www.mathsisfun.com/data/correlation.html
6
What is Correlation?
Correlation strength
https://images.app.goo.gl/zZXtjBLR2BcjRpK79
7
Example of non-linear correlation
It gets so hot that people aren't going near the
shop, and sales start dropping
https://www.mathsisfun.com/data/correlation.html
8
Example of non-linear correlation
It gets so hot that people aren't going near the
shop, and sales start dropping
https://www.mathsisfun.com/data/correlation.html
9
Example of Correlated Variables
• Greater understanding of data
• Can hint at potential causal
relationships
• Business decision based on
correlation: increase electricity
production when temperature
increases
10
Example of Correlated Variables
Correlation does not
necessarily imply
causality!
11
Example rank correlation
“If a university has a higher-ranked football team, then is it
likely to have a higher-ranked basketball team?”
Football
ranking
University team
1 Melbourne
2 Monash
3 Sydney
4 New South Wales
5 Adelaide
6 Perth
Basketball
ranking
University team
1 Sydney
2 Melbourne
3 Monash
4 New South Wales
5 Perth
6 Adelaide
12
Biology example - Microarray data
Each chip contains thousands of tiny probes corresponding
to the genes (20k - 30k genes in humans). Each probe
measures the activity (expression) level of a gene
Gene 1
expression
Gene 2
expression
… Gene 20K
expression
0.3 1.2 … 3.1
13
Microarray dataset
Gene 1 Gene 2 Gene 3 … Gene n
Time 1 2.3 1.1 0.3 … 2.1
Time 2 3.2 0.2 1.2 … 1.1
Time 3 1.9 3.8 2.7 … 0.2
… … … … … …
Time m 2.8 3.1 2.5 … 3.4
• Each row represents measurements at some time
• Each column represents levels of a gene
14
Correlation analysis on Microarray data
Can reveal genes that exhibit similar patterns ⇨ similar or
related functions ⇨ Discover functions of unknown genes
15
Genetic network
Connect genes with high
correlation – understand
behaviour of groups of genes
http://www.john.ranola.org/?page_id=116
16
Salt Causes High Blood Pressure50
60
70
80
90
0 50 100 150 200 250
Median diastolic blood pressure (mmHg)
Median urinary sodium excre on (mmol/24hr)
Intersalt: an
international study of
electrolyte excretion and
blood pressure. Results
for 24 hour urinary
sodium and potassium
excretion.
British Medical Journal;
297: 319-328, 1988.
17
Or Does It!?50
60
70
80
90
0 50 100 150 200 250
Median diastolic blood pressure (mmHg)
Median urinary sodium excre on (mmol/24hr)
Intersalt: an international study of
electrolyte excretion and blood
pressure. Results for 24 hour
urinary sodium and potassium
excretion.
British Medical Journal;
297: 319-328, 1988.
If we exclude these
four ‘outliers’, which
are non-industrialised
countries with non-salt
diets, we get a quite
different result!
18
Spurious Correlation
Correlation ≠ Causality
https://images.app.goo.gl/FVr8BhxWmQMxCB5f7
https://assets.businessinsider.com/real-maps-ridiculous-
correlations-2014-11?jwsource=cl
19
Why is correlation important?
• Discover relationships
• One step towards discovering causality A causes B
Example: Smoking causes lung cancer
 Feature ranking: for building better predictive models
A good feature to use, is a feature that has high correlation with the
outcome we are trying to predict
20
Break
21
Pearson Correlation
COMP20008
School of Computing and Information Systems
22
Correlation
23
• Objects can be represented with different measure scales
d(temp,ice-cr)= 540324
d(temp,elect)= 12309388
• Euclidean distance: does not give a clear intuition about how well
variables are correlated
Problems of Euclidean distance
Day 1 Day 2 Day 3 … Day m
Temperature 20 22 16 … 33
#Ice-creams 50223 55223 45098 … 78008
#Electricity 102034 105332 88900 … 154008
24
Problems of Euclidean distance
Cannot discover
variables with similar
behaviours/dynamics
but at different scale
25
Problems of Euclidean distance
Cannot discover
variables with similar
behaviours/dynamics
but in the opposite
direction (negative
correlation)
26
Assessing linear correlation – Pearson
correlation
• We will define a correlation measure rxy, assessing samples from
two features x and y
• Assess how close their scatter plot is to a straight line (a linear
relationship)
Pearson Product Moment Correlation
• Range of rxy lies within [-1,1]:
• 1 for perfect positive linear correlation
• -1 for perfect negative linear correlation
• 0 means no correlation
• Absolute value |r| indicates strength of linear correlation
27
Pearson's correlation coefficient (r)
𝑟 = 𝑟𝑥𝑦 = σ𝑖=1
𝑛 (𝑥𝑖 − ҧ 𝑥)(𝑦𝑖 − ത𝑦)
(σ𝑖=1
𝑛 (𝑥𝑖 − ҧ 𝑥)2) × (σ𝑖=1
𝑛 (𝑦𝑖 − ത𝑦)2)
ҧ
𝑥 = 1
𝑛 ෍
𝑖=1
𝑛
𝑥𝑖 ത𝑦 = 1
𝑛 ෍
𝑖=1
𝑛
𝑦𝑖
𝑟 = 𝑟𝑥𝑦 = σ𝑖=1
𝑛 𝑥𝑖
∗ × 𝑦𝑖
∗
(σ𝑖=1
𝑛 𝑥𝑖
∗2) × (σ𝑖=1
𝑛 𝑦𝑖
∗2)
𝑥𝑖
∗ = 𝑥𝑖 − ҧ 𝑥 𝑦𝑖
∗
= 𝑦𝑖 − ത𝑦 28
Pearson coefficient example
Height (x) Weight (y)
1.6 50
1.7 66
1.8 77
1.9 94
• How do the values of x and y move (vary)
together?
• Big values of x with big values of y?
• Small values of x with small values of y?
29
Pearson coefficient example
30
Interpreting Pearson correlation values
In general it depends on your domain of application. Jacob Cohen
has suggested
• 0.5 is large
• 0.3-0.5 is moderate
• 0.1-0.3 is small
• less than 0.1 is trivial
31
Properties of Pearson's correlation
• Range within [-1,1]
• Is sensitive to outliers
• Can only detect linear relationships
y = a.x + b + noise
• Cannot detect non-linear relationships
y = x3 + noise
• Scale invariant: r(x,y)= r(x, Ky)
• Multiplying a feature’s values by a constant K makes no difference
• Location invariant: r(x,y)= r(x, K+y)
• Adding a constant K to one feature’s values makes no difference
32
Pearson correlation examples
https://en.wikipedia.org/wiki/Pearson_product-moment_correlation_coefficient
33
2017 Exam question 3a
34
2016 exam question 2a)
35
• Correlation <> Causality
• http://tylervigen.com/spurious-correlations
• Google trends correlation
• https://trends.google.com/trends/
36
Break
37
Mutual Information
COMP20008
School of Computing and Information Systems
This topic is at the more
difficult end of material
covered in this subject
38
Recap: Pearson correlation – assess
linear correlation between two features
https://www.mathsisfun.com/data/correlation.html
39
What about non-linear correlation?
Pearson correlation
is not suitable for
this scenario (value
less than 0.1)
https://www.mathsisfun.com/data/correlation.html
40
Entropy
Mutual Information
41
Entropy – intuition
• Entropy is a measure used to quantify the amount of uncertainty
in an outcome
• Randomly select an element from
A. {1,1,1,1,1,1,1,2} versus
B. {1,1,2,2,3,3,4,4}
• In which case are you more uncertain about the outcome of the
selection? Why?
• More uncertain, less predictable => high entropy (b.)
• Less uncertain, more predictable => low entropy (a.)
42
Another example
Consider the sample of all people in this subject. Each person is
labelled young (<30 years) or old (>=30 years)
• Randomly select a person and inspect whether they are young or
old.
• How surprised am I likely to be by the outcome?
• Suppose I repeat the experiment using a random sample of
people catching the train to the city in peak hour?
• How surprised am I likely to be by the outcome?
43
A recap on logarithms (to the base 2)
• y = log2x (y is the solution to the question “To what power do I
need to raise 2, in order to get x?’’)
• 2*2*2*2=16 which means log216 = 4 (16 is 2 to the power 4)
• log2 32 = 5
• log2 30 = 4.9
• log2 1.2 = 0.26
• log2 0.5 = -1
In what follows, we’ll write log instead of log2 to represent binary
log
44
Entropy
Given a feature 𝜲, assuming 𝜲 has
k number of categories (bins),
then 𝐻 𝜲 is its entropy.
𝐻 𝜲 = − ෍
𝑖=1
𝑘
𝑝𝑖 log2 𝑝𝑖
X : Flipping a special coin, head: 75%, tail: 25%
𝐻 𝑋 = − 𝑝𝐻 log2 𝑝𝐻 + 𝑝𝑇 log2 𝑝𝑇 = −[0.75 log2 0.75 + 0.25 log2 0.25] = 𝟎. 𝟖𝟏
𝑝𝑖: proportion of points in the 𝑖-th category (bin)
45
Entropy practice example
A B B A C C C C A
We have 3 categories (A,B,C) for a feature 𝜲
9 objects, each object is in one category.
What is the entropy of this sample data of 9 objects?
P_A=3/9 P_B=2/9 P_C=4/9
Answer: 𝐻 𝜲 = 1.53
𝐻 𝜲 = − ෍
𝑖=1
𝑘
𝑝𝑖 log2 𝑝𝑖
46
How would you compute the entropy for
the “Likes to sleep” feature?
Person Likes to sleep
1 Yes
2 No
3 Maybe
4 Never
5 Yes
6 Yes
7 Never
8 Yes
𝜲: Likes to sleep
𝐻 𝜲 = − ෍
𝑖=1
𝑘
𝑝𝑖 log2 𝑝𝑖
47
Properties of entropy
• H(X) ≥ 0
• Entropy is maximized for uniform distribution (highly
uncertain what value a randomly selected object will
have)
• Entropy – when using log base 2 – measures
uncertainty of the outcome in bits. This can be
viewed as the information associated with learning
the outcome
48
Variable discretisation
• Pre-processing: continuous
features are first discretised
into bins (categories). E.g.
small [0,1.4], medium
(1.4,1.8), large [1.8,3.0]
Object Height Discretised Height
1 2.03 large
2 1.85 large
3 1.23 small
4 1.31 small
5 1.72 medium
6 1.38 small
7 0.94 small
49
Variable discretisation: Techniques
• Equal-width bin
Divide the range of the continuous feature into equal length intervals
(bins). If speed ranges from 0-100, then the 10 bins are [0,10), [10,20),
[20,30), …[90,100]
• Equal-frequency bin
Divide range of continuous feature into equal frequency intervals (bins).
Sort the values and divide so that each bin has roughly same number of
objects
• Domain knowledge: assign thresholds manually
Car speed:
• 0-40: slow
• 40-60: medium
• >60: high
50
Discretisation example
• A 3-bin equal width discretization
• Bin-width: 21−2
3 = 6.333
• [2, 8.333), [8.333, 14.666), [14.666, 21]
Given the values 2, 2, 3, 10, 13, 15, 16, 17, 19, 19, 20, 20, 21
51
Discretisation example
• A 3-bin equal frequency discretization
• [2, 13), [13, 19), [19, 21] – by hand
• (1.999, 13], (13, 19], (19, 21] – pandas
Given the values 2, 2, 3, 10, 13, 15, 16, 17, 19, 19, 20, 20, 21
52
Break
53
Mutual Information
COMP20008
School of Computing and Information Systems
54
Conditional entropy - intuition
Suppose I randomly
sample a person. I
check if they wear
glasses – how
surprised am I by
their age given their
wearing glasses
status?
Person WearGlasses(X) Age (Y)
1 No young
2 No young
3 No young
4 No young
5 Yes old
6 Yes old
7 Yes old
55
Conditional entropy H(Y|X)
𝐻 𝑌 𝑋 = ෍
𝑥∈𝑋
𝑝 𝑥 𝐻(𝑌|𝑋 = 𝑥)
Measures how much
information is needed to
describe outcome Y, given
that outcome X is known.
Suppose X is Height and Y
is Weight.
Object Height (X) Weight (Y)
1 big light
2 big heavy
3 small light
4 small light
5 small light
6 small light
7 small heavy
56
Conditional entropy H(Y|X)
𝐻 𝑌 𝑋 = ෍
𝑥∈𝑋
𝑝 𝑥 𝐻(𝑌|𝑋 = 𝑥)
𝐻 𝑌 𝑋 = ൗ2 7 ∗ 𝐻 𝑌 𝑋 = 𝑏𝑖𝑔 + ൗ5 7 ∗ 𝐻 𝑌 𝑋 = 𝑠𝑚𝑎𝑙𝑙
𝐻 𝑌 𝑋 = 𝑥 = − ෍
𝑦∈𝑌
𝑝 𝑦 𝑥 log2 𝑝 𝑦 𝑥
𝐻 𝑌 𝑋 = 𝑏𝑖𝑔 = − ൗ1 2 log2 ൗ1 2 + ൗ1 2 log2 ൗ1 2 = 1
𝐻 𝑌 𝑋 = 𝑠𝑚𝑎𝑙𝑙 = − ൗ4 5 log2 ൗ4 5 + ൗ1 5 log2 ൗ1 5 = 0.721
𝐻 𝑌 𝑋 = ൗ2 7 ∗ 1 + ൗ5 7 ∗ 0.721 = 0.801
Object Height
(X)
Weight
(Y)
1 big light
2 big heavy
3 small light
4 small light
5 small light
6 small light
7 small heavy
57
𝑀𝐼 𝑋, 𝑌 = 𝐻 𝑌 − 𝐻 𝑌 𝑋
𝑀𝐼 𝑋, 𝑌 = 𝐻 𝑋 − 𝐻 𝑋 𝑌
𝐻 𝑌 𝑋 = ෍
𝑥∈𝑋
𝑝 𝑥 𝐻(𝑌|𝑋 = 𝑥)
where X and Y are features (columns) in a dataset
• MI (mutual information) is a measure of correlation
• The amount of information about Y we gain by knowing X, or
• The amount of information about X we gain by knowing Y
Mutual information definition
58
Mutual information example 1
𝑀𝐼 𝑋, 𝑌 = 𝐻 𝑌 − 𝐻 𝑌 𝑋
𝐻 𝑌 = − ෍
𝑦∈𝑌
𝑝(𝑦) log2 𝑝(𝑦)
𝐻 𝑌 𝑋 = ෍
𝑥∈𝑋
𝑝 𝑥 𝐻(𝑌|𝑋 = 𝑥)
𝐻 𝑌 = − ൗ5 7 log2 ൗ5 7 + ൗ2 7 log2 ൗ2 7 = 0.8631
𝐻 𝑌 𝑋 = ൗ6 7 ∗ 𝐻 𝑌 𝑋 = 𝑠𝑚𝑎𝑙𝑙 + ൗ1 7 ∗ 𝐻 𝑌 𝑋 = 𝑏𝑖𝑔
𝐻 𝑌 𝑋 = ൗ6 7 ∗ − ൗ5 6 log2 ൗ5 6 − ൗ1 6 log2 ൗ1 6 + ൗ1 7 ∗ −1 ∗ log21 = 0.5571
𝑀𝐼 𝑋, 𝑌 = 0.8631 − 0.5571 = 0.3059
Object Height
(X)
Weight
(Y)
1 small light
2 big heavy
3 small light
4 small light
5 small light
6 small light
7 small heavy
59
Mutual information example 2
𝑀𝐼 𝑋, 𝑌 = 𝐻 𝑌 − 𝐻 𝑌 𝑋
𝐻 𝑌 = − ෍
𝑦∈𝑌
𝑝(𝑦) log2 𝑝(𝑦)
𝐻 𝑌 𝑋 = ෍
𝑥∈𝑋
𝑝 𝑥 𝐻(𝑌|𝑋 = 𝑥)
𝐻 𝑌 = 1.379
𝐻 𝑌 𝑋 = 0.965
𝑀𝐼 𝑋, 𝑌 = 𝐻 𝑌 − 𝐻 𝑌 𝑋 = 0.414
Object Height (X) Weight (Y)
1 big light
2 big heavy
3 small light
4 small jumbo
5 medium light
6 medium light
7 small heavy
60
Properties of Mutual Information
• The amount of information shared between two variables X and Y
• MI(X,Y)
• large: X and Y are highly correlated (more dependent)
• small: X and Y have low correlation (more independent)
• 0 ≤ MI(X,Y) ≤ ∞
• Sometimes also referred to as ‘Information Gain’
61
Mutual information: normalisation
• MI(X,Y) is always at least zero, may be larger than 1
• In fact, one can show it is true that
• 0 ≤ MI(X,Y) ≤ min(H(X),H(Y))
• Thus if want a measure in the interval [0,1], we can define
normalised mutual information (NMI).
• NMI(X,Y) = MI(X,Y) / min(H(X),H(Y))
• NMI(X,Y) = MI(X,Y) / max(H(X),H(Y))
• NMI(X,Y) = MI(X,Y) / mean(H(X),H(Y))
• NMI(X,Y)
• large: X and Y are highly correlated (more dependent)
• small: X and Y have low correlation (more independent)
62
Normalised Mutual Information
Example 1:
𝑀𝐼 = 0.3059
𝑁𝑀𝐼 = 0.517
Example 2:
𝑀𝐼 = 0.414
𝑁𝑀𝐼 = 0.300
Object Height (X) Weight (Y)
1 small light
2 big heavy
3 small light
4 small light
5 small light
6 small light
7 small heavy
Object Height (X) Weight (Y)
1 big light
2 big heavy
3 small light
4 small jumbo
5 medium light
6 medium light
7 small heavy
63
Pearson correlation=-: -0.0864
Normalised mutual information (NMI)= 0.43
(3-bin equal spread bins)
64
Pearson correlation=-0.1
Normalised mutual information (NMI)=0.84
65
Pearson correlation=-0.05
Normalised mutual information (NMI)=0.35
66
Examples
• Pearson?
• NMI?
67
Examples
• Pearson: 0.08
• NMI: 0.009
68
Computing MI with class features
Identifying features that are highly correlated with a class feature
Compute MI(HoursSleep, Happy), MI(HoursExercise, Happy),
and MI(HoursStudy, Happy), MI(HairColour, Happy). Retain most predictive feature(s)
HoursSleep HoursExercise HairColour HoursStudy Happy
(class feature)
12 20 Brown low Yes
11 18 Black low Yes
10 10 Red medium Yes
10 9 Black medium Yes
10 10 Red high No
7 11 Red high No
6 15 Brown high No
2 13 Brown high No
69
Advantages and disadvantages of MI
• Advantages
• Can detect both linear and non-linear dependencies (unlike Pearson)
• Applicable and very effective for use with discrete features (unlike
Pearson correlation)
• Disadvantages
• If feature is continuous, it first must be discretised to compute mutual
information. This involves making choices about what bins to use.
• This may not be obvious. Different bin choices will lead to different
estimations of mutual information.
70
Choose the statement that is not true
• Mutual information can't detect the existence of linear relationships in
the data, but Pearson correlation can
• Mutual information can be used to assess how much one feature is
associated with another
• Mutual information can detect the existence of non-linear
relationships in the data
• Mutual information doesn't indicate the "direction" of a relationship,
just its strength
71
Question 2ai) from 2016 exam
a) Richard is a data wrangler. He does a survey and constructs a
dataset recording average time/day spent studying and average
grade for a population of 1000 students:
Student Name Average time per
day spent studying
Average Grade
… …. ….
i) Richard computes the Pearson correlation coefficient between Average time per day
studying and Average grade and obtains a value of 0.85. He concludes that more time
spent studying causes a student’s grade to increase. Explain the limitations with this
reasoning and suggest two alternative explanations for the 0.85 result.
72
Question 2aii) from 2016 exam
a) Richard is a data wrangler. He does a survey and constructs a
dataset recording average time/day spent studying and average
grade for a population of 1000 students:
Student Name Average time per
day spent studying
Average Grade
… …. ….
ii) Richard separately discretises the two features Average time per day spent studying
and Average grade, each into 2 bins. He then computes the normalised mutual
information between these two features and obtains a value of 0.1, which seems
surprisingly low to him. Suggest two reasons that might explain the mismatch between
the normalised mutual information value of 0.1 and the Pearson Correlation
coefficient of 0.85. Explain any assumptions made.
73
End of Correlation topic
74
Acknowledgements
• Materials are partially adopted from …
• Previous COMP2008 slides including material produced by James Bailey,
Pauline Lin, Chris Ewin, Uwe Aickelin and others
• See also
• Correlation and Causality – Don’t Confuse Them (13-2):
https://www.youtube.com/watch?v=dDlgAiTHNqI
75
Create 5 exam style questions from the above. The questions should  have multiple parts ie parts a,b, and c,d and e and be a mix of 80% applicaiton of relevant principles. Inlcude relevant images, formulas and other resources needed for the question aswell.
Write the questions using overleaf code. Do not include the answers