Understanding How Schools Work with Canonical Correlation Analysis | by Liana Mehrabyan | Towards Data Science

Us data scientists spend most of our time analysing relationships and patterns in our data. Most of our exploratory tools, however, focus on one-to-one relationships. But what if we want to have a more generalised view and find commonalities and patterns between certain groups of variables?
This post includes:
an introduction to Canonical Correlation Analysis that lets us identify associations among groups of variables at a time.
a CCA tutorial in Python on how school environment affects students’ performance.
Suppose we want to find out how a school’s ambience affects its students’ academic success. On one hand, we have variables about the level of support, trust and collaboration in their learning environment. On the other hand, we have students’ academic records and test results.
CCA lets us explore associations between these two sets of variables as a whole, rather than considering them on an individual basis. Loosely speaking, we come up with a collective representation (a latent variable called canonical variate) for each of these variable sets in a way that the correlation between those variates is maximised.
Before diving into a pile of equations, let’s see why it’s worth the effort.
With CCA we can:
Find out whether two sets of variables are independent or, measure the magnitude of their relationship if there is one.
Interpret the nature of their relationship by assessing each variable’s contribution to the canonical variates (i.e. components) and find what dimensions are common between the two sets.
Summarise relationships into a lesser number of statistics.
Conduct a dimensionality reduction that takes into account the existence of certain variable groups.
Given two sets of variables:
We construct the first pair of Canonical Variates as linear combinations of the variables in each group:
where the weights (a1, ... ap), (b1, ... , bq) are chosen in a way that the correlation between the two variates is maximised.
Having our pair of covariates:
The Canonical Correlation Coefficient is the correlation between the canonical variates CVX and CVY.
To compute the second pair of covariates, we conduct the same process by adding one more constraint: each new variate should be orthogonal and uncorrelated to the previous ones.
We compute min(p,q) pairs in a similar fashion and end up with min(p,q) components ready to explore. (Note that the number of variables in each set doesn’t have to be the same.)
Just as in PCA, we project our data onto min(p,q) latent dimensions. However, not all of them might be informative and important. Let’s see the makeup and interpretation of our canonical variates in the example below.
We’ll be using two variable groups from NYC schools dataset:
Rigorous Instruction %
Collaborative Teachers %
Supportive Environment %
Effective School Leadership %
Family-Community Ties %
Trust %
Average ELA Proficiency
Average Math Proficiency
As for the tool, we’ll be using the pyrcca implementation.
We start with separating each of our variable groups in a single dataframe:
import pandas as pdimport numpy as npdf = pd.read_csv('2016 School Explorer.csv')# choose relevant featuresdf = df[['Rigorous Instruction %',      'Collaborative Teachers %',     'Supportive Environment %',       'Effective School Leadership %',   'Strong Family-Community Ties %',    'Trust %','Average ELA Proficiency',       'Average Math Proficiency']]# drop missing valuesdf = df.dropna()# separate X and Y groupsX = df[['Rigorous Instruction %',      'Collaborative Teachers %',     'Supportive Environment %',       'Effective School Leadership %',   'Strong Family-Community Ties %',    'Trust %'      ]]Y = df[['Average ELA Proficiency',       'Average Math Proficiency']]
Convert group X into numeric variables and standardise the data:
for col in X.columns:    X[col] = X[col].str.strip('%')    X[col] = X[col].astype('int')# Standardise the datafrom sklearn.preprocessing import StandardScalersc = StandardScaler(with_mean=True, with_std=True)X_sc = sc.fit_transform(X)Y_sc = sc.fit_transform(Y)
After relevant preprocessing, we’re ready to apply CCA. Note that we set the regularisation parameter to 0 as regularised CCA is out of the scope of this post (we will get back to this in future posts though).
import pyrccanComponents = 2 # min(p,q) componentscca = pyrcca.CCA(kernelcca = False, reg = 0., numCC = nComponents,)# train on datacca.train([X_sc, Y_sc])print('Canonical Correlation Per Component Pair:',cca.cancorrs)print('% Shared Variance:',cca.cancorrs**2)
Cannonical Correlations
>> Canonical Correlation Per Component Pair: [0.46059902 0.18447786]>> % Shared Variance: [0.21215146 0.03403208]
For our two pairs of canonical variates, we have canonical correlations of 0.46 and 0.18 respectively. So, latent representations of school ambience and students’ performance do have a positive correlation of 0.46 and share 21 percent of variance.
Squared canonical correlation represents the shared variance by the latent representations of variable sets, and not the variance inferred from the sets of variables themselves.
Canonical Weights
In order to access the weights assigned to our standardised variables (a1, ... , ap) and (b1, ... , bq) we use cca.ws:
cca.ws>> [array([[-0.00375779,  0.0078263 ],           [ 0.00061439, -0.00357358],           [-0.02054012, -0.0083491 ],           [-0.01252477,  0.02976148],           [ 0.00046503, -0.00905069],           [ 0.01415084, -0.01264106]]),     array([[ 0.00632283,  0.05721601],           [-0.02606459, -0.05132531]])]
Given these weights, the canonical variates of set Y, for example, were calculated with the following formula:
where the weights can be interpreted as the coefficients in a linear regression model. We should take into account that it is not highly recommended to rely on weights when interpreting individual variable contribution to covariates.
Here’s why:
Weights are subject to variability from one sample to another.
Weights can be highly affected by multicollinearity (which is quite common for same-context variable groups).
Relying on canonical loadings instead is a more common practice.
Canonical Loadings
Canonical loadings are nothing more than the correlation between the original variable and the canonical variate of that set. For example, to assess the contribution of Trust in school environment representation, we calculate the correlation between the variable Trust and the resulting variate for variable set X.
Calculating loadings for group Y in the first variate:
print('Loading for Math Score:',np.corrcoef(cca.comps[0][:,0],Y_sc[:,0])[0,1])print('Loading for ELA Score:',np.corrcoef(cca.comps[0][:,0],Y_sc[:,1])[0,1])>> Loading for Math Score: -0.4106778140971078>> Loading for ELA Score: -0.4578120954218724
Canonical Covariates
Finally, we might want to access the covariate values directly, be it for visualisation or any other purpose.
To do so, we need:
# CVX cca.comps[0]# First CV for X cca.comps[0][:,0]# Second CV for Xcca.comps[0][:,1]# CVYcca.comps[1]# First CV for Ycca.comps[1][:,0]# Second CV for Y cca.comps[1][:,1]
Hope you find this helpful and use CCA more in your EDA routine.
-Keep Exploring
[1] Bilenko N., Gallan., 2016, Pyrcca: Regularized Kernel Canonical Correlation Analysis in Python and Its Applications to Neuroimaging, Frontiers in Neuroinfomatics
[2] Dattalo, P.V., 2014, A Demonstration of Canonical Correlation Analysis with Orthogonal Rotation to Facilitate Interpretation