How to Create a Correlation Matrix with Too Many Variables in R | by Catherine Williams | Towards Data Science

I used the Kaggle House Prices Dataset which has 79 explanatory variables. In my analysis, I tried to look at correlations between all of the variables and realized there are just too many variables to make sense of any typical visual aid.
I tried several different packages and tools and decided that I can manipulate corrplot to do what I want the best. Without any manipulations, this is what the correlation matrix looks like. This is obviously a very unhelpful visualization. There are definitely ways to change this view built into the package, but none of them can really handle this many variables.
library(corrplot)df_cor <- cor(df)corrplot(df_cor)
With a lot of trial and error and skimming Stack Overflow, I was able to create a function to make this process much easier. First, it will convert all variables to numeric values (if not already). Then, it will drop duplicates and perfect correlations (correlations with itself). These are obviously not useful.
corr_simple <- function(data=df,sig=0.5){  #convert data to numeric in order to run correlations  #convert to factor first to keep the integrity of the data - each value will become a number rather than turn into NA  df_cor <- data %>% mutate_if(is.character, as.factor)  df_cor <- df_cor %>% mutate_if(is.factor, as.numeric)  #run a correlation and drop the insignificant ones  corr <- cor(df_cor)  #prepare to drop duplicates and correlations of 1       corr[lower.tri(corr,diag=TRUE)] <- NA   #drop perfect correlations  corr[corr == 1] <- NA   #turn into a 3-column table  corr <- as.data.frame(as.table(corr))  #remove the NA values from above   corr <- na.omit(corr)   #select significant values    corr <- subset(corr, abs(Freq) > sig)   #sort by highest correlation  corr <- corr[order(-abs(corr$Freq)),]   #print table  print(corr)  #turn corr back into matrix in order to plot with corrplot  mtx_corr <- reshape2::acast(corr, Var1~Var2, value.var="Freq")    #plot correlations visually  corrplot(mtx_corr, is.corr=FALSE, tl.col="black", na.label=" ")}corr_simple()
Next, it will set up the data frame to see the raw correlations in a table since raw numbers can be helpful. The data frame is sorted by the highest correlation first.
In order to reduce the sheer quantity of variables (without having to manually pick and choose), Only variables above a specific significance level threshold are selected. It is set to 0.5 as the initial default.
After the table is produced, it will return the following, filtered out, correlation matrix chart. Only the correlations with a high enough significance level will have a colored circle. This further helps to cut out the noise if there are still a lot of remaining variables.
This ‘corr_simple’ function can be run again and again after some feature engineering, or with different significance levels. This really helps for faster analysis and to see only the relevant data. With more variables, it may be necessary to play with different significance levels and/or use more feature engineering to reduce the number of correlated variables, and then re-run the function until results are readable and useful.