Using The Predictive Power Score in R | by Luca Zavarella | Towards Data Science

In recent months Florian Wetschoreck published a story on Toward Data Science’s Medium channel that attracted the attention of many data scientists on LinkedIn thanks to its very provocative title: “RIP correlation. Introducing the Predictive Power Score”. Let’s see what it is and how to use it in R.
The Predictive Power Score (PPS) is a normalized index (it ranges from 0 to 1) that tells us how much the variable x (be it numerical or categorical) could be used to predict the variable y (numerical or categorical). The higher the PPS index, the more the variable x is decisive in predicting the variable y.
The conceptual similarity of PPS with the correlation coefficient is evident, although the following differences exist:
PPS also detects non-linear relationships between x and y.
PPS is not a symmetrical index. This means that PPS(x, y) ≠ PPS(y, x). In other words, it is not said that if x predicts y, then y also predicts x.
PPS allows both numerical and categorical variables.
Basically, PPS is an asymmetric nonlinear index that is applicable to all types of variables for predictive purposes.
Behind the scene it implements Decision Trees as learning algorithms due to their robustness to outliers and poor data pre-processing.
The score is calculated on the test sets of a default of 4-fold cross-validation given by the scikit-learn cross_val_score function and it is given by different metrics according to the type of problem (regression or classification) defined by the target variable:
Regression: Mean Absolute Error (MAE) normalized to the [0, 1] interval given a baseline of “naïve” values of y calculated as the median of the target variable.
Classification: Weighted F1 normalized to the [0, 1] interval given a baseline of “naïve” values of y calculated as the most common value of the target variable or a random value (sometimes, a random value has a higher F1 than the most common value).
You can get into the details of the Python code thanks to the fact that the Predictive Power Score project was released as open source on Github.
The two indexes come from different domains and fundamental distinctions must be made:
Pearson correlation is given by the normalized covariance between two numerical variables. Covariance depends on the deviations of the two variables from their respective means, so it’s a statistical measure. Given two numerical variables, Pearson correlation is a descriptive index well defined mathematically that gives the goodness-of-fit for the best possible linear function describing the relation between the variables.
PPS tries to solve the issues of only linear correlations and only numeric variables being measured in a correlation analysis by applying decision tree estimation. It is derived from performance metrics of that estimation. At the time this post was written (version 1.1.0), it gets by default a random sample (stratified if needed) of 5000 rows from the input dataset in order to speed up the calculations (the size of the sample can be modified using a proper parameter). No tuning is done to get the optimal model parameters for the decision tree. There is therefore the possibility of overfitting or underfitting. Moreover, PPS results could be different at each run over the same dataset due to the randomness inherent to the algorithm (random seed could be used to guarantee the reproducibility of results). So, PPS could be inaccurate. But its aim is not to give an exact score, rather the general notion of dependency and a fast result.
Both the indexes should be used during the Exploration Data Analysis phase. As the same author says:
The PPS clearly has some advantages over correlation for finding predictive patterns in the data. However, once the patterns are found, the correlation is still a great way of communicating found linear relationships.
The author lists several use cases in his article where PPS may add value:
Finds every relationship that the correlation finds and more
Feature selection
Detect information leakage
Find entity structures in the data via interpreting the PPS matrix as a directed graph.
UPDATE: An R-specific package has been developed in late December 2020: https://github.com/paulvanderlaken/ppsr
As mentioned before, the project ppscore is open sourced and it’s developed in Python. There is currently no porting project of ppscore in R. So how to use the ppscore functions in an R script? It’s possible to use the prowess of R along with the programming capabilities of Python and vice-versa thanks to some libraries created for interoperability between Python and R. Among the others, two of them are the most widely used:
rpy2: an interface to R running embedded in a Python process
reticulate: a comprehensive set of tools for interoperability between Python and R thanks to embedded session of Python within R sessions
In our scenario, reticulate will allow ppscore functions to be called from an R script.
First of all, if you didn’t install Python on your machine, I suggest to install Miniconda (a small, bootstrap version of Anaconda that includes only conda, Python, and few other useful packages), letting the installer add the Python installation path to your PATH environment variable in case of a Windows machine.
Python developers usually use virtual environments. A virtual environment is a tool that helps to keep dependencies required by different projects separated by creating self-contained directory trees that contain Python installations for particular versions of Python, plus a number of additional packages. Conda is a tool that helps to manage environments and packages.
In this case a new Python environment will be created in order to install the ppscore required packages on it. You can use the following R script to prepare a new Python environment for the ppscore library:
# Install reticulateif( !require(reticulate) ) {  install.packages("reticulate")}# Load reticulatelibrary(reticulate)# List current python environmentsconda_list()# Create a new environemnt called 'test_ppscore'conda_create(envname = "test_ppscore")# Install the ppscore package and its dependencies using pip into the test_ppscore environment.# Requirements are listed here: https://github.com/8080labs/ppscore/blob/master/requirements.txtconda_install(envname = "test_ppscore", packages = "pandas", pip = TRUE)conda_install(envname = "test_ppscore", packages = "scikit-learn", pip = TRUE)conda_install(envname = "test_ppscore", packages = "ppscore", pip = TRUE)# Check if the new environment is now listedconda_list()# Make sure to use the new environmentuse_condaenv("test_ppscore")# Import the ppscore Python module in your R sessionpps <- import(module = "ppscore")
Functions and other data within Python modules and classes can be accessed via the $ operator (as well as interacting with an R list). Imported Python modules support code completion and inline help. For example, the matrix function in the pps module can be accessed as follows:
If you want to learn more about reticulate, you can go through its home page.
Starting from the example the author included in the repository, it’s possible to replicate the Titanic PPScore Heatmap using the matrix function:
The Variable Importance given the target “Survived” is obtainable filtering properly the output (complete interaction between predictors) given by the matrix function. But it is also possible to use directly the predictor function to directly have the same result, being careful to use the same random_seed parameter within both functions, in order to avoid to obtain different results.
The Variable Importance plot given the target variable “Survived” is the following:
The R code used to get the above plots is the following:
library(reticulate)library(readr)library(dplyr)library(ggplot2)heatmap <- function(df, x, y, value,                    main_title = "Heatmap", legend_title = "Value",                    x_title = "feature", y_title = "target") {    x_quo <- enquo(x)  y_quo <- enquo(y)  value_quo <- enquo(value)    res <- ggplot( df, aes(x = !!x_quo, y = !!y_quo, fill = !!value_quo) ) +    geom_tile(color = "white") +    scale_fill_gradient2(low = "white", high = "steelblue",                         limit = c(0,1), space = "Lab",                          name="PPScore") +    theme_minimal()+ # minimal theme    # theme(axis.text.x = element_text(angle = 45, vjust = 1,     #                                  size = 12, hjust = 1)) +    coord_fixed() +    geom_text(aes(x, y, label = round(!!value_quo, 2)), color = "black", size = 4) +    theme(      axis.text.x = element_text(angle = 45, vjust = 1,                                  size = 12, hjust = 1),      axis.text.y = element_text(size = 12),      panel.grid.major = element_blank(),      panel.border = element_blank(),      panel.background = element_blank(),      axis.ticks = element_blank()    ) +    xlab(x_title) +    ylab(y_title) +    labs(fill = legend_title) +    guides(fill = guide_colorbar(barwidth = 1, barheight = 10,                                 title.position = "top", title.hjust = 1)) +    ggtitle(main_title)    return(res)  }lollipop <- function(df, x, y,                     main_title = "Variable Importance",                     x_title = "PPScore", y_title = "Predictors",                     caption_title = "Data from Titanic dataset") {    x_quo <- enquo(x)  y_quo <- enquo(y)      res <- ggplot(df, aes(x=!!x_quo, y=forcats::fct_reorder(!!y_quo, !!x_quo, .desc=FALSE))) +    geom_segment( aes(x = 0,                      y=forcats::fct_reorder(!!y_quo, !!x_quo, .desc=FALSE),                      xend = !!x_quo,                      yend = forcats::fct_reorder(!!y_quo, !!x_quo, .desc=FALSE)),                  color = "gray50") +    geom_point( color = "darkorange" ) +    labs(x = x_title, y = y_title,         title = main_title,         #subtitle = "subtitle",         caption = caption_title) +    theme_minimal() +    geom_text(aes(label=round(!!x_quo, 2)), hjust=-.5, size = 3.5    ) +    theme(panel.border = element_blank(),           panel.grid.major = element_blank(),          panel.grid.minor = element_blank(),           axis.line = element_blank(),          axis.text.x = element_blank())    return(res)  }df <- read_csv("https://raw.githubusercontent.com/8080labs/ppscore/master/examples/titanic.csv")df <- df %>%   mutate( Survived = as.factor(Survived) ) %>%   mutate( across(where(is.character), as.factor) ) %>%   select(    Survived,    Class = Pclass,    Sex,    Age,    TicketID = Ticket,    TicketPrice = Fare,    Port = Embarked  )use_condaenv("test_ppscore")pps <- import(module = "ppscore")# PPScore heatmapscore <- pps$matrix(df = df, random_seed = 1234L)score %>% heatmap(x = x, y = y, value = ppscore,                  main_title = "PPScore for Titanic's predictors", legend_title = "PPScore")# Variable importancevi <- pps$predictors( df = df, y = "Survived", random_seed = 1234L)vi %>%  mutate( x = as.factor(x) ) %>%  lollipop( ppscore, x,            main_title = "Variable Importance for target = 'Survived'",            x_title = "PPScore", y_title = "Predictors",            caption_title = "Data from Titanic dataset")
Denis Boigelot published a quite famous picture about Pearson correlation examples on Wikimedia Commons, sharing also the R code used to get this picture:
Starting from that code, it’s possible to calculate the PPScore for the same distributions. Here the results for X→Y:
Here the results for Y→X:
It’s clear that at the time this post was written
PPScore struggles to identify linear relationships in presence of even minimal dispersion of the points from the line of origin (as highlighted in yellow and red in previous pictures).
These cases show
how important it is also to use a correlation index to check linearity in order not to run into wrong conclusions.
I shared these results with the author and he confirmed they are studying a solution to solve linear relationships issues. He also shared the reasons why PPScore 1.1.0 doesn’t work well with linear relationships at this link.
Conversely, PPScore provides good results on some distributions for which Pearson correlation was not sufficient (as highlighted in green in previous pictures).
The R code used to get the above Boigelot distributions is the following:
# Install packagespkgs <- c("dplyr","ggplot2","ggpubr","mvtnorm")for (pkg in pkgs) {  if (! (pkg %in% rownames(installed.packages()))) { install.packages(pkg) }}# Load packageslibrary(mvtnorm)library(dplyr)library(ggplot2)library(ggpubr)library(reticulate)# FunctionsMyPlot <- function(xy, xlim = c(-4, 4), ylim = c(-4, 4), eps = 1e-15,                    metric = c("cor", "ppsxy", "ppsyx")) {    metric <- metric[1]    df <- as.data.frame(xy)  names(df) <- c("x", "y")      if (metric == "cor") {        value <- round(cor(xy[,1], xy[,2]), 1)        if (sd(xy[,2]) < eps) {            #title <- bquote("corr = " * "undef") # corr. coeff. is undefined      title <- paste0("corr = NA") # corr. coeff. is undefined          } else {      #title <- bquote("corr = " * .(value))      title <- paste0("corr = ", value)    }        subtitle <- NULL      } else if (metric == "ppsxy") {        pps_df <- pps$matrix(df = df, random_seed = 1111L)        value <- pps_df %>%       filter( x == "x" & y == "y" ) %>%       mutate( ppscore = round(ppscore, 1) ) %>%       pull(ppscore)        title <- bquote("pps"[X%->%Y] * " = " * .(value))        subtitle <- NULL      } else if (metric == "ppsyx") {        pps_df <- pps$matrix(df = df, random_seed = 1111L)        value <- pps_df %>%       filter( x == "y" & y == "x" ) %>%       mutate( ppscore = round(ppscore, 1) ) %>%       pull(ppscore)        title <- bquote("pps"[Y%->%X] * " = " * .(value))        subtitle <- NULL      }    ggplot(df, aes(x, y)) +    geom_point( color = "darkblue", size = 0.2 ) +    xlim(xlim) +    ylim(ylim) +    labs(title = title,         subtitle = subtitle) +    theme_void() +    theme( plot.title = element_text(size = 10, hjust = .5) )  }MvNormal <- function(n = 1000, cor = 0.8, metric = c("cor", "ppsxy", "ppsyx")) {    metric <- metric[1]    res <- list()  j <- 0    for (i in cor) {    sd <- matrix(c(1, i, i, 1), ncol = 2)    x <- rmvnorm(n, c(0, 0), sd)    j <- j + 1    name <- paste0("p", j)    res[[name]] <- MyPlot(x, metric = metric)  }    return(res)}rotation <- function(t, X) return(X %*% matrix(c(cos(t), sin(t), -sin(t), cos(t)), ncol = 2))RotNormal <- function(n = 1000, t = pi/2, metric = c("cor", "ppsxy", "ppsyx")) {    metric <- metric[1]    sd <- matrix(c(1, 1, 1, 1), ncol = 2)  x <- rmvnorm(n, c(0, 0), sd)    res <- list()  j <- 0    for (i in t) {    j <- j + 1    name <- paste0("p", j)    res[[name]] <- MyPlot(rotation(i, x), metric = metric)  }    return(res)}Others <- function(n = 1000, metric = c("cor", "ppsxy", "ppsyx")) {    metric <- metric[1]    res <- list()    x <- runif(n, -1, 1)  y <- 4 * (x^2 - 1/2)^2 + runif(n, -1, 1)/3  res[["p1"]] <- MyPlot(cbind(x,y), xlim = c(-1, 1), ylim = c(-1/3, 1+1/3), metric = metric)    y <- runif(n, -1, 1)  xy <- rotation(-pi/8, cbind(x,y))  lim <- sqrt(2+sqrt(2)) / sqrt(2)  res[["p2"]] <- MyPlot(xy, xlim = c(-lim, lim), ylim = c(-lim, lim), metric = metric)    xy <- rotation(-pi/8, xy)  res[["p3"]] <- MyPlot(xy, xlim = c(-sqrt(2), sqrt(2)), ylim = c(-sqrt(2), sqrt(2)), metric = metric)    y <- 2*x^2 + runif(n, -1, 1)  res[["p4"]] <- MyPlot(cbind(x,y), xlim = c(-1, 1), ylim = c(-1, 3), metric = metric)    y <- (x^2 + runif(n, 0, 1/2)) * sample(seq(-1, 1, 2), n, replace = TRUE)  res[["p5"]] <- MyPlot(cbind(x,y), xlim = c(-1.5, 1.5), ylim = c(-1.5, 1.5), metric = metric)    y <- cos(x*pi) + rnorm(n, 0, 1/8)  x <- sin(x*pi) + rnorm(n, 0, 1/8)  res[["p6"]] <- MyPlot(cbind(x,y), xlim = c(-1.5, 1.5), ylim = c(-1.5, 1.5), metric = metric)    xy1 <- rmvnorm(n/4, c( 3,  3))  xy2 <- rmvnorm(n/4, c(-3,  3))  xy3 <- rmvnorm(n/4, c(-3, -3))  xy4 <- rmvnorm(n/4, c( 3, -3))  res[["p7"]] <- MyPlot(rbind(xy1, xy2, xy3, xy4), xlim = c(-3-4, 3+4), ylim = c(-3-4, 3+4), metric = metric)    return(res)}output <- function( metric = c("cor", "ppsxy", "ppsyx") ) {    metric <- metric[1]    plots1 <- MvNormal( n = 800, cor = c(1.0, 0.8, 0.4, 0.0, -0.4, -0.8, -1.0), metric = metric );  plots2 <- RotNormal(200, c(0, pi/12, pi/6, pi/4, pi/2-pi/6, pi/2-pi/12, pi/2), metric = metric);  plots3 <- Others(800, metric = metric)    ggarrange(    plots1$p1, plots1$p2, plots1$p3, plots1$p4, plots1$p5, plots1$p6, plots1$p7,    plots2$p1, plots2$p2, plots2$p3, plots2$p4, plots2$p5, plots2$p6, plots2$p7,    plots3$p1, plots3$p2, plots3$p3, plots3$p4, plots3$p5, plots3$p6, plots3$p7,        ncol = 7, nrow = 3  )  }#-- Main -------------------------------------use_condaenv("test_ppscore")pps <- import(module = "ppscore")output( metric = "cor" )output( metric = "ppsxy" )output( metric = "ppsyx" )
The Predictive Power Score (PPS) index proposed by Florian Wetschoreck tries to help data scientists in giving a hint to find any kind of relationship between two variables during the EDA phase of a project, being them numerical or categorical. PPS could be inaccurate, but its aim is not to give an exact score, rather the general notion of dependency between two variables and a fast result.
Using the Boigelot distributions, it was shown that PPS (the current version is 1.1.0) is able to identify non-linear relationships not identified by the correlation index. Conversely, PPS seems to struggle to identify linear relationships in presence of even minimal dispersion of the points from the line of origin.
Although PPScore was developed in Python, its functions can also be used in an R script thanks to reticulate.
All the code can be found on this Github repository.