Scatter Plots with Plotly Express | by Darío Weitz | Towards Data Science

You could say that it was a dream team of data science: a physics Ph.D. from Harvard, a Silicon Valley analyst in energy, a member of Facebook, and a guy who worked in a research startup, also in the Valley. Together, Alex Johnson, Jack Parmer, Matthew Sundquist, and Chris Parmer, founded Plotly, a technical computing company in Montreal, Canada.
The Plotly visualization tool was built around 2013 using Python and the Django framework, with a front end using JavaScript and the visualization library D3.js, HTML, and CSS. So, at its core, Plotly is actually a JavaScript library. Remember that D3.js is a JavaScript library for producing dynamic, interactive data visualizations in web browsers. But Python developers hardly ever need to engage with the JavaScript library directly. They can use plotly.py, an interactive, open-source, and browser-based graphing library for Python.
In 2019 the company made a quantitative leap with the launch of Plotly Express, a new high-level Python visualization library. Plotly Express is a high-level wrapper for Plotly.py fully compatible with the rest of the Plotly ecosystem, simple, powerful, and somewhat similar to Seaborn. It’s free and can be used in commercial applications and products. The library includes functions to plot trendlines and maps, as well as to perform faceting and animations. With Plotly Express you can make interactive graphics online but you can also save them offline.
The last version of plotly may be installed using pip:
$ pip install plotly==4.14.3
or conda:
$ conda install -c plotly plotly=4.14.3
Scatter plots are used to determine if a pair of numerical variables are correlated. They are suitable for distribution analysis of two numerical variables. They should not be used to show trends over time. They are also not suitable for comparison analysis. Besides, they are not appropriate when the nature of the message is to show quantities.
Starting from a dataset representing two numerical variables that are indicated by points in a Cartesian plane x-y, the message is narrated from the shape that these data points generate, through revealing the existence or not of a correlation. Such correlation can be positive or negative and is obtained by representing a significant number of data points. Even though each point indicates exact numerical values, the purpose of the visualization is to determine whether or not a relationship or correlation exists between the numerical variables represented, rather than focusing on the exact values indicated.
A correlation is defined as a measure (a metric) to evaluate the relationship between two variables. You can calculate (using an equation) the correlation coefficient (r) that takes values ​​between 1 to -1: a value close to 0 indicates no correlation; a value close to 1 indicates a very strong direct relationship between the two variables; a value close to -1 indicates a very strong inverse relationship between them; values ​​greater than 0.7 or -0.7 indicate, respectively, strong direct or inverse relationships; values ​​below 0.3 or -0.3 indicate weak or null direct or inverse relationships.
Three important features of the dataset can be found in a scatter plot: 1.- Outliers, a piece of data that is very different from all the others in the dataset and does not seem to fit the same pattern. Those anomalous values might ​​represent valuable information to analyze. First of all, it must be verified that the existence of these anomalous values ​​is not due to errors while measuring the data; 2.- Gaps, an interval that contains no data. The visualization of gaps between data justifies an in-depth analysis that explains their presence; 3.- Clusters, isolated groups of data points which can also merit a particular analysis of the reason for their presence in the graph. Of course, gaps and clusters might also represent errors in the data collection methodology.
A regression line is habitually added to the scatter plot. Also named as Line of Best Fit or Trendline, it mathematically expresses the relationship between both numerical variables. A trendline is meant to mimic the trend of the data. It’s a line of best fit that approximates the direction in which the plotted points appear to be heading. Usually, the goal of the regression lines is to estimate some unmeasured values of the independent variable utilizing the interpolation technique or use it for forecasting purposes via extrapolation. Special care should be taken not to confuse correlation with causation.
First, we import Plotly Express usually as px:
import plotly.express as px
In our first example, we are going to determine if there is a correlation between two datasets generated with the Numpy function random.randint:
import plotly.express as pximport numpy as np## set the seed to a random number obtained from /dev/urandom## or its Windows analog, ## or if neither of those is available, it will use the clock.np.random.seed()rand_x = np.random.randint(1,100,50)rand_y = np.random.randint(1,100,50)## trendline = 'ols' allow us to draw a trendlinefig = px.scatter(x = rand_x, y = rand_y, trendline = 'ols')fig.write_image(path + "figscat.png")fig.show()
Clearly, there is no correlation. The trendline is based on the Ordinary Least Square method (OLS). Previously, you must install statsmodels and its dependencies to use the argument trendline.
It's a nice chart, but not enough for today's standards. We can improve it significantly using functions from the Plotly Express library.
We suggest using a procedure similar to the following:
First, select an effective visual encoding to map your data values to a graphical feature and determine the corresponding parameters. For the scatter plots in this article, the Plotly Express function is px.scatter and the corresponding parameters are: dataframe; a numerical variable for the x-axis; a numerical variable for the y-axis; color to include other variables, especially categorical ones; the trendline method for the regression line; and the hover data and hover name. The hover_name property controls which column is displayed in bold as the tooltip title, whilst the hover_data argument accepts a list of column names to be added to the hover tooltip.
Second, analyze if your storytelling needs faceting. The faceting approach splits a chart into a matrix of panels, each one showing a different subset of the dataset based on a categorical variable. You can do it by column facet_col = [‘col_name’], or by row.
Third, update the chart with update.layout: set the title; the name of the x-axis and the name of the y-axis; set the figure dimensions with width and height.
Forth, add annotations with add_annotation. Remember that annotations relate to text placed in arbitrary positions in the chart. They usually consist of text, arrows, or other shapes.
Fifth, analyze if you need animation frames. Animated plots are a great way to display the dynamics of the underlying data. Dynamic systems are systems that evolve over time. We are not going to include animations because our dataframe doesn’t include a time dimension.
Finally, export the file: you can do that as a static format like PDF, SVG, PNG, or similar using fig.write_image(path + “fig_scat.png”) or save it as an HTML page for zooming or seeing the tooltips fig.write_html(path + ‘fig_scat.html’).
We worked with a dataset downloaded from Kaggle [2]. The dataset consists of 10,000 bank customers mentioning their age, salary, marital status, credit card limit, credit card category, education level, and additional features. The bank manager is disturbed with customers leaving their credit card services (attrited customers). So, we are going to determine if there is any relationship between some numerical variables and the attrition condition.
To avoid congestion and overlapping, we selected randomly 1500 of those customers as a sample of the population under study by means of df = df.sample(frac = 0.15).
First, we analyzed if there was any relationship between customer age and the limit in credit imposed by the bank. So, Customer_Age was chosen as the numerical variable for the x-axis and Credit_Limit as the numerical variable for the y-axis. We included color = ‘Gender’ to verify if there was a gender bias in the bank management directives. We saved the chart in a PNG static format. If you want to use the following code, don’t forget to import pandas as pd.
df = pd.read_csv(path + 'CreditCardCustomersCols.csv',         index_col =  False, header = 0,sep = ';', engine='python')## get a random selection of 1500 rows, a fifteen-percent of the    ## dataframedf = df.sample(frac = 0.15)fig = px.scatter(df, x="Customer_Age", y="Credit_Limit",         color =   'Gender',   trendline = ‘ols’)fig.update_layout(title =       'Relation between Customer Age and Credit Limit',        xaxis_title = 'Customer Age', yaxis_title = 'Credit Limit',        width = 1600, height = 1400)fig.write_image(path + "figscat1.png")fig.show()
Plotly Express allows you to retrieve the model parameters using results = px.get_trendline_results(fig). You may choose between .summary() for a complete statistical summary, or .params for the equation of the line, or .rsquared to get the statistical measure of fit.
# retrieve the model parametersresults = px.get_trendline_results(fig)results = results.iloc[0]["px_fit_results"].summary()coef1 = results.iloc[0]["px_fit_results"].paramscoef2 = results.iloc[1]["px_fit_results"].paramsrsq1 = px.get_trendline_results(fig).px_fit_results.iloc[0].rsquaredrsq2 = px.get_trendline_results(fig).px_fit_results.iloc[1].rsquaredprint(results)print(coef1, coef2)print(rsq1, rsq2)
Fig. 4 shows the same numerical variables faceted on the categorical variable facet_col = ‘Attrition_Flag’. This variable only takes the following two values: [Existing Customer, Attrited Customer]. Similar weak correlations and the gender bias are shown in both panels.
In the following two figures, we change one of the numerical variables (Months on Book instead of Customer Age), replace the categorical variable associated with the color argument (Marital Status), and add an annotation (35K Cluster). The second figure is the faceted version of the first one (facet_col = ‘Attrition_Flag’), also based on the categorical variable related to the condition of customers’ attrition. Both figures are exported as HTML files with the variable Education Level included in the tooltips.
fig2 = px.scatter(df, x="Months_on_book", y="Credit_Limit",           color = 'Marital_Status',hover_data = ['Education_Level'])fig2.update_layout(title =      'Relation between Months on Book and Credit Limit',      xaxis_title = 'Months on Book' ,       yaxis_title = 'Credit Limit', width = 800, height = 600)fig2.add_annotation( # add a text callout with arrow      text="35K Cluster", x= 35, y=35000,       arrowhead=1,showarrow=True)fig2.write_html(path + 'figscat2.html')fig2.show()
There is no correlation between the numerical variables. Also, there is no relationship between the variables and the attrition condition or the marital status. We decide not to plot the trendlines because more than two regression lines on the screen might confuse the audience.
To sum up:
You might use a scatter plot when you want to show the relationship or correlation that exists between two numerical variables;
Plotly Express is an excellent choice for building and customizing charts that include trendlines and faceting.
If you find this article of interest, please read my previous (https://medium.com/@dar.wtz):
Diverging Bars, Why & How, Storytelling with Divergences
towardsdatascience.com
Slope Charts, Why & How, Storytelling with Slopes