Discovering the Treasures of 22 R Exploratory Analysis Packages
By: Theophano Mitsa
Tags: Exploratory Data Analysis, Data Science

Discovering the Treasures of 22 R Exploratory Analysis Packages

@fergregory/AdobeStock

Python is all the rage these days in the Data Science field; however, R has been quietly growing for years and gathering amazing functionality currently contained in almost 17K packages (at the time of this writing). So it is not an exaggeration to say that it has amassed a treasure trove of packages in many areas of data science, one of which is Exploratory Data Analysis (EDA). With the word EDA, we encapsulate the following functionality:

General Dataframe description: Dimensions, types of data (continuous/discrete),missing cases.

Univariate statistics: Mean, variance, and other variable statistics, expressed in tabular form or plots, such as histogram and boxplot.

Bivariate statistics: Variable relations, expressed with correlations in a tabular form or graphical form, scatter plots, etc.

Outliers,i.e., data points that are significantly different from other data points.

As you can expect, much of the EDA functionality is replicated in various R packages, and therefore it is important to sift through the commonalities. In this post, we will perform the EDA of a dataset and harvest the R EDA packages, such that for each EDA step, we present some of the best options.

1. R EDA Packages and our dataset

Let us first get familiar with the names of the packages we will be exploring. Here they are in alphabetical order: AEDA, chemometrics, corrplot, DataExplorer, dataMaid, ExPanDaR,extremevalues, funModeling, ggcorrplot, Hmisc, inspectdf, OutlierDetection, outlier, PerformanceAnalytics, ggcorplot, Ggally, mvoutlier, prettyR, psych, RtutorR, skimr, summarytools. Auxiliary packages we will be using: dplyr, ggplot2, janitor, plotly, gmodels.

Our dataset is professor-salaries.csv (available on Kaggle). The dataset’s dimensions are 397 X6. The six columns are rank of the professor (can be assistant professor, associate professor, full professor), discipline (can be A or B), years since receiving the Ph.D. degree (yrs.since.phd.), years of service (yrs.service), gender (sex), and salary. Originally, the dataset has no missing values, however, because we wanted to check the behavior of the various packages when there are missing cases, we introduced an NA in the yrs.since.phd column and an NA in the yrs.service column.

2. Dataframe Description

Here we will get information on dataset dimensions, number/types of variables, unique rows, duplicate data, and missing data. We pose a question and after each question, there is code and code output. In each code snippet, the first command library(package) shows the package we use. If no library statement is included, then the used functions are from the base R package.

Q: What are the dimensions of my dataset?

dim() from the base package has the answer, where sal is the name of our dataset.

Q: How many variables does my dataset have and what are their types?

str() from base tells us. Part of the output is shown below.

Q: How many duplicate rows are there if any?

Here we use the unique() function from the base package. As can be seen below, there are four duplicate rows.

Q: To probe further, which are the duplicate rows?

Package janitor has the function get_dupes() for that. Part of the output is shown below.

Q: Are there any missing data?

Function introduce() from package DataExplorer give us the answer. As can be seen below, there are two missing values.

Q: To probe further, which variables have missing values?

The versatile apply() function from base R gives us the answer. We see that the variables yrs. since.phd, yrs.service have a missing value each.

Q: In which rows, do these missing values appear?

Rows that do not get selected by complete.cases() are the answer.

3. Univariate Statistics

3.1 Univariate Column Statistics

There are a plethora of functions in the aforementioned R packages, that report univariate statistics. I found most useful the following:

For basic univariate statistical information : The summary() function from the base package. This function reports the following statistics: min, max, mean, median, 1st/3rd quartile for continuous variables, number of NA’s, and the number of observations per level for categorical variables.

: The summary() function from the base package. This function reports the following statistics: min, max, mean, median, 1st/3rd quartile for continuous variables, number of NA’s, and the number of observations per level for categorical variables. For detailed univariate statistical information: The describe() function from the psych package. Its output includes min, max, median, mean, trimmed mean, range, skewness, kurtosis.

The describe() function from the psych package. Its output includes min, max, median, mean, trimmed mean, range, skewness, kurtosis. For ease-of-reporting: The makeDataReport() function from the dataMaid package, creates a convenient HTML file with the following information for each continuous variable: min, max, median,1st/3rd quartile, number of missing observations, number of unique values, and a histogram. For categorical variables, it provides information about the number of observations per level.

The univariate description functions of packages summarytools, inspectdf, Hmisc, prettyR, funModeling, skimr can be found in the R script in my github directory (path given at the end of the article).

3.2 Univariate Group Statistics

Sometimes, we would like to know how the statistics of a continuous variable (e.g. salary) are affected by the different levels of a categorical variable (i.e, sex). Below are two ways to achieve this:

For some basic information: We can use the tapply() function from the base package. Below, tapply() reports the mean salaries for male and female professors.

For more detailed group information: The describeBy() information from the psych package:

For sophisticated grouping and reporting: We can use the very popular dplyr functions filter(), group_by(), and summarize(). Below is an example, where we display the average salary of women and men of discipline A according to their rank:

Because a picture is worth a thousand words: Boxplots and bar plots, created with ggplot2 functions.

Figures 1 and 2 below yield valuable insights:

(a) Some male faculty have extraordinarily high salaries.

(b) The salaries of female faculty are compressed towards the lower end

(c ) There are many more full professors than Assistant or Associate ones

(d) Male faculty outnumber female faculty at all ranks.

Figure 1. Boxplot of male and female salaries.

Figure 2. Bar plot of rank with the color determined by the faculty’s gender.

4. Bivariate Statistics

4.1 Interactive Scatter Plots

A standard way to view relationships between two variables is a scatter plot. We can make the scatter plot much more informative if we make it interactive. This can be achieved with the package plotly. To view your plots on plotly.com, get an account on Chart studio, and then store your authentication credentials (username, API key) in your R session as follows:

Interactive plots on plotly.com, can also be embedded in your online publications, as shown in Figures 3 and 4. Here is the code for the interactive scatter plot of variables yrs. since.phd versus salary, with variable sex used to color the dots of the plot.

Figure 3. Interactive scatter plot.

4.2 Interactive 3D plots

Package plotly also allows the creation of interactive 3D plots, where we can zoom in, change the viewing angle, etc. Figure 4 shows the 3D plot of salary versus yrs.since.phd and yrs.service, where the color is provided by variable sex.

Figure 4. Interactive 3D plot

4.3 Correlations

Correlation is another way to view the relationship between two variables. Several packages in R compute correlations and display them either numerically or graphically.

4.3.1 Correlation Displayed Numerically

The stats package has the cor() function that displays the correlation of numerical variables in a tabular form:

In order to determine whether the correlation is statistically significant, we can use the cor. test() method of the stats package. For example, the correlation between salary and yrs. service is statistically significant because it yields a p-value=6.901e-12

4.3.2 Correlation Displayed Graphically

corrplot is a package that offers many insightful ways to view numeric correlations. You get to choose your main shape (circle, square, pie, ellipse, etc) and the degree of correlation is displayed by varying the size of the shape and/or its color. Below in Figure 5, we use the pie shape. The weaker the correlation, the more incomplete the pie shape and also the lighter the color.

Figure 5. Correlations of numeric variables

Package ggcorrplot produces both numeric and graphical correlation output. Its numeric output is very similar to the one produced by package stats and its graphical output is similar to the one produced by corrplot, so we will not be showing it here.

Package PerformanceAnalytics contains the function chart.Correlation(), which displays the correlation of the numerical variables in the upper half panels, while in the diagonals it plots the histograms of the numerical variables, and in the bottom half, it displays the scatter plots of pairs of the variables (Figure 6).

Figure 6. The output of chart.Correlation().

Now, we will look at three packages (DataExplorer, psych, and GGally), which, along with numerical variable correlations, also include categorical variable information. Package DataExplorer contains the function plot_correlation() that displays the correlation in colored cells. It is interesting to note that it also displays the correlations with the different levels of categorical variables as displayed below (Figure 7).

Figure 7. Output of plot_correlation().

Packages psych and Ggally provide the functions pairs. panels() and ggpairs() respectively that display correlations, histograms, and scatter plots. My favorite is the output of ggpairs() that gives you a bird’s eye view of the dataset’s variables and their relationships, including the categorical variables as well (Figure 8).

Figure 8. The output of ggpairs() function.

4.4 Test of Independence and Statistically Significant Differences

Although not a traditional part of exploratory analysis, I personally get many useful insights from performing independence tests on contingency tables. These tables have at least two rows and two columns and display multivariate frequency counts of at least two categorical variables. Statistical tests, such as the chi-squared test and the Fisher test, can be used to determine whether the displayed categorical variables are independent.

Both chi-square and Fisher tests yield similar p-values (chi-square test p-value =0.0140, Fisher test p-value=0.0114) that indicate rank and gender are not independent (with a significance level of 0.05).

Regarding continuous variables, quite often, we would like to know whether the continuous variable varies between two groups. For example, we would like to know whether the salaries of the male faculty are statistically different from the salaries of female professors. As shown below, a t-test can help us answer this question:

The t-test yields a p-value=0.002664 (significance level=0.05), which means that the salaries of male and female faculty are statistically different. To probe a bit further, we wonder whether the difference might be due to the different numbers of men and women, at the different ranks. So to eliminate the effect of rank, let us do a t-test on the salaries of male and female full professors (rank=’Prof’). This test indicates the salaries of the two groups are NOT statistically different when the variable rank is fixed(p-value = 0.3098 with significance level=0.05)

5. Outliers

Outliers are points that are distant from the overall distribution of a variable. They can be completely useless, for example resulting from errors in measurement, or instrumental in anomaly detection applications. Here we will discuss how to detect them, not whether or how we could remove them. We will distinguish between univariate and multivariate outlier detection. In the univariate case, we want to fight the extreme values of a single variable. In the multivariate case, we are interested in finding multidimensional outliers, which are very useful in clustering applications, for example.

5.1 Univariate Outlier Detection

Here we will examine three methods of univiate outlier detection that utilize: (a) boxplots (b) package extremevalues (c ) package OutlierDetection.

(a) Boxplot: In a boxplot, outliers are identified as points that lie in a distance greater than 1.5 times the interquartile range either below or above the whiskers. Boxplots in R, mark the outliers, however, they do not show their value. As previously, we can use plotly to make them interactive and this way, enable the user to click on the outliers and view their values:

Figure 9. Interactive Boxplot

By clicking on the largest male outlier in Figure 9, we see that its value is $231,545.00. The next two outliers (that actually overlap) have values of $205,500.00 and $204,000.00 respectively.

(b) Package extremevalues: It provides a function that finds the outliers (getOutliers()) using regression on a subset of the data. In Figure 10, which is created using function outlierPlot(), we see again that three values were identified as outliers.

Figure 10. QQ plot of outliers.

( c ) Package OutlierDetection: It provides UnivariateOutlierDetection(), a function whose default method for outlier detection is bootstrapping. This method is based on the clever observation that outliers are occasionally not included in bootstrap estimates [1]. As can be seen in Figure 11 below, it identifies a lot more than 3 outliers, but the three most distant ones at locations 44, 365, 250, correspond to the ones identified in the histogram with salary values $231,545, $205,500, $204,000.

Figure 11. Output of UnivariateOutlierDetection()

5.2 Multivariate Outlier Detection

Here we will discuss methods in packages OutlierDetection and chemometrics.

(a) Package OutlierDetection provides a function with the same name, which given two variables (in our case salary and yrs. service), creates a scatter plot denoting the outliers in red, as shown below in Figure 12. Some notes about the code below: (a) We need to remove the NAs, in order to get the correct output from OutlierDetection(). If you recall, we introduced one NA in variable yrs.service. (b) In the fourth line, we reverse the columns of our dataframe in order to have variable salary appear as variable 2 on the plot and therefore obtain a plot that has the same orientation as in Figure 11 above. (c ) Finally, because NA removal removed one row from our dataframe, the indices of the salary outliers shown in Figure 12, are -1 the indices of the salary outliers shown in Figure 11 (43,249,364).

Figure 12. Output of OutlierDetection()

(b) Package chemometrics has a function Moutlier() that detects outliers using the Mahalanobis distance, which measures the distance between two points. In contrast to the Euclidean distance, the Mahalanobis distance takes into account the correlation of the point values, and the distances are measured relative to a centroid. Only when the points are uncorrelated, the Mahalanobis distance becomes equal to the Euclidean distance.

As we can see above, the point at location 43 has the highest Mahalanobis distance, followed by points 282, 131, 330, etc. If we look back at Figure 12, we can see that these numbers correspond to some of the most distant outliers seen on the plot, so there is an overall agreement between the Mahalanobis distance method and the bootstrapping method in identifying the outliers. Finally, package mvoutliers also uses the Mahalanobis distance to compute outliers, however, it uses the robust estimate of the Mahalanobis distance. The results are similar to those of the regular Mahalanobis distance, albeit ordered differently.

6. Automatic EDA

Finally, let us discuss automatic EDA, i.e, the ability to create a complete (descriptive, univariate, bivariate) EDA report with a single command. We will discuss four packages that offer this capability:

Data Explorer

AEDA (Available from https://github.com/tuanle618/AEDA

ExpanDaR

RtutoR

DataExplorer, is a popular R EDA package and its EDA report (report.html) is generated as:

It provides a good summary of missing data, univariate distribution (histogram, qqplot, bar chart), correlation analysis, bivariate analysis (boxplot, scatterplot), guided by the target variable. It also includes a principal component analysis.

The EDA report of AEDA (MainReport.rmd) is created as:

An HTML format of the report can be created by loading the .rmd report on Rstudio and clicking on the Knit button. Its strong points are a detailed report of numeric variables, including skewness, kurtosis, and advanced analysis that includes cluster summary report, principal component analysis, multidimensional scaling report, and factor analysis report.

The strong point of ExPanDaR’s auto-EDA is that it allows you to launch a Shiny application:

This Shiny application allows interactive specification of multiple variables, including:

factors for which to display the bar chart (including the choice of an additional factor on the same chart).

numeric variables for which to display the histogram (including specification of cells’ number). Note that after each histogram is displayed, extreme observations are also reported.

pairs of variables for which to display the scatter plot. Also, it allows the specification of the variable that is reflected in the color of the plot.

the target variable and independent variables of regression analysis. Also the specification of categorical variables for fixed effects.

Finally, package RtutorR provides a concise, basic, but convenient Powerpoint EDA report:

Its univariate analysis contains histograms of numeric variables (and min, max, percentile information) as well as bar plots of categorical variables (and the number of occurrences per level). Its bivariate analysis contains boxplots of numerical variables grouped by the categorical variables and scatter plots of numerical variables (along with the correlation coefficient).

You can find my R script and pdf versions of automated EDA reports by the aforementioned packages in my github directory https://github.com/theomitsa/R-EDA

HTML versions of auto-EDA reports can be found at HTML link

Thanks for reading!

References

[1] Singh, K. and M. Xie, Bootlier Plot-Bootstrap Based Outlier Detection Plot, vol. 65, part 3, pp. 532–559, 2003.