Basic Statistics: A BrushUp
By: Bowen Brinegar
Tags: Stats, Statistics, Data Science, Data

If you are like me, you’ve taken a stats course, most of us have. Often, taking a class does not signify mastery more than skilled retention, however. With this article, I hope to reintroduce some to why stats can be fun (and it is going to be a part of a series I hope to write). I want to cover descriptive statistics such as Mean, Median, Mode, Range, Variance, Standard Deviation, Percentiles, and reading a scatter plot and histogram. The sample dataset will be of Wine Rating per characteristic of wine (Data Source). The medium won’t be anything more advanced than excel. All steps are also listed below the article for reference if you wish to do it independently.

First, we want to get a 10,000 ft. understanding of our data; it is a good idea to find summary statistics for each column. The summary statistics of the “quality” column of the wine data set tells a lot:

1 — sample size

2 — minimum and maximum values,

3 — the most common value (mode)

4 — the average (mean)

5 — the middle value (median)

6 — the range (max value less min value)

7 — the standard deviation (sqrt of the sum of squared distances from the mean over the n number values — 1)

All in all, standardized measures that, when compared, can immediately tell you if there are outliers, if they are miss entries, if the data is tight or scattered, or if the data needs further inspection.

Summary Statistics of Quality Column in our Data Set

After doing this, creating a histogram for each column will further help you contextualize the data. Histograms make value range “bins” and then count each data point that fits each “bin.”

In the histogram for the quality column, we can see that not every value from 0–10 is present in the data set. We can see that the mode is in the 5 “bin,” but there are many 6s as well. We can also see that while there are a few high and low rates, most of the ratings are average. We can start to get a feeling for what we want to do with the data. Next, we must find out the statistical composer of each predominant group (three in this case: high, mid, and low). To find this, do a filter on the data set for each type (3 ratings, 5 + 6 ratings, and 8 ratings in this case), then create summaries for each.

Custom Summary of Low Values (~3 ratings)

Custom Summary of Mid Values (~5–6 ratings)

Custom Summary of High Values (~8 ratings)

From this, we can tell then for the most part the factors probably have little to do with the quality of the wine. But this is data science and “probably” is not an acceptable deliverable. The above gives us a sense of extreme similarities between the three major groupings of quality, however, it is not easily interpretable. The next step is to create correlation coefficients for each grouping and then describe the percentiles (50th being the Median, 0th being the minimum, 100th being the maximum) in which each grouping falls.

Correlation Coef. of Low Values (~3 ratings)

Correlation Coef. of Mid Values (~5–6 ratings)

Correlation Coef. of High Values (~8 ratings)

As we can see by these summaries, our initial estimate of the custom summaries was accurate, there is not much correlation between the quality of the wine and the factors listed in this dataset. We could create scatter plots for each of these categories to get a visual understanding of the correlation coefficient, but instead, we will describe the significance of the number. A correlation coefficient ranges from -1 to 1. A -1 represents a strong negative correlation, a 0 represents no correlation, and 1 represents a strong correlation. For example, using the following sample data, you will get the following results.

Sample Data

No Correlation (Close to 0 Coef.)

Strong Negative Correlation (Close to -1 Coef.)

Strong Positive Correlation (Close 1 Coef.)

Now that we understand that our dataset is not offering important information to the cause of poor versus great quality wine, we have to put on our thinking caps. What are the hidden variables? If it is not the acidic and chemical composure, what is it? Maybe branding, prices, vintage, the heritage of winery, grape, type, packaging, flavor profile, etc.

One of the key takeaways of this exercise was that we know have a quality framework for how we can look at more data related to the quality of wines. We have also knocked one piece off our docket (at least for now): we know that this data is not the key. Knowing what does not work is almost as important as knowing what works. Like all sciences, to make discoveries or support hypothesis, one must do a lot of grunt work to have a diverse range of qualifying findings. It takes patience, practice, and persistence to come up with numbers that are significant.