The 80/20 Split Intuition and an Alternative Split Method | by The Data Detective | Towards Data Science

I am an inquisitive person. It’s my nature. It’s why I deconstructed many of my Christmas presents as a child, I wanted to know “why”. So when I ask the question, “Why are we using an 80/20 training/test split on this dataset?” and don’t get a specific answer, it drives me crazy. I dove into the ‘why’ of train/test splits. There is probably way more out there than what I can find, but I am going to cover a couple of train/test splitting practices that are out there in the “interverse” and see how they measure up or if it even matters.
Pareto Principle
The Pareto Principle is also called the 80/20 rule. The general point is that, in most cases, 80% of effects come from 20% of causes. For those that want to travel down this rabbit hole, feel free to check out the Wikipedia. While the preface of the principle was based on wealth distribution, it statistically does come close to explaining many human, machine, and environmental phenomena. For this reason, and typically without knowledge of the source, many use the 80/20 split for training and testing.
Scaling Law
In 1997, a new method was discussed in a paper called A scaling law for the validation-set training-set size ratio (Guyon). Here, they reference “ the best training/validation split for a specific problem: preventing overtraining of neural networks. They find that the fraction of patterns reserved for the validation set should be inversely proportional to the square root of the number of free adjustable parameters. Our result generalizes and confirms their result.” (Guyon, 1997). In essence, the split is determined by how many unique features are in the dataset (not including the target) and not the number of observations, which is contrary to most opinions on the subject.
Let’s find out if the scaling law outperforms an assumed Pareto split using our trusty Titanic data. This dataset has already been imputed to fill nulls from a previous post Preprocessing: Encode and KNN Impute All Categorical Features Fast.
First, we’ll load the data, make sure all of the columns are visible and run a .head() method to verify the data.
import pandas as pdimport numpy as npfrom sklearn.model_selection import train_test_splitsplit_data = pd.read_csv('titanic_encoded_scaled-Copy1.csv')pd.options.display.max_columns = Nonesplit_data.head()
Next, we will get rid of the unnamed column and split the data into X and y variables. Since we are testing model improvement, it was best to use a multi-class target, so the deck1 column was used (7 classes).
split_data = split_data.drop('Unnamed: 0', axis=1)X = split_data.iloc[:,:-2]y = split_data.iloc[:,-2]
From another previous post, we found that this data scaled best with the PowerTransform scaler set to the ‘yeo-johnson’ method. If you want to take a peek at that, just take a look: Preprocessing: Differences in Standardization Methods
from sklearn.preprocessing import PowerTransformeryj = PowerTransformer(method = 'yeo-johnson')yj_data = yj.fit_transform(X)
Now, we will split our data into train and test using the sklearn library. First, the Pareto Principle (80/20):
#Pareto Principle SplitX_train, X_test, y_train, y_test = train_test_split(yj_data, y, test_size= 0.2, random_state= 123)
Next, we will run the function to apply the scaling law and split that data into different variables:
#discover scaling law splitcolumns = 14test = 1/np.sqrt(columns)train = 1 - testprint(test)print(train)#Scaling Law SplitA_train, A_test, b_train, b_test = train_test_split(yj_data, y, test_size= test, random_state= 123)0.2672612419124244 #test0.7327387580875756 #train
Almost 7% difference between the Pareto and Scaling Law splits. Now to run a standard logistic regression model for both datasets:
#perform logistic regression using pareto splitfrom sklearn.linear_model import LogisticRegressionbn1 = LogisticRegression(max_iter=1000)bn1.fit(X_train, y_train)pp_preds = bn1.predict(X_test)# use logistic regression using scaling law splitbn2 = LogisticRegression(max_iter=1000)bn2.fit(A_train, b_train)sl_preds = bn2.predict(A_test)
Here is the classification report for the Pareto Principle:
#classification report from pareto splitfrom sklearn.metrics import classification_reportprint(classification_report(y_test, pp_preds))                 precision  recall   f1-score    support         0.0       0.00      0.00      0.00         5         1.0       0.38      0.33      0.36        15         2.0       0.37      0.59      0.45        17         3.0       0.38      0.16      0.22        19         4.0       0.52      0.34      0.41        38         5.0       0.67      0.87      0.76        69         6.0       0.82      0.88      0.85        16    accuracy                           0.59       179   macro avg       0.45      0.45      0.44       179weighted avg       0.55      0.59      0.55       179
A decent 59% accuracy. Now, the Scaling Law report:
#classification report from scaling lawprint(classification_report(b_test, sl_preds))                precision    recall  f1-score   support         0.0       0.50      0.20      0.29         5         1.0       0.31      0.26      0.29        19         2.0       0.35      0.52      0.42        21         3.0       0.30      0.12      0.17        26         4.0       0.61      0.47      0.53        53         5.0       0.71      0.88      0.78        97         6.0       0.84      0.89      0.86        18    accuracy                           0.61       239   macro avg       0.52      0.48      0.48       239weighted avg       0.59      0.61      0.59       239
An increase of 2% accuracy and improvement in precision, recall and f1-score!
Wrapping it up
In the grand scheme of things, 2% is a pretty significant increase, in my opinion. Of course, this is a simple multi-class Logistic Regression model, but as I try to encourage, test both split methods on whatever dataset you have. It really is a simple algorithm that can make that extra difference.
As always, you can find my work on Github. This particular repo is here. Have a great one!