Shuffling Rows in Pandas DataFrames | by Giorgos Myrianthous | Towards Data Science

Data shuffling is a common task usually performed prior to model training in order to create more representative training and testing sets. For instance, consider that your original dataset is sorted based on a specific column.
If you split the data then the resulting sets won’t represent the true distribution of the dataset. Therefore, we have to shuffle the original dataset in order to minimise variance and ensure that the model will generalise well to new, unseen data points.
In today’s short guide we will discuss how to shuffle the rows of pandas DataFrames in various ways. Specifically, we will explore how to do so using
the sample() method in pandas
the shuffle() method in scikit-learn
the random.permutation() methods in numpy
First, let’s create an example pandas DataFrame that we’ll reference throughout this article in order to demonstrate how to shuffle the rows in many different ways.
import pandas as pddf = pd.DataFrame({    'colA': [10, 20, 30, 40, 50, 60],    'colB': ['a', 'b', 'c', 'd', 'e', 'f'],    'colC': [True, False, False, True, False, True],    'colD': [0.5, 1.2, 2.4, 3.3, 5.5, 8.9],})print(df)   colA colB   colC  colD0    10    a   True   0.51    20    b  False   1.22    30    c  False   2.43    40    d   True   3.34    50    e  False   5.55    60    f   True   8.9
The first option you have for shuffling pandas DataFrames is the panads.DataFrame.sample method that returns a random sample of items. In this method you can specify either the exact number or the fraction of records that you wish to sample. Since we want to shuffle the whole DataFrame, we are going to use frac=1 so that all records are returned.
df = df.sample(frac=1)print(df)   colA colB   colC  colD4    50    e  False   5.50    10    a   True   0.52    30    c  False   2.41    20    b  False   1.25    60    f   True   8.93    40    d   True   3.3
If you want the shuffling to be reproducible, then also make sure to specify a valid value for the random_state argument. Additionally, you may sometimes want to reset the index of the returned DataFrame. In this case, the following should do the trick:
df = df.sample(frac=1).reset_index(drop=True)
Another function you can use in order to shuffle a DataFrame is sklearn.utils.shuffle() as shown below:
from sklearn.utils import shuffledf = shuffle(df)print(df)   colA colB   colC  colD4    50    e  False   5.55    60    f   True   8.91    20    b  False   1.22    30    c  False   2.40    10    a   True   0.53    40    d   True   3.3
Again, make sure to configure random_state argument should you wish the results to be reproducible.
The other option we have is the numpy.random.permutation method that will randomly permute a sequence.
import numpy as npdf = df.iloc[np.random.permutation(len(df))]print(df)   colA colB   colC  colD3    40    d   True   3.30    10    a   True   0.55    60    f   True   8.91    20    b  False   1.22    30    c  False   2.44    50    e  False   5.5
Once again, if you want the results to be reproducible you will have to set the random seed of numpy. For instance,
np.random.seed(100)
In today’s short guide we discussed the importance of data shuffling in the context of Machine Learning model. Additionally, we explored how to shuffle pandas DataFrames using sample() method, shuffle() method in sklearn and random.permutations() method in numpy.
If you want to learn more about how to split the data into training, testing and validation sets once the original dataset has been shuffled then make sure to read the article below.
towardsdatascience.com
Become a member and read every story on Medium. Your membership fee directly supports me and other writers you read.
You may also like