How to Sample a Dataframe in Python Pandas | by Angelica Lo Duca | Towards Data Science

It may happen that you need only some rows of your Python dataframe. You can achieve this result, through different sampling techniques.
In this tutorial, I illustrate the following techniques to perform rows sampling through Python Pandas:
random sampling
sampling with condition
sampling at a constant rate
The full code can be downloaded from my Github repository.
In this tutorial, I exploit the iris dataset, provided by the scikit-learn library and I convert it to a pandas dataframe:
from sklearn.datasets import load_irisimport pandas as pddata = load_iris()df = pd.DataFrame(data.data, columns=data.feature_names)
The dataset is composed of 4 columns and 150 rows.
Given a dataframe with N rows, random Sampling extract X random rows from the dataframe, with X ≤ N. Python pandas provides a function, named sample() to perform random sampling.
The number of samples to be extracted can be expressed in two alternative ways:
specify the exact number of random rows to extract
specify the percentage of random rows to extract. The percentage is expressed as a number between 0 and 1.
In this case you can pass the parameter n to the sample() function, as follows:
subset = df.sample(n=100)
In the previous example, the sample() function extracts 100 random rows. You can check the shape of the subset resulting dataset through the shape function:
subset.shape
which gives the following output:
(100, 4)
If you want to specify the percentage of random rows to be extracted, you can pass the frac parameter as input of the sample() function:
subset = df.sample(frac=0.5)
In the previous example, the sample() function extracts the 50% of random rows. Note that you can specify just one between the n and the frac parameters.
Sampling with condition permits to extract only some rows which satisfy a given condition. Firstly, the condition must be specified. For example, as a condition, you could select only the rows where the value of the sepal width (cm) column is less than 3:
condition = df['sepal width (cm)'] < 3
The variable condition is a series of the same size of df, containing True/False, depending on whether the row meets the condition or not.
Then, you retrieve the indexes associated to the rows which satisfy the previous condition:
true_index = condition[condition == True].index
In the current example, 57 rows satisfy the condition, thus you can sample at maximum 57 rows.
The sample() function can be applied to perform sampling with condition as follows:
subset = df[condition].sample(n = 10)
Another sampling strategy is sampling at a constant rate, which means that you want that there is a constant distance between two adjacent samples. For example, you may want to sample with a rate of 4, as shown in the following figure:
In this case, you firstly specify the rate:
rate = 10
and then, simply you can extract samples:
subset = df[::rate]
In the iris dataset the number of samples is 150, thus a sample rate of 10 will produce a subset with 15 rows:
subset.shape
which gives the following output:
(15, 4)
Once you have extracted a subset of the dataset, you can also extract the remaining part. This strategy can be used, for example, if you want to split your dataset in training and test set, for example, without using the train_test_split()function provided by the scikit-learn library.
Two possible solutions can be adopted to extract the remaining part of the dataset. Both the solution produce the same result.
The first solution drops the rows of the extracted dataframe (subset) in the original dataframe df and stores the result in a new dataframe. This can be achieved by passing the list of indexes to be deleted to the drop() function:
remaining = df.drop(labels=subset.index)
The second solution selects only the rows, in the original dataframe df, where the index is not in the list of indexes of the extracted dataframe subset:
remaining = df[~df.index.isin(subset.index)]
In this tutorial, I have illustrated how to perform dataframe sampling in different ways:
random sampling, which extract N random rows;
sampling with condition, which applies a condition and then extracts a N random rows;
sampling at a constant rate.
If you wanted to be updated on my research and other activities, you can follow me on Twitter, Youtube and Github.