Your Ultimate Data Manipulation & Cleaning Cheat Sheet | by Andre Ye | Towards Data Science

The majority of a data science project comprises of data cleaning and manipulation. Most of these data cleaning tasks can be broken down into six areas:
Imputing Missing Values. Standard statistical constant imputing, KNN imputing.
Outlier/Anomaly Detection. Isolation Forest, One Class SVM, Local Outlier Factor outlier detection algorithms.
X-Variable Cleaning Methods. Applying custom functions, removing duplicates, replacing values.
Y-Variable Cleaning Methods. Label encoding, dictionary mapping, one-hot encoding.
Joining DataFrames. Concatenating, merging, and joining.
Parsing Dates. Auto-format-detecting string-to-datetime converting, datetime objects to numbers.
Images created by author unless explicitly stated otherwise.
Missing values often plague data, and given that there are not too many of them, they can be imputed (filled in).
Simple Imputing Methods are statistical constant measures like the mean or the median which fills in NaN (missing values) with the statistical measure of each column. The parameter strategy can be substituted with ‘mean’, ‘median’, ‘most_frequent’ (mode), or ‘constant’ (a manual value with parameter fill_value).
from sklearn.impute import SimpleImputerimputer = SimpleImputer(strategy='mean')data = imputer.fit_transform(data)
KNN Imputing is the most popular and complex method for imputing missing values, in which the KNN algorithm finds other data points similar to one with a missing value within multidimensional space.
from sklearn.impute import KNNImputerimputer = KNNImputer()data = imputer.fit_transform(data)
Before using KNN and other distance-based algorithms, the data needs to be scaled or normalized to eliminate differences in scale (for example, one column representing number of children and another representing annual salary — these values cannot be taken at face value). Using KNN imputing follows the following process:
Scale/normalize the data.KNN-impute to fill in missing values.Inverse scale/normalize the data.
Scale/normalize the data.
KNN-impute to fill in missing values.
Inverse scale/normalize the data.
Isolation Forest is an algorithm to return the anomaly score of a sample. The algorithm isolates observations by creating paths by randomly selecting a feature, randomly selecting a split value, the path length representing its normality. Shorter paths represent anomalies — when a forest of random trees collectively produce shorter path lengths for particular samples, they are highly likely to be anomalies.
from sklearn.ensemble import IsolationForestidentifier = IsolationForest().fit(X)identifier.predict(X)
The output of predictions of the anomaly detector is an array of scores from -1 to 1, positive scores representing higher chances of being anomalies.
One Class SVM is another unsupervised method for detecting outliers, suited for high-dimensional distributions where an anomaly detection method like Isolation Forest would develop too much variance.
from sklearn.svm import OneClassSVMidentifier = OneClassSVM().fit(X)identifier.predict(X)
Local Outlier Factor is the third of three commonly used outlier identifiers. The anomaly score of each sample — the Local Outlier Factor — measures the local deviation of density given a sample with respect to its neighbors. Based on the K-Nearest Neighbors, samples that have substantially lower density than their neighbors are considered outliers.
Because this algorithm is distance based, the data needs to be scaled or normalized before it is used. This algorithm can be seen as a non-linear high-variance alternative to Isolation Forest.
from sklearn.neighbors import LocalOutlierFactormodel = LocalOutlierFactor().fit(X)model.predict(X)
For all three anomaly algorithms, it is the data scientist’s choice to eliminate all anomalies. Be sure that anomalies are not just data clusters themselves — make sure that the number of anomalies are not too excessive in number. A PCA visualization can confirm this.
Applying a function to a column is often needed to clean it. In the case where cleaning cannot be done by a built-in function, you may need to write your own function or pass in an external built-in function. For example, say that all values of column b below 2 are invalid. A function to be applied can then act as a filter, returning NaN values for column elements that fail to pass the filter:
def filter_b(value):     if value < 2:          return np.nan     else:          return value
A new cleaned column, ‘cleaned_b’, can then be created by applying the filter using pandas’ .apply() function:
data['cleaned_b'] = data['b'].apply(filter_b)
Another common use case is converting data types. For instance, converting a string column into a numerical column could be done with data[‘target’].apply(float) using the Python built-in function float.
Removing duplicates is a common task in data cleaning. This can be done with data.drop_duplicates(), which removes rows that have the exact same values. Be cautious when using this — when the number of features is small, duplicate rows may not be errors in data collection. However, with large datasets and mostly continuous variables, the chance that duplicates are not errors is small.
Sampling data points is common when a dataset is too large (or for another purpose) and data points need to be randomly sampled. This can be done with data.sample(number_of_samples).
Renaming columns is done with .rename, where the parameter passed is a dictionary where the key is the original column name and the value is the renamed value. For example, data.rename({‘a’:1, ‘b’:3}) would rename the column ‘a’ to 1 and the column ‘b’ to 3.
Replacing values within the data can be done with data.replace(), which takes in two parameters to_replace and value, which represent values within the DataFrame that will be replaced by other values. This is helpful for the next section, imputing missing values, which can replace certain variables with np.nan so imputing algorithms can recognize them.
More handy pandas functions specifically for data manipulation can be found here:
towardsdatascience.com
Label Encoding is required for categorical y-variables. For example, if data has two classes ‘cat’ and ‘dog’, they need to be mapped to 0 and 1, as machine learning algorithms operate purely on mathematical bases.
One simple way to do this is with the .map() function, which takes a dictionary in which keys are the original class names and the values are the elements they are to be replaced.
data['target'] = data['target'].map({'cat':0, 'dog':1})
However, in the case where there are too many classes to manually map with a dictionary, sklearn has an automated method:
from sklearn.preprocessing import LabelEncoderencoder = LabelEncoder().fit(data['target'])data['target'] = encoder.transform(data['target'])
The benefit of using this method of label encoding is that data can be inverse transformed — meaning from numerical values to their original classes — using encoder.inverse_transform(array).
One-Hot Encoding is advantageous over label encoding in certain scenarios with several classes when label encoding places quantitative measures on the data. Label encoding between classes ‘dog’, ‘cat’, and ‘fish’ to 0, 1, and 2 assumes that somehow ‘fish’ is larger than ‘dog’ or ‘dog’ is smaller than ‘cat’.
In one-hot encoding, each y-value is a vector the length of the number of distinct classes, with a ‘1’ indicating a unique index within the vector and the remaining values marked with ‘0’s. For example, ‘dog’ may be assigned the vector [1, 0, 0], ‘cat’ to [0, 1, 0], and ‘fish’ to [0, 0, 1].
pandas has a built-in function, get_dummies, which automatically takes in a series or another array form and outputs the one-hot encoded DataFrame. For example, below the command pd.get_dummies(data[‘target’]).
Concatenation is the top-down joining of DataFrames. For example, pd.concat([df1, df2]) will yield a master DataFrame where df2 is stacked below df1.
Merging is the left-to-right process that merges two DataFrames horizontally, instead of vertically with concat(). For example, pd.merge(left_df, right_df, on='ID') will merge the left DataFrame and the right DataFrame based on the ‘ID’ column.
Joining allows for more specialized merging. By default, merging is an inner join, which only fits together rows where the keyword is in both data frames. Other types of joining include left outer joins, where all keywords in the left DataFrame are included, with rows in the right DataFrame included only if their keywords exist in the left one.
The join type of choice can be outlined in a parameter how=”outer” or “right” within merge:
pd.merge(left = left_df, right = right_df, how = 'outer', on = 'id')
Dates are perhaps the most difficult data objects to deal with, but are also essential to the data. Time is the most important variable in real-life contexts, and it’s important not to let difficulty working with it intrude accommodating it in your data.
Converting strings to dates is common — almost never do datasets come with datetime objects that can be handily accessed. The best library for converting strings to dates is dateutil, which can automatically infer the location of days, months, and years, which need to be specified with other libraries.
from dateutil import parser
dateutil can accommodate a wide variety of dates, where each of the commands yields the same result (datetime.datetime(2040, 7, 3, 0, 0)):
parser.parse(‘7/3/40’)
parser.parse(‘7–3–2040’)
parser.parse(‘7.3.40’)
parser.parse(‘July 3rd, 2040’)
parser.parse(‘July 3 2040’)
parser.parse(‘2040 July 3’)
parser.parse(‘3 July 2040’)
Using .apply(parser.parse) to a column can convert strings of almost any form into a datetime object, where attributes of it can be extracted.
Converting dates to numbers is important because while time is essential for a model’s consideration, it cannot handle datetime objects. Instead, time can be represented as an integer, where each date represents the number of days passed since the earliest date in the data. In the example below, 8 March 2016 is the earliest date in the dataset (this needs to be found and replaced).
import datetimedef convert_date_to_num(date):     return (date - datetime.datetime(5,8,16)).daysdata['number date'] = data['date'].apply(convert_date_to_num)
The function is applied to the date column (assuming the date column’s elements are datetime objects) using .apply().
Be sure to bookmark this page for easy reference if you found it helpful. If you enjoyed, you may also enjoy other data science cheat sheets: