How to transform variables in a pandas DataFrame | by Zolzaya Luvsandorj | Towards Data Science

Whether it’s for preparing data to extract insights or for engineering features for a model, I think one of the fundamental skills for individuals working with data is their ability to reliably transform data to the desired format. As part of data cleaning, data preparation, data munging, data manipulation, data wrangling, data enriching, data preprocessing (whew! 😅), there is often a need to transform variables/columns/features to a more suitable form . In other words, raw data often needs a makeover ✨ to be more useful.
I assume the reader (👀 yes, you!) has access to and is familiar with Python including installing packages, defining functions and other basic tasks. If you are new to Python, this is a good place to get started.
I have used and tested the scripts in Python 3.7.1 in Jupyter Notebook. Let’s make sure you have the right tools before we start deriving.
We will use the following powerful third party packages:
numpy: Numerical computing library and
pandas: Data analysis library.
To keep things manageable, we will create a small dataframe which will allow us to monitor inputs and outputs for each task in the next section. In a hypothetical world where I have a collection of marbles 🔮, let’s assume the dataframe below contains the details for each kind of marble I own. (Psst! You may have to copy over the code to your Jupyter Notebook or code editor for a better format.)
# Import packagesimport numpy as npimport pandas as pd# Update default settings to show 2 decimal placepd.options.display.float_format = '{:.2f}'.format# Create a small dataframedf = pd.DataFrame({'name': ['bob 2012', 'Ava 2013', 'Aby 007', 'XYZ 8', 'GRZ x7', 'Boo VIII', 'Joy 2020'],                   'p_date': ['2020-02-01', '2020-05-01', '2020-06-30', '2020-04-15', '2020-01-04', '2020-03-21', '2020-07-08'],                   'count': [80, 30, 10, 60, 40, 20, np.nan],                   'colour': ['pink', 'teal', 'velvet', 'pink', 'green', 'teal', 'pink'],                   'radius': [1, 2, 3, 4, 5, 6, 7],                   'unit': ['cm', 'inch', 'cm', 'cm', 'inch', 'cm', 'cm']})df
Each row represents a kind of marble. Columns are defined as:◼ name: Name for each marble (first part is the model name and second is the version)◼ purchase_date: Date I purchased a kind of marbles◼ count: How many marbles I own for a particular kind◼ colour: Colour of the kind◼ radius: Radius measurement of the kind (yup, some are quite big 🙊)◼ unit: A unit for radius
If this doesn’t make much sense, don’t worry too much as it’s only a toy data.
In this section, we will look at some examples on transforming different data types. We will be creating new columns containing the transformation so that the original variables are not overwritten. Alternative codes to achieve the same transformation are provided for reference where possible. From these list of alternatives, hope you will find a trick or two for take away for your day-to-day data manipulation. Now, it’s time for a makeover! ✨
Please note that the underlying logic for some methods shown can be applied to any data types.
🔦 Type: Create a conditional variable based on 2 conditions.
🔒 Task: Radius is not directly comparable across kinds as they are expressed in different units. Let’s create a variable showing radius in cm for consistency.
🔑 Answer: We will call the new variable radius_cm. Given that 1 inch equals 2.54 cm, we can summarise the conditions as follows:1) If unit is ‘cm’ then radius_cm = radius2) If unit is ‘inch’ then radius_cm = 2.54 * radius
We can create radius_cm using the script below:
# Method using np.wheredf['radius_cm'] = np.where(df['unit']=='inch', 2.54 * df['radius'], df['radius'])# ============== ALTERNATIVE METHODS ==============## Method using np.select# df['radius_cm'] = np.select([df['unit']=='inch'], [2.54 * df['radius']], default=df['radius'])## Method applying lambda function with if# df['radius_cm'] = df[['radius', 'unit']].apply(lambda x: 2.54 * x[0] if x[1]=='inch' else x[0], axis=1)## Method A using loc# df.loc[df['unit']!='inch', 'radius_cm'] = df['radius']# df.loc[df['unit']=='inch', 'radius_cm'] = 2.54 * df['radius']## Method B using loc (works as long as df['radius'] has no missing data)# df['radius_cm'] = df['radius']# df.loc[df['unit']=='inch', 'radius_cm'] = 2.54 * df['radius']# Inspect resultsdf
💡 Quick tip: To comment or decomment code in a Jupyter Notebook, select a chunk of code and use [Ctrl/Cmd + /] shortcut if you don’t already know.
_________________________________________________________________🔦 Type: Create a conditional variable based on 2 conditions (Categorise).
🔒 Task: Create a variable describing marble size based on its radius in cm.
🔑 Answer: We will call the new variable size. Let’s define ‘big’ as marbles with radius of 5 cm or higher, and anything lower as ‘small’. Therefore, the conditions are:1) If radius_cm ≥ 5 then size = ‘big’2) If radius_cm < 5 then size = ‘small’
We can create size using the script below:
# Method applying lambda function with ifdf['size'] = df['radius_cm'].apply(lambda x: 'big' if x>=5 else 'small')# Inspect resultsdf.sort_values('radius_cm')
I haven’t provided any alternative for this task to avoid repetition as any method from the first task can be used here.
📌 Exercise: Try doing the same transformation using a different method by referencing methods shown in the first task.
_________________________________________________________________
🔦 Type: Create a calculated variable.
🔒 Task: Calculate sphere volume for marbles.
➗ Formula:
🔑 Answer: We can create volume using the script below:
# Method using evalpi = np.pidf.eval("volume = 4/3 * @pi * radius**3", inplace=True)# ============== ALTERNATIVE METHOD ==============## Method using simple operators# df['volume'] = 4/3 * np.pi * df['radius']**3## Method using pandas wrappers# df['volume2'] = df['radius'].pow(3).mul(4/3*np.pi)## Short description for pow, mul and a few other wrappers:## add: for addition ## sub: for subtractaction## mul: for multiplication## div: for division## pow: for exponential power (Tip: can use pow(0.5) or pow(1/2) for square root)# Inspect resultsdf
_________________________________________________________________🔦 Type: Segment numerical values into equal width bins (Discritise).
🔒 Task: Create a variable that splits the marbles into 2 bins of equal width based on their counts.
🔑 Answer: We will call the new variable cut.◼ min count = 10◼ max count = 80◼ range count = max — min = 70◼ bin width = range / number of bins = 70 / 2 = 35As count ranges from 10 to 80 marbles, having 2 bins would mean that the first bin would be 10 to 45 and the second 45 to 80, each with an equal width of 35. Currently, we have defined bins to be inclusive of the rightmost edge with the default setting: right=True. This means if we had 45 marbles for a kind, it would fall into the lower bin (i.e. 10–45). We could easily change this behaviour to be exclusive of the rightmost edge by adding right=False inside the function below. Please also see my note in the next task.
We can create cut using the script below:
# Method using pandas.cutdf['cut'] = pd.cut(df['count'], bins=2, labels=['bin1', 'bin2'])# Inspect resultsdf[['count', 'cut']].sort_values('count')
_________________________________________________________________
🔦 Type: Segment numerical values into equal sized bins (Discritise).
🔒 Task: Create a variable that splits the marbles into 2 equal sized buckets (i.e. quantiles) based on their counts.
🔑 Answer: We will call the new variable qcut.◼ sorted count in ascending order: 10, 20, 30, 40, 60, 80◼ # records = 6◼ # quantiles = 2◼ # records per quantile = # records / # quantiles = 6 / 2 = 3As count has 6 non-missing values in it, having equal sized buckets would mean that the first quantile would include: 10, 20, 30 and the second would include: 40, 50, 60, each with an equal size of 3.
Of note, if you are interested to view the exact cut-off points for either the equal width or equal sized bins, one way to do so is to leave out label argument from the function. Viewing the exact cut-off points will provide clarity on how the points that are on the edge are treated when discretizing.
So anyway... getting back to qcut, we can create it using the script below:
# Method using pandas.qcutdf['qcut'] = pd.qcut(df['count'], q=2, labels=['q1', 'q2'])# Inspect resultsdf[['count', 'cut', 'qcut']].sort_values('count')
Notice the difference between cut and qcut?
_________________________________________________________________
🔦 Type: Create a conditional variable based on 3+ conditions (Group).
🔒 Task: Create a variable that abbreviates pink into ‘PK’, teal into ‘TL’ and all other colours (velvet and green) into ‘OT’.
🔑 Answer: We will call the new variable colour_abr. So the conditions are:1) If colour is ‘pink’ then colour_abr = ‘PK’2) If colour is ‘teal’ then colour_abr = ‘TL’3) If colour is either ‘velvet’ or ‘green’ then colour_abr = ‘OT’.
We can create colour_abr using the script below:
# Method using replacedf['colour_abr'] = df['colour'].replace({'pink': 'PK', 'teal': 'TL', 'velvet': 'OT', 'green': 'OT'})# ============== ALTERNATIVE METHODS ==============## Method A using map# mapping = {'pink':'PK', 'teal': 'TL', 'velvet': 'OT', 'green': 'OT'}# df['colour_abr'] = df['colour'].map(mapping)## Method B using map (works as long as df['colour'] has no missing data)# df['colour_abr'] = df['colour'].map({'pink':'PK', 'teal': 'TL'})# df['colour_abr'].fillna('OT', inplace=True)## Method using nested np.wheres# df['colour_abr'] = np.where(df['colour']=='pink', 'PK', np.where(df['colour']=='teal', 'TL', 'OT'))## Method using np.select# df['colour_abr'] = np.select([df['colour']=='pink', df['colour']=='teal'], ['PK', 'TL'] , default='OT')## Method applying lambda function with nested ifs# df['colour_abr'] = df['colour'].apply(lambda x: 'PK' if x=='pink' else ('TL' if x=='teal' else 'OT'))## Method using list comprehension# df['colour_abr'] = ['PK' if x=='pink' else ('TL' if x=='teal' else 'OT') for x in df['colour']]## Method A using loc# df.loc[df['colour'] == 'pink', 'colour_abr'] = 'PK'# df.loc[df['colour'] == 'teal', 'colour_abr'] = 'TL'# df.loc[df['colour'].isin(['velvet', 'green']), 'colour_abr'] = 'OT'## Method B using loc (works as long as df['colour'] has no missing data)# df['colour_abr'] = 'OT'# df.loc[df['colour'] == 'pink', 'colour_abr'] = 'PK'# df.loc[df['colour'] == 'teal', 'colour_abr'] = 'TL'# Inspect resultsdf[['colour', 'colour_abr']].sort_values('colour_abr')
If we were just renaming the categories instead of grouping, we could also use either of the following method from .cat accessor in addition to the methods shown above:
# Create a copy of colour and convert type to categorydf['colour_abr'] = df['colour'].astype('category')# Method using .cat.rename_categoriesdf['colour_abr'].cat.rename_categories({'green':'GN', 'pink':'PK', 'teal':'TL', 'velvet': 'VT'}, inplace=True)# ============== ALTERNATIVE METHOD ==============## Method using .cat.categories## Make sure to get the order of the categories right## Check the order with by running df['colour_abr'].cat.categories# df['colour_abr'].cat.categories = ['GN', 'PK', 'TL','VT']# Inspect resultsdf[['colour', 'colour_abr']].sort_values('colour_abr')
See this documentation for more information on .cat accessor. 🐱
_________________________________________________________________
🔦 Type: Parse a string (Extract a part from a string).
🔒 Task: Parse name such that we have new columns for model and version.
🔑 Answer: We will now use a method from .str accessor to extract parts:
# Method using .str.splitdf[['model', 'version']] = df['name'].str.split(' ', expand=True)# ============== ALTERNATIVE METHOD ==============## Method applying lambda function# df['model'] = df['name'].apply(lambda x: x.split(' ')[0])# df['version'] = df['name'].apply(lambda x: x.split(' ')[1])# Inspect resultsdf[['name', 'model', 'version']]
_________________________________________________________________
🔦 Type: Concatenate or combine columns (Opposite of task above).
🔒 Task: Combine values in model (make it uppercase) and radius in a new column.
🔑 Answer: We will now use the script below to concatenate:
# Method using + operatordf['mod_rad'] = df['model'].str.upper() + '_' + df['radius'].astype(str)# ============== ALTERNATIVE METHOD ==============## Method using chained .str methods# df['mod_rad'] = df['model'].str.upper().str.cat(df['radius'].astype(str), sep="_")# Inspect resultsdf
See this documentation for more information on .str accessor. 🎻
_________________________________________________________________
🔦 Type: Parse a datetime (Extract a part from a datetime).
🔒 Task: Extract the days of the week, and years of purchase.
🔑 Answer: We will now use method from .dt accessor to extract parts:
# Convert type to datetimedf['p_date'] = pd.to_datetime(df['p_date'])# Method using .dt.day_name() and dt.yeardf['p_dname'] = df['p_date'].dt.day_name()df['p_year'] = df['p_date'].dt.year# Inspect resultsdf[['p_date', 'p_dname', 'p_year']]
_________________________________________________________________📌 Exercise: Try extracting month and day from p_date and find out how to combine p_year, p_month, p_day into a date.
See this documentation for more information on .dt accessor. 🕟
By scrolling the pane on the left here, you could browse available methods for the accessors discussed earlier. Look out for pandas.Series.xxx.yyy where xxx can be substituted with either cat, str or dt, and yyy refers to the method.
Now we will get familiar with assign, which allows us to create multiple variables at one go. Once tested, we can combine the steps like below:
# Convert type to datetimedf['p_date'] = pd.to_datetime(df['p_date'])# Derive variablesdf = df.assign(radius_cm = np.where(df['unit']=='inch', 2.54 * df['radius'], df['radius']),               # Referenced radius as radius_cm hasn't been created yet               size = list(map(lambda r, u: 'big' if ((u=='cm') & (r>=5)) | ((u=='inch') & (2.54*r>=5))                                else 'small', df['radius'], df['unit'])),               volume = 4/3 * np.pi * df['radius']**3,               cut = pd.cut(df['count'], bins=2, labels=['bin1', 'bin2']),               qcut = pd.qcut(df['count'], q=2, labels=['q1', 'q2']),               colour_abr = df['colour'].replace({'pink': 'PK', 'teal': 'TL', 'velvet': 'OT', 'green': 'OT'}),               # If you find a way to combine the following 2 lines in 1 line inside assign(), feel free to teach me how               model = df['name'].str.split(' ', expand=True)[0],               version = df['name'].str.split(' ', expand=True)[1],               mod_rad = df['name'].str.split(' ', expand=True)[0].str.upper() + '_' + df['radius'].astype(str),               p_dname = df['p_date'].dt.day_name(),               p_year = df['p_date'].dt.year)# Inspect resultsdf
Does this script look a bit hectic? If you focus line by line, you will see that each line is a slightly transformed version of the code that we have learned from section 2.
As a final note, when creating variables, if you make a mistake, you could always overwrite the incorrect variable with the correct one or delete it using the script below ⛔️:
# Method A using deldel(df['var_name'])# ============== ALTERNATIVE METHODS ==============## Method B using del# del df['var_name']## Method using drop# df.drop(columns='var_name', inplace=True)
Would you like to access more content like this? Medium members get unlimited access to any articles on Medium. If you become a member using my referral link, a portion of your membership fee will directly go to support me.
Thank you for reading my post. I hope that you have learned something ✂️. In case you are interested, here are links to the some of my other posts:◼️️ Introduction to NLP — Part 1: Preprocessing text in Python◼️ Introduction to NLP — Part 2: Difference between lemmatisation and stemming◼️ Introduction to NLP — Part 3: TF-IDF explained◼️ Introduction to NLP — Part 4: Supervised text classification model in Python
Keep transforming! Keep, keep transforming variables! 🎤 (sing along! 🙋)You keep transforming! You keep, keep transforming variables! 🎶Keep transforming! Keep, keep transforming variables! 🎵Go transform your data... 🎼
Did you guess my song reference? (hint: L[a-z]{4})
Bye for now 🏃💨