Create New Column Based on Other Columns in Pandas | Towards Data Science

As part of data processing or feature engineering, we usually need to create additional columns out of existing ones. In today’s short guide, we will explore how to perform such operations in pandas.
Specifically, we will explore how to do so in a few different ways, using
apply() method
numpy.select() method (for a vectorised approach)
loc property
First, let’s create an example DataFrame that we’ll reference throughout the article in order to demonstrate a few concepts and showcase how to create new columns based on values from existing ones.
import pandas as pddf = pd.DataFrame(    [        (1, 'Hello', 158, True, 12.8),        (2, 'Hey', 567, False, 74.2),        (3, 'Hi', 123, False, 1.1),        (4, 'Howdy', 578, True, 45.8),        (5, 'Hello', 418, True, 21.1),        (6, 'Hi', 98, False, 98.1),    ],    columns=['colA', 'colB', 'colC', 'colD', 'colE'])print(df)   colA   colB  colC   colD  colE0     1  Hello   158   True  12.81     2    Hey   567  False  74.22     3     Hi   123  False   1.13     4  Howdy   578   True  45.84     5  Hello   418   True  21.15     6     Hi    98  False  98.1
If you need to apply a method over an existing column in order to compute some values that will eventually be added as a new column in the existing DataFrame, then pandas.DataFrame.apply() method should do the trick.
For example, you can define your own method and then pass it to the apply() method. Let’s suppose we want to create a new column called colF that will be created based on the values of the column colC using the categorise() method defined below:
def categorise(row):      if row['colC'] > 0 and row['colC'] <= 99:        return 'A'    elif row['colC'] > 100 and row['colC'] <= 199:        return 'B'    elif row['colC'] > 200  and row['colC'] <= 299:        return 'C'    return 'D'
All you need to do is to pass the above method to apply() as a lambda expression:
df['colF'] = df.apply(lambda row: categorise(row), axis=1)print(df)   colA   colB  colC   colD  colE colF0     1  Hello   158   True  12.8    B1     2    Hey   567  False  74.2    D2     3     Hi   123  False   1.1    B3     4  Howdy   578   True  45.8    D4     5  Hello   418   True  21.1    D5     6     Hi    98  False  98.1    A
For more simple operations, you can specify the lambda expressions directly to the apply() method. For instance, let’s suppose we want to create another column called colG that will add together the values of columns colC and colE. The following should do the trick:
df['colG'] = df.apply(lambda row: row.colC + row.colE, axis=1)print(df)   colA   colB  colC   colD  colE colF   colG0     1  Hello   158   True  12.8    B  170.81     2    Hey   567  False  74.2    D  641.22     3     Hi   123  False   1.1    B  124.13     4  Howdy   578   True  45.8    D  623.84     5  Hello   418   True  21.1    D  439.15     6     Hi    98  False  98.1    A  196.1
Now a more vectorised approach (and potentially better in terms of performance) is to use NumPy’s select() method as described below.
Again, let’s suppose we want to create a new column called colF that will be created based on the values of the column colC. This time, instead of defining a function we will instead create a list containing the desired conditions.
import numpy as npconditions = [  np.logical_and(df['colC'].gt(0), np.less_equal(df['colC'], 99)),  np.logical_and(df['colC'].gt(100), np.less_equal(df['colC'],199)),  np.logical_and(df['colC'].gt(200), np.less_equal(df['colC'],299)),]
Then we define an additional list containing the corresponding values that the new column will contain. Note that in the list below we do not include the default value D.
outputs = ['A', 'B', 'C']
And finally, we use the select() method to apply the conditions and also specify the default value that will be used whenever none of the specified conditions is met.
df['colF'] = pd.Series(np.select(conditions, outputs, 'D'))print(df)    colA   colB  colC   colD  colE colF0     1  Hello   158   True  12.8    B1     2    Hey   567  False  74.2    D2     3     Hi   123  False   1.1    B3     4  Howdy   578   True  45.8    D4     5  Hello   418   True  21.1    D5     6     Hi    98  False  98.1    A
Finally, another option is the loc property that in some occasions might be more efficient compared to apply() method. Note that this approach may also be a little bit more verbose compared to the solutions we discussed previously.
df.loc[ np.logical_and(df['colC'].gt(0), np.less_equal(df['colC'], 99)),  'colF'] = 'A'df.loc[ np.logical_and(df['colC'].gt(100), np.less_equal(df['colC'], 199)),'colF'] = 'B'df.loc[ np.logical_and(df['colC'].gt(200), np.less_equal(df['colC'], 299)),'colF'] = 'C'df['colF'].fillna('D', inplace=True)print(df)   colA   colB  colC   colD  colE colF0     1  Hello   158   True  12.8    B1     2    Hey   567  False  74.2    D2     3     Hi   123  False   1.1    B3     4  Howdy   578   True  45.8    D4     5  Hello   418   True  21.1    D5     6     Hi    98  False  98.1    A
In today’s short guide we discussed to add new columns in pandas DataFrames based on the values of existing columns. Specifically, we showcased how to do so using apply() method and loc[] property in pandas, as well as using NumPy’s select() method in case you are interested into a more vectorised approach.
Become a member and read every story on Medium. Your membership fee directly supports me and other writers you read. You’ll also get full access to every story on Medium.
You may also like