From Excel to Python
By: Edward De Jesus
Tags: Excel, Python, Vlookup, Numpy, Pandas

As a beginner programmer with an Excel background, I often found it hard to execute in Python common tasks that I was familiar executing in Excel. When trying to do common tasks such as changing a columns’ format, filtering a column to a subset of rows, creating new columns based on information from other columns, etc., I often ran into errors and became frustrated in the process.

Python is very powerful and has many advantages over Excel including dealing with large datasets and being capable of automating workflow, just to name a few. However learning how to program can have a very steep learning curve, so having patients and sticking with it will be rewarding in the long run.

Below I highlight a few tasks commonly performed in Excel and I showcase the code to perform similar tasks in Python.

Display and Format

One common task done in Excel is changing numeric data to the number format. Numbers can become hard to read especially when dealing with large numbers. Below I apply formatting options from the Pandas Library to fictitious data.

pd.options.display.float_format = ‘${:,.2f}’.format

The numeric data in the Salary column is separated by commas for better reading.

For more information on different format styles, check out this article.

For more display options and settings, check out the Panda’s library documentation.

Another common task performed is Excel is coverting columns with dates to a dates format to allow for slicing the data by year, month or day. You can do this in Python using Pandas’ to_datetime function. Below the date of birth column is converted to the datetime format.

df['date of birth'] = pd.to_datetime(df['date of birth'])

The date of birth column is converted to the datetime format.

Duplicates

As a novice programmer, I found it challenging to identify duplicated entries in a database using code. To identify, view and remove duplicates, I first check if there any duplicates in the database by getting a total of all duplicated entries in the database using the duplicated method from the Pandas library.

df.duplicated(['date of birth', 'Name']).sum()

1

You can then view the rows containing duplicates using Pandas’ .loc method. The duplicated method’s keep parameter is set to False to view the duplicated rows that appear earlier and later in the database.

rows_to_check_for_duplicates = ['date of birth', 'Name'] df.loc[df.duplicated(rows_to_check_for_duplicates, keep=False), :]

View of duplicated entries.

To delete the duplicated row, you can use the drop_duplicates method. The inplace parameter is set to True to affect the underlying data in place, making sure the the duplicates are removed from original copy of the database.

df.drop_duplicates(['date of birth', 'Name'], inplace=True)

The duplicated row gets removed from the database.

VLOOKUP function

VLOOKUP is one of the most commonly used Excel function. It is used to look up data in table organize vertically. For example, it can be used to look up an employee’s name using their ID number. A similar function can be perform using Pandas’s .loc method.

df.loc[df['Id']=='12340', :]

Below is another example, but this time only returning the employees name instead of the entire row.

df.loc[df.Id == '12340', 'Name'] 2 Scott, Michael

Name: Name, dtype: object

IF function

Similar to Excel’s IF function, which runs a logical test and returns one value if the condition is True, and another if the condition is False, the np.where function from the Numpy Library is used below to create a new column that classifies salaries to under or over 100k depending on the amount on the Salary column.

df['under_over_100K'] = np.where(df.Salary > 100000,

'Over 100k',

'Under 100K'

)

A new column is created based on the condition of whether or not the salary is over $100,000.

Delimiter

Excel’s Text to Column tool allows for splitting data into separate columns based on a delimiter character such as a tab, comma, semicolon, space, and others. The same can be accomplished using Python’s split string method. Below the address column is split by the comma and space delimiter(the code below uses the pipe, |, which in Python stand for the OR operator) and create city, state and zip code columns to include the information separately.

df['City'] = df.address.str.split(', | ', expand = True)[0]

df['State'] = df.address.str.split(', | ', expand = True)[1]

df['Zip Code'] = df.address.str.split(', | ', expand = True)[2]

The address column is split into a city, state, and zip code columns using the split method.

Concatenating Data

Concatenating data can be thought of as the opposite of delimiting data, where, instead of splitting columns, you are joining two or more strings into one column. Below I join the City, State and Zip Code columns (In python language, the + symbol is used to join strings) back into the address column I delimited above.

df['address'] = df['City'] +', '+ df['State']+' '+df['Zip Code']

Moving the position of columns

The positions of columns can be changed by creating a list of the columns arranged in the preferred order. Below I arrange the list of the columns to position the Salary column back to the end of the dataframe.

cols = ['Id',

'date of birth',

'Name',

'address',

'City',

'State',

'Zip Code',

'Salary',

] df = df[cols]

The Salary column is moved back to the end of the table.

Columns totals and counts

The Pandas Library provides many summary functions that can operate on a column and produce a single value. Below is an example using the sum and count function to get the total for the Salary and Name columns respectively.

df['Salary'].sum()

430000.0 df['Name'].count()

4

For a complete list of summary functions, check out the Pandas Library website’s Computations / descriptive stats section.

Sorting Data

Panda’s sort_values method is used to sort columns in a dataframe. To sort by multiple levels, a list can be passed in the by parameter indicating the column names in the order to be sorted by. The ascending parameter is used to control the order. This parameter can also accepts alist for multiple sort orders in respect to the list of columns provided to sort by.

df.sort_values(['City', 'State', 'Name'], ascending=[False, False, True])

This table is sorted by City, State and Name in descending, descending, and ascending order, respectively.

Conditional Formatting

Conditional formatting is also possible in Python with the Panda’s library styling properties. The code below applies conditional formatting to the Salary column.

def salary_color_string(x):

if x >= 122500:

return 'darkgreen'

if x >= 110000:

return 'green'

else:

return 'limegreen'



def salary_color(x):

return f'background-color: {salary_color_string(x)}; color: white' format_dict = {

'Salary': '${:,.2f}'

} (df

.style

.format(format_dict)

.applymap(salary_color, subset=['Salary'])

.hide_index()

)