Why I switched from Excel to Python?
By: Wole Olorunleke
Tags: Data Science, Business Intelligence, Pandas, Excel, Data Analysis

Photo by Myriam Jessier on Unsplash

After spending a good portion of my adult life using Excel to carry out all forms of analysis from Undergraduate Assignments and Projects to Product Performance Metrics evaluation, in 2020, I finally decided to complete my switch to Python to improve the speed and reproducibility of my results.

Anyone who uses Excel often enough knows how tedious working with large files can get. Those large files and their accompanying computations can force you to take mandatory breaks as your computer is nothing but useless in that period.

Some Background

I joined gomoney to lead the Strategy and Analytics Function (Data and Financial) to guide performance analysis, financial analysis, and provide the strategy to help achieve our growth targets.

The goal here was to ensure the product was performing according to the agreed benchmarks and measuring these metrics frequently enough to ensure we were nimble with our market approach.

At the time, this involved measuring the following daily:

Transaction Performance;

User Acquisition Performance;

Google Analytics Information and Metrics

In the first few weeks, this was not tedious as I was using a decently powered computer and created what I had considered a “scalable” approach.

My “Scalable” approach:

These are the steps that I thought were scalable at the time.😂

I see you judging me!

Download the transaction reports for the previous day as a CSV

Add this information to the Excel Sheet with all transactions.

Apply all the transformations needed for my analysis

Create a mini-dashboard to evaluate all the results from the transformation.

My Experience

For the first three (3) weeks, this was phenomenal, I thought I had conquered the reporting, and my thoughts were around how to improve the reporting to get more information and better evaluate the metrics such as Retention and Customer Activation (I will talk about these in future posts)

At the end of Week 3, I noticed that the report was taking about 18 minutes, and I just assumed this was because I had not restarted my computer in the last ten (10)days. The slow down persisted into the new week and got worse over the next couple of week. Still, as anybody who uses Excel knows, this tends to happen with reports that include dashboards, so I just scheduled the work till I did not have to use my computer actively and accepted this to be my “new normal” (before COVID ruined the meaning of that phrase).

…the straw that broke the camels back

In June, we ran a marketing campaign that drove up adoption and, consequently, the solution's usage. This campaign drove up downloads, activation, transaction value and volume. The increased volume meant the report now required an average of 90 minutes with my fan at full blast and the laptop sitting on our Air Conditioning unit.

Then it happened, the report was running for a while, and suddenly my computer just went off and restarted itself.

Me when the computer restarted after 90 minutes of running a report

And at that point, I knew that I needed a new solution to this. I had been tinkering with Python for years. Still, I had never made the switch entirely, and I had not used it for anything outside the occasional Linear Regression due to the results' speed and reliability.

When this happened, I decided to double down and design a training plan that replicated the things I regularly did with Excel in Python to get my feet wet and manage the transition.

How I made the transition?

I decided to understand the most fundamental libraries for Data Analysis and picked those I was going to learn.

I also wanted a place where I could see the analysis in real-time, and this is where Jupyter Notebooks came in handy as opposed to Pure Python where the output would have required print statements and could have been a bit tedious.

An actual Scalable Approach

Connect directly to the Database to get all the information required for transactions based on the period under consideration

Create a transformation script that captures all the changes required to carry out the analysis

Prepare a function that outputs all the considerations for the period and then make decisions or carry out further analysis if there is a need for this.

Feeling like a boss

Guide to Getting Acquainted with Python (Using the Chipotle Data set)

Though I achieved the Scalable Approach, I had to take baby steps to get there, and those baby steps are itemised below.

I will use the Chipotle dataset to capture some of the things that I would consider the first steps to seeing Python as a natural extension of Excel.

Before we jump ahead, let me provide some context on the libraries that will be the most useful for your transition.

The Primary Libraries for Data Analysis

Pandas — This is an open-source library that can be used for data analysis, data manipulation and data visualisation (Though, data visualisation in pandas is quick, it is a bit limited and you should consider other libraries such as Seaborn and Plotly — my personal favourite)

— This is an open-source library that can be used for data analysis, data manipulation and data visualisation (Though, data visualisation in pandas is quick, it is a bit limited and you should consider other libraries such as Seaborn and Plotly — my personal favourite) Numpy — This is a scientific computing library that brings speed to a lot of computation. The execution speed comes from the implementation that converts Python to well-written C Code that is optimised for numerical analysis. Numpy is fast, like really fast.

— This is a scientific computing library that brings speed to a lot of computation. The execution speed comes from the implementation that converts Python to well-written C Code that is optimised for numerical analysis. Numpy is fast, like really fast. Matplotlib (I’m not too fond of it, but it is a necessary evil) — This is a visualisation library with a lot of flexibility. Still, the default leaves a lot to be desired. I don’t use it, but it provides an excellent introduction to visualisation. My preferred library is Plotly (I will write some posts on this as I love the library because of its flexibility).

import pandas as pd

import numpy as np

import matplotlib.pyplot as plt

Jupyter Notebook: Viewing and Understanding your data and analysis

Jupyter Notebook

Jupyter Notebook comes in handy where you need to understand your data, manipulate your data, and see your evaluation results. It runs in what is called a REPL (Read → Evaluate →Print → Loop) model. REPL means that Jupyter reads your code, Evaluates it, Prints an output (optional) and then repeats this in the next cell.

The installation is super easy, and once you go with the Anaconda Distribution Default installation, you are good to go.

In a case where you have space constraints, you can go with mini-conda installation and install only the packages you want.

If you don’t have time for all of these and want to get started or use a work computer where installations are limited, I suggest you go with Google Colab and get started right away. For Google Colab, there is no installation, and you can get right into your analysis.

Open my CSV and Excel Files in Python

This involved reading files that are typically double-clicked and switching this to lines of code. The good thing is that only two code lines are required to open an XLS/XLSX/CSV file in excel.

Reading CSV and EXCEL files was achieved by importing pandas and using the read_table or read_csv or read_excel options.

import pandas as pd

df = pd.read_table('http://bit.ly/chiporders')

OR

import pandas as pd

df = pd.read_csv('chiporders.csv')

The output here shows the number of rows and the number of columns that are contained in the dataset, but you are unable to see all the information at a glance, so it is advisable to “understand” your data using describe and info.

df.info()

Interpreting the output:

Columns in Pandas are only one data type. Hence, the consistency in Dtype.

order_id — This means that the order_id column contains no empty rows (non-null), and all the values in this column are integers;

— This means that the column contains no empty rows (non-null), and all the values in this column are integers; quantity — This means that the quantity column contains no empty rows (non-null), and all the values in the columns are integers

— This means that the column contains no empty rows (non-null), and all the values in the columns are integers item_name — This means that the item_name contains no empty rows and all the values in this column are objects. I would be careful with items because a lot of data types can be represented as objects by default, so a cursory review of the column is usually necessary to make the necessary steps

— This means that the contains no empty rows and all the values in this column are objects. I would be careful with items because a lot of data types can be represented as objects by default, so a cursory review of the column is usually necessary to make the necessary steps choice_description — This means that the choice_description column contains 3,376 non-null rows which means that 1,246 columns here have no information. There are ways to manage missing values, but that would be communicated later and is a function of your exercise's objective and an understanding of the data set.

— This means that the column contains 3,376 non-null rows which means that 1,246 columns here have no information. There are ways to manage missing values, but that would be communicated later and is a function of your exercise's objective and an understanding of the data set. item_price — This means that the item_price column contains no empty rows and all the values in this column are objects. However, we know that prices should not be objects and should be decimals (2 Decimal places typically for finance professionals when reporting currency outside JPY 4DP). The issue here is the dollar sign.

df.describe()

The describe function returns an evaluation of the numeric values in the dataframe (table we imported) and provides context to how the data is distributed.