How to Analyse Data in Python without Coding | by Tobias Krabel | Towards Data Science

Disclaimer: ⚠️ Silly jokes ahead! Also, I’m a co-creator of bamboolib.
Data is everywhere, and now more than ever, we all have free tools at our fingertips to explore them. One of the most popular python libraries for data wrangling and exploration is pandas (I guess you’ve read that sentence for the 1,354th time now). But, let’s be honest, it doesn’t have the most intuitive syntax. This not only makes it pretty tough to learn, it’s also quite hard to keep the syntax in your head if you don’t work with pandas day in, day out.
pandas doesn’t have the most intuitive syntax
Also, writing pandas code sometimes feels a bit like chopping vegetables with a nordic war axe: it does the job, and I somehow feel cool while doing it, but when looking at the result, it just doesn’t look good, so I’d rather not actively participate in that mess.
With that weird image in my mind, I felt a strong urge to create a GUI that generates Python code for me so that I can focus on getting insights from my data and sharing them with my beloved ones... um... I mean, my co-workers.
bamboolib is a free GUI that lets you prepare and explore your data. For every time you edit your data and for every interactive plot you create with the tool, it will create the python code for you. This way, you can track your steps, repeat your analyses, and create interactive reports.
Focus on getting & sharing insights instead of coding
This blog post is for you if you want to analyse and explore data with Python pandas using a Jupyter Notebook/JupyterLab but a) are a complete beginner to it, b) don’t want to look up pandas syntax all the time or b) simply want to understand your data quickly without having to hack too much code into your notebook.
You will learn:
How to explore a data set you’ve never seen before (fingers crossed) instantly without having to write a single line of code yourself
while still producing python code on the way.
Concerning the format of the blog post, I thought I’d do an experiment: I’ll explore a data set I’ve never seen before and go through my EDA steps with you. So, no matter how messy/clean/boring/exciting the data, I will explore and write a blog post about it which I will share with you. And you (reading these lines) are the witness that I did!
So, where to get the data from? Since I used to code R and write about it quite a bit, I went to the R for data science “TidyTuesday” site and searched for the first random data set that caught my attention.
And there I found it: data on “Global deforestation — Soybean production and use” ❤️. That sounds as much fun as “spending a month on a lonely island with your parents-in-law ...”, but at least it caught my attention.
In short, the data set shows how many tonnes of soybeans are produced for animal feed, human food, and fuel.
Before we get started, let’s install bamboolib as described in the docs.
pip install bamboolibpython -m bamboolib install_extensions
Let’s open a Jupyter Notebook or JupyterLab, which ever you prefer. You can get the data yourself by typing
import pandas as pdsoybean_use = pd.read_csv('https://raw.githubusercontent.com/rfordatascience/tidytuesday/master/data/2021/2021-04-06/soybean_use.csv')
Let’s take a look at it using bamboolib. Run the following code in your notebook cell:
import bamboolib as bamsoybean_use
The column entity caught my attention, as I’m not sure what values to expect in it. The first observations are from Africa, so I thought entity may contain continents, but scrolling through the data reveals that entity also contains countries and some artificial clusters such as “Net Food Importing Developing Countries”.
I’m only interested in the countries though. So how to remove the rest?
I have a feeling that code refers to Country Code (just kidding, it actually says so in the data description), so if I drop all rows that have no code, I should only have countries in my data. But before blindly throwing away any rows, I check first that entities with missing codes are indeed anything but countries (“cause we are careful data scientists!”). I do so by filtering all rows where code is missing and then looking at the filtered data to verify that there are no countries in the data anymore.
For that, I search for “filter” in the search field and use the UI to filter all rows where code is missing
I give that temporary data the new name missings just because I didn’t want to overwrite the original data set. After executing the filter, I get some feedback from the interface that 8,163 rows (82% of the data) have been removed. Also, bamboolib has generated the following code for me:
missings = soybean_use.loc[soybean_use['code'].isna()]
When I click on the entity column header of the new missings dataframe, I get a column summary:
I clicked on the show all value counts button and scrolled through all entity values, and — as expected — there are no countries in the data anymore (If you thought “Northern Africa” is a country, please send me a message :-)).
Since I know now that dropping rows where code is missing won’t throw away any countries, I undo that filter from above (by clicking the undo button) and select all rows where code is not missing.
After the filter, my data looks like so:
Now, let’s take a look at the remaining columns.
year is obvious.
human_food contains the amount of soy (in tonnes) used for human food (tempeh, tofu, soy milk, edamame beans, etc)
animal_feed contains the amount of soy (in tonnes) directly fed to animals.
processed contains the amount of soy (in tonnes) processed into vegetable oil, biofuel, and processed animal feed such as soybean cake (mmmm ... soybean cake)
Right now, the data is in wide format, but I’d like to have it in long format for easier plotting (Having long format data — also known as tidy data — makes aggregations and plotting more convenient. Why, you may ask? You will see in a bit).
For that, we need to reshape our data.
I could go ahead now and try to explain a reshape from wide to long in text, but I’d rather put some ❤️ into a nice colorful image that explains that even better. In the end, what I want to get is the following:
In my case, ID will correspond to the three columns entity, code, and year.
In the UI, I search “reshape” and get two options to choose from. Since I wanna go wide → long, I choose the “Unpivot/Melt” transformation.
I select entity, code, and year as ID columns. Et voilà:
The generated code is:
# Step: Melt columns based on the index columns entity, code, yearsoybean_use_long = soybean_use.melt(id_vars=['entity', 'code', 'year'])
I’ll also give the two columns variable and value more intuitive names. For that, I use the “rename columns” transformation.
The generated code is:
# Step: Rename multiple columnssoybean_use_long = soybean_use_long.rename(columns={'value': 'soy_use_in_tonnes', 'variable': 'category'})
With this data, I feel comfortable to start some visualizations. For example, I’d like to show the following:
How the global soy production has changed over time (assuming I have data for every country on the planet)How worldwide soy use has changed in each categoryAnd many more, but then this would become as boring as one of those 4 hours long dia shows hosted by my grandma. So let’s just stick with the two items above.
How the global soy production has changed over time (assuming I have data for every country on the planet)
How worldwide soy use has changed in each category
And many more, but then this would become as boring as one of those 4 hours long dia shows hosted by my grandma. So let’s just stick with the two items above.
But before we get to answering the questions, I’m just curious to see what are the countries with the highest soy use in tonnes, so I sort my data descending by soy_use_in_tonnes.
which produces the code
# Step: Sort column(s) soy_use_in_tonnes descending (Z-A)soybean_use_long = soybean_use_long.sort_values(by=['soy_use_in_tonnes'], ascending=[False])
When doing that, I see that there’re still entries in our data which are not from a country but comprise the whole world as a total!
This is quite fortunate! I wanted to display global soy use anyway, so instead of producing the total numbers myself, I simply use the data from the world entity.
I do so by filtering the rows where entity is equal to “World”. I call that data soy_use_world. Your poor thumb (or pointer if you’re using computer and mouse to read this) will probably scroll you through a lot more stuff today, so I spare you the screenshots here. The resulting code is:
# Step: Keep rows where entity is one of: Worldsoy_use_world = soybean_use_long.loc[soybean_use_long['entity'].isin(['World'])]
So with soy_use_world in my hands, let’s finally create some interactive plots!
How has the global soy production changed over time? To answer that, we use soy_use_world and sum soy_use_in_tonnes by year. For that, I search “aggregate” in the text field.
The code:
# Step: Group by and aggregatesoy_use_world_by_year = soy_use_world.groupby(['entity', 'year']).agg(soy_use_in_tonnes_sum=('soy_use_in_tonnes', 'sum')).reset_index()
Then, I click on Create plot.
Let’s create an area chart (cause line graphs are so 2020!) that shows the soy production over time.
The code that has been produced is:
import plotly.express as pxfig = px.area(soy_use_world_by_year, x='year', y='soy_use_in_tonnes_sum')fig
Next, I visualize the soy use in each category over time. For that, I undo the aggregation step from above and create the following visualisation:
Remember me writing about reshaping the data from wide to long above? No? No worries, you’re not the only one skimming blog posts for relevant content ;-). But if you read everything until now, you’d finally seen why I reshaped the data. I did so because with data in long format, I only have to select category as the color attribute of my plot to create a stacked bar chart with plotly. That’s the beauty of what Hadley Wickham calls tidy data.
Interesting! It looks like most soy is processed into oil, fuel, and soybean cake.
The last thing I wanted to show you: with the plot creator, you can tweak your graph quite a bit. Let’s give it a nicer color palette and general figure theme , create a catchy figure title, and remove the axes labels.
The final plot looks like this:
Pretty neat, ey? Well, the code is not as neat:
import plotly.express as pxfig = px.bar(soy_use_world, x='year', y='soy_use_in_tonnes', color='category', color_discrete_sequence=px.colors.qualitative.Pastel, template='presentation', title='Where has my soy gone? Global soy use in tonnes', labels={'soy_use_in_tonnes': ' ', 'year': ' '})fig
It’s good that I hadn’t have to google that stuff myself!
In this blog post, we looked at a data set hopefully neither of us has seen before (at least I haven’t!) — the global production and use of soy. We explored, prepared and visualized the data in a few minutes without googling syntax and writing a single line of code, while still creating code (*mind blow*).
I hope bamboolib will help you get insights from your data quicker and with less cognitive load in your upcoming projects.
And one last thing: You can extend bamboolib with plugins (e.g. to create a transformation like “capitalize all column names, then only keep columns that end with ‘_xyz’, and finally compute the percentage difference between each pair of columns”). Read more here about how to write plugins yourself for bamboolib.
If you liked this post, please let me know what you would like to read about next!