Data Exploration 101 with Pandas. Pandas is one of the most powerful... | by Günter Röhrich | Towards Data Science

If you are new to data science, a student running out of time just ahead of an exam or you simply want to see how others may use Pandas to explore and analyse data, follow me here. For this quick introduction, we will use the daily temperature of major cities dataset which is available through Kaggle.com.
As a starting point for every data exploration task, it is necessary to import the dataset. Python makes this very easy through the ‘open()’ function, however as we are about to use pandas library anyways, the conventional open is not the way to go. Pandas provides a variety of read_ functions, that allows opening the file and converting it to a dataframe at the same time.
With just two lines you are importing data and create a dataframe object which is just brilliant to work with. Other than common Python data structures (think of a dictionary), operations can be performed on all, or just e filtered or even grouped set of data points . Just imagine you had to accomplish this task only using common data structures, this would be a nightmare.
Pandas provides you with incredibly handy and easy to apply possibilities to preview and modify and export data. In the next few lines, we will go through a couple of very useful examples to manipulate a dataframe. What you might observe, pandas is quite consistent (filtering, accessing rows, etc.) with other famous Python libraries, but also languages like R.
Almost every time a new dataset is analysed you might consider two things:
What are the ranges your data points lie in — this is essential in order to get a “feeling” for the data and also check on consistency, reliability and outliersDo you see invalid or missing values in your dataframe
What are the ranges your data points lie in — this is essential in order to get a “feeling” for the data and also check on consistency, reliability and outliers
Do you see invalid or missing values in your dataframe
Using just those to lines will allow you to examine your dataset and estimate, whether you might be required to do a lot more data cleansing. Clearly describe() makes no sense on Month, Day or Year, but AvgTemperature is essential to grasp. As we can see here, this measure is very likely not to be Celsius (mean temperature is 56 degrees), but Fahrenheit.
      Month        Day          Year          AvgTemperaturecount 2.906327e+06 2.906327e+06 2.906327e+06  2.906327e+06mean  6.469163e+00 1.571682e+01 2.006624e+03  5.600492e+01std   3.456489e+00 8.800534e+00 2.338226e+01  3.212359e+01min   1.000000e+00 0.000000e+00 2.000000e+02 -9.900000e+0125%   3.000000e+00 8.000000e+00 2.001000e+03  4.580000e+0150%   6.000000e+00 1.600000e+01 2.007000e+03  6.250000e+0175%   9.000000e+00 2.300000e+01 2.013000e+03  7.550000e+01max   1.200000e+01 3.100000e+01 2.020000e+03  1.100000e+02
Sometimes there are use cases that require that data extracts can be copied and inserted easily, I would recommend using to_markdown() function which provides great results for this purpose —indeed one of my favorites.
|         | Country   |   AvgTemperature ||--------:|:----------|-----------------:|| 2906322 | US        |             28   || 2906323 | US        |             27.6 || 2906324 | US        |             29   || 2906325 | US        |             28.8 || 2906326 | US        |             28.7 |
We further need to check whether there are “not a number” values. As we can see, only State has missing fields, which is totally fine for us. Keep in mind, for doing more sophisticated tasks like machine learning, removing as well as inter- or extrapolating missing data might be a key exercise for the data scientist.
Region            FalseCountry           FalseState              TrueCity              FalseMonth             FalseDay               FalseYear              FalseAvgTemperature    Falsedtype: bool
In order to get the unique values of a column the unique() function is very useful — we might use it on the grouped object as well.
data.Region.unique()array(['Africa', 'Asia', 'Australia/South Pacific', 'Europe',       'Middle East', 'North America',       'South/Central America & Carribean'], dtype=object)
You will likely have a much better feeling for the data, when the data is illustrated or summarized. If your data is numeric, you might find scatter or boxplots very handy.
Aside: There are many different libraries that allow plotting data in a very beautiful and quick manner. From a personal standpoint, data exploration without visualizations is often hard and a lot more time consuming as data irregularities may be identified quite a bit later in the process.
Here is just a quick look at another visualization that immediately provides a feeling for outliers, very dense scatter plot that combines a numeric and a categorical variable:
As we could see in the describe-table we initially created, temperature is given in Fahrenheit, for a non-American resident, it might get a bit more difficult to work with the data — especially when it comes to the “something seems wrong here”-part.
Pandas allows us to perform actions on columns in a very easy and straightforward way.
The more versatile approach is applying a function to the column. The result is the same as above, however, we can be a lot more creative in applying functions through this way.
Apply( ) allows the user to create sophisticated functions and apply them on a single or even several columns. Also note, other than defining a function (as our to_celsius function), lambda expressions can also be used very easily— this is probably most useful when used only once.
If you face complex modification tasks on a more frequent basis, you might want to build functions that are capable of parsing the cell values through regular expressions. In this case you might want to combine regular expressions with pandas’ apply function. If you feel you need a refresher on Regex for data science, take a look at the link below:
towardsdatascience.com
When data is grouped together, Pandas creates a groupby-object. There are plenty of functions that can be applied onto a grouped object like unique(), mean(), min() or max() — just to name a couple. Please note, you cannot directly “show” a groupby-object as this would result in an output like:
<pandas.core.groupby.generic.DataFrameGroupBy object at 0x000001F2B6879CD0>
In order to obtain mean values, the following line will help:
data.groupby([‘Region’,’Country’,’State’,’City’,’Year’]).mean()
Again, mean values do not make any sense for month or days, hence it would be useful to omit these columns. Please also note that the above table would require to reset the index in order to show properly with per-line-entries.
Very often you might only be interested in a small portion of the given data. For this reason, the dataframe can be minimised to a smaller one through a simple filtering operation:
vienna = data[data['City']=='Vienna'].copy() # great city
The created copy of the dataframe will only comprise values where the ‘City’ is ‘Vienna’. If we needed additional filter requirements, pandas allows to concat criteria through ampersand or pipes.
In general, we could use the main dataframe to plot the relevant information, however I prefer using simple plotting operations on smaller datasets. Filtering data in the plot function makes it hard to read and understand. The advantages of using a dedicated copy of a dataframe becomes very clear when we are looking at the next lines of code:
With just 3 lines of code, we are able to produce a line chart including a linear trend:
If we counted the lines of code we used here and the number of data manipulations we made with those, there is no doubt that Pandas is a remarkably powerful tool to manipulate data without the need of extensive documentation reading and practice.
If you wanted to gain a better understanding of what we have covered in this article, I suggest starting with the documentation down below and going through the details that seem relevant for you. Keep in mind, like it is with every learning task, doing is better than (just) reading.
pandas.pydata.org
matplotlib.org
www.w3schools.com
As always, see you next time and stay safe!
Image by Gustav Gullstrand, Thanks!