fabiochiu/medium-articles huggingface dataset:
Trinitario, Criollo or Forastero? A Data Analyst’s way of discovering the Best Chocolate Bar
By: Iva Simon Bubalo
Tags: World Chocolate Day, Matplotlib, Python, Exploratory Data Analysis, Women In Ai

Trinitario, Criollo or Forastero? A Data Analyst’s way of discovering the Best Chocolate Bar

Photo by Egor Lyfar on Unsplash

Can’t decide on which chocolate bar to eat? Tempted to try every chocolate bar like there is no tomorrow? Well, today is your lucky day! We’ll help you with making that very important decision. But first —

Happy World Chocolate Day!!

Commemorated on 7th of July, World Chocolate Day is about celebrating self-indulgence and learning more about the cacao beans that go into making the best chocolate bars around the world. Let us help you make that choice!

World Chocolate Day is said to celebrate the introduction of chocolate in Europe in 1550. Once cacao beans made their way over to Europe they were still used as a drink flavored with sugar and cinnamon.

The very first solid chocolate bar, known as “Dutch cocoa” was created in 1847 in Britain by Joseph Fry. This magical treat came into existence by pressing a mixture of melted cacao butter and sugar into a solid shape.

Are you too confused about cacao and cocoa? The word ‘cocoa’ is adapted from the word ‘cacao’. Cacao beans refer to those which are not roasted or raw beans while cocoa beans are processed or roasted ones.

It wasn’t until 1875 that the world was introduced to the milk chocolate bar by a Swiss chocolatier called Daniel Peter who invented milk chocolate by mixing powdered milk developed by Henri Nestlé with the liquor.This partnership became the basis of the Nestlé Company.

So are you ready for some serious chocolaty bar-charts and box-plots?
Using matplotlib to create a custom seaborn jointplot
By: Edgar Avalos Gauna
Tags: Python3, Matplotlib, Jointplot, Seaborn, Subplot

Photo by Isaac Smith on Unsplash

Matplotlib is a comprehensive and vast library for creating visualizations in Python. It offers many tools to visualize data in a clear manner. Yet, there are some other libraries that will help you to “upgrade” your charts to a new level, letting them to be more awesome from what they already are. One of this libraries is seaborn. Often people use them together on their daily basis analysis. The reason being is that seaborn works on Matplotlib.

One of the coolest plots I have found in seaborn is the jointplot. According to seaborn documentation, this method allows you to draw a plot of two variables with bivariate and univariate graphs. In other words, it creates a scatter plot within a grid which will show you both variable’s distributions as shown as follows:

Plot created By Edgar Avalos Gauna

For the previous plot, heights and weigths data set from Kaggle was used. This dataset is really useful and simple. It comprisses of 10,000 datapoints, each one with 3 different features: Gender, Height, and Weight. There are some interesting analysis that can be performed on this dataset as shown here. Yet, when using the same dataset on the jointplot, it is not possible to see a difference between men and women. For instance, if we want to do a deeper analysis on weights, the jointplot is not as useful as other tools might be. After going through the jointplot documentation, I haven´t found any argument that will help me on this task. Therefore I decided to create my own jointplot from scratch, using Matplotlib subplots as base grid.

First thing is, we need to replicate each plot from the jointplot. Then each plot will be ensemble together using Matplotlib subplots. Finally some tuning aspects will be introduced to make our plot prettier. Let’s start by importing our data:

After importing our data, the next step is to create each plot from the joint plot. In order to do it, each plot must show 2 sets of data, one for men and one for women. For the scatter plot, I used the kdeplot function from seaborn. The code looked like this:

So far so good. The next step is create the plot distribution for height and weight. Let’s start with heights as it will go on top of the scatter plot:

Similarly, we will create weight’s distribution. Altough, for this plot, it is importand to rotate it 90 degrees to be similar to what we get from the joint plot. This can be done really easy with the addition of the vertical argument to the code of the distplot:

Finally, let us use the subplots function from Matplotlib to create a 2 by 2 grid. Additionaly, the off option will allow us to remove the upper right plot axis:

Now let´s put them all together. I added some other options so the final result looks much nicer.

And that’s it. I hope this will help you to improve your plots and your seaborn skills.
DATA CLEANING & DEALING WITH OUTLIERS USING DATA IMPUTATION TECHNIQUES
By: Kiran Lakhani
Tags: Machine Learning, Outliers, Data Cleaning, Data Science, Matplotlib

→ What do these NaN values indicate ??

They are the missing values in the data set.

We can handle them in two ways:

1, By eliminating the rows of missing values. (Generally, not recommended as it might reduce the data set to some extent leading to less data to be trained)

2, By using an Imputer to find the best possible substitute to replace missing values

Count of total number of NaN values

→ In order to know the total no. of NaN values we use lambda function.

→ Syntax: data1.apply(lambda x:sum(x.is.null()),axis=0)

→ isnull()- checks whether the value is null or not.

→ axis=0- selects all the columns & axis=1 checks all the rows as shown in the figure.

Any other value given to axis results into an error as shown

→ Let us now start to fill these values by using Imputer function present in the sklearn.preprocessing module, as shown in fig: 1.By writing the above line the missing values are replaced by the mean of all the other values present in the data set. Now, we need to convert our data set (i.e data 1) into array in order to perform operations on it.

fig. 1

→ Syntax: newdataframe=olddataframe.iloc[:,:].values

→ Syntax: imputer_variable.fit(data2[:,[0,1]])

data2[:,[0,1]]=imputer_variable(data2[:,[0,1]])

→ display data2

→[:,:] indicates all the rows and all the columns respectively.

→ As shown in the fig. the data is converted into arrays in order to perform operations on it as array is part of numpy library.

→ Here, we are trying to fit all the rows of 0th & 1st index columns into the imputer, so that the NaN values get replaced by the strategy used.

fig. mean_imputation

→ Now, as we see in the fig: ‘mean_imputation’ we find that the voids (NaN) are successfully filled by the imputer.

→ It is just an assumption that placing mean values in place of missing values. However, we need to test other strategies as well to choose the best.

Now, let us go for median strategy as shown below: (syntax of imputer remains same)

Median strategy

most_frequent/ mode stratergy

Value of Median is 31.5 and value of most_frequent is 23

Now after all these we need to choose which strategy suits our algorithm. We can find it out using the box plot as shown below:

→This plotting is before removing outliers.

→ Outliers are the values which exceed the range (or) it is also referred to as out of bound data

(as we have seen this in our previous blog)

→ Here we find that our output ozone is mostly effected by solar → Plotting the graphs of all the three imputation techniques with outliers for future comparison.

→ The code says plotting Ozone on y-axis & Solar on x-axis

→ The axis line coinciding the blue dots (data points) is called the axhline. The ‘r’ indicates the line to be red colour and the we are viewing the line in the linear form (-)

→ Finally, displaying the graph → The same procedure is followed for all the imputers as shown below for median, mode.
Illustration with Python: Weak Law of Large Numbers
By: Chaya Chaipitakporn
Tags: Probability, Data Science, Matplotlib, Python

The weak law of large numbers states that with a sufficiently larger number of trials there is a very high probability that the mean of observation will be close to an expected value. In other words, as the number of trials goes to infinity the mean of observation convergers to an expected value with a very high probability.

Where Xn is a mean of n trials, μ is an expected value, ε is a margin of error which is greater than 0. The full detail of the theorem can be found in this link.

Before I show the code, there is one thing I want to point out, even though the theorem said about a number of trials, we can use this theorem with sample size if each sample is independent and identically distributed. For example, suppose we are interested in a number of right-handed people and we want to study 100 samples. If we go to a mall and pick one person at a time, ask and repeat 100 times (100 trials), the expectation will be the same as picking 100 people and ask at once. Because each sample has the same expected value, so we sum the expected values and divided by the number of sample and we will get the same expected values. I will use python code to show the theorem as the following steps

Step:

1.) Create a gamma distribution with shape = 2 and scale = 2 as a population.

shape, scale = 2., 2. # mean=4, std=2*sqrt(2)

s = np.random.gamma(shape, scale, 1000000)

2.) Set a sample size to 100 at first, sample 50 times and collect a mean of each time, then increase the sample size by 500, repeat the step until the sample size reach 8100.

samplemeanlist = [] # list of sample mean

l = [] # list of smaple size, for x-axis of box plots

numberofsample = 50 # number of sample in each sample size



# set sample size (i) between 100 to 8100, step by 500

for i in range(100,8101,500):

# set x-axis

l.append(i)

# list of mean of each sample

ml = []

# sample 50 time.

for n in range(0,numberofsample):

# random pick from population with sample size = i

rs = random.choices(s, k=i)

# calculate the mean of each sample and save it in list of mean.

ml.append(sum(rs)/i)



# save the 50 sample mean in samplemeanlist for box plots.

samplemeanlist.append(ml)

3.) Plot a boxplot of each sample size
Illustration with Python: Confidence Interval
By: Chaya Chaipitakporn
Tags: Confidence Interval, Probability, Data Science, Matplotlib, Python

This article uses knowledge in the central limit theorem, and also the concept of the weak law of large numbers and Chebyshev’s inequality, you can review these topics by visiting the links. If you would like to follow along, you can get the code from this link: Jupyter Notebook.

In my opinion, this topic is the most confusing theorem compare with others, even the article in Wikipedia mentions the misunderstanding about it, so I will try my best to explain it.

Before diving in, keep in mind that the mean of the population (the thing we what to estimate) is a constant, there is no randomness about the number.

The confidence interval is an estimator we use to estimate the value of population parameters. The interval will create a range that might contain the values. When we create the interval, we use a sample mean. Recall the central limit theorem, if we sample many times, the sample mean will be normally distributed.

I create the sample mean distribution to demonstrate this estimator.
Animating a 3D scatterplot with Matplotlib
By: Pierre Segonne
Tags: Animation, Scatter Plots, Matplotlib, Python

Evolution of a swarm in 3D

Python, together with Matplotlib allow for easy and powerful data visualisation. It was originally developed for 2D plots, but was later improved to allow for 3D plotting. Furthermore, an animation module also allows for dynamical plotting. That is our goal today: Animate a scatter plot in 3D with Matplotlib.

Generate Data

Let’s first generate some dummy data. We want our data-structure to consist of a list of arrays of positions of our animated points (also called elements). Each element of the list being the positions at a different iteration.

Data Generation for Our Simulation

Here, as can be seen in the gist above, a simple strategy is adopted. An initial position is randomly sampled, for each element, from a normal distribution. Similarly, a speed is sampled from, for each element, from a normal distribution. At each iteration, the position is updated as below:

P(k) = P(k-1) + S

Where P(k) is the position at iteration k, P(k-1) at iteration k-1 and S the speed.

Animation Function

The animation of the Matplotlib figure requires an animation function as explained in the doc. This function’s purpose is to update the data of the different plots contained in the figure.

Here again it is pretty simple, for a given iteration, the value shown by the scatters (There is one scatter per point) is updated with the associated coordinates.

Please note the use of data[iteration][i,0:1] and not data[iteration][i,0]. This is because the scatter function of Matplotlib expects an object with a len, and therefore not a number.

Please also note the use of _offsets3d. It is a private and is a workaround found here.

Main Function

Finally, a main function, which regroups the creation of the figure, some utilities and most importantly the call to animate the figure, is required.

Please remark that it allows to save the animated scatter plot under a mp4 file. You might experience troubles when trying to save. FFMEG is indeed required. I was able to solve this issue, on OS X, by running in the terminal:

brew install yasm

brew install ffmpeg

pip install ffmpeg-python

Full Script

The full running script is accessible with the following gist:

Other Works

I’d like to give a shoutout to two very nice tutorials about animation with Matplotlib.

And to the official example from matplotlib, which is very instructive on the general animation Mechanism.
Illustration with Python: Chebyshev’s Inequality
By: Chaya Chaipitakporn
Tags: Probability, Data Science, Matplotlib, Python

Chebyshev’s inequality is an extremely useful theorem when combining with other theorem and it is a bedrock of confidence interval. In this blog, I will illustrate the theorem and how it works using python.

Before we start coding, let’s look at the theorem. The Chebyshev’s inequality states that, for wide class of probability distribution, no more than a certain amount of values can be more than a certain distance from the mean, with the formula as follow.

Chebyshev’s inequality

Where X is a random variable, μ is an expected value of X, σ is a standard deviation of X and k > 0.

For example, the probability that a distance from an expected value is far more than 3 standard deviation is less or equal than one over nine.

Chebyshev’s inequality with k = 3

According to the formula, if k increases, the probability will decrease. I will illustrate the theorem using python, but I will not use to formula, instead, I will use python to create a population and show how the theorem works.

Step:

1.) Create a population of 1,000,000 values, I use a gamma distribution with shape = 2 and scale = 2, but the theorem works with other distribution as well, such as uniform distribution.

shape, scale = 2., 2. # mean=4, std=2*sqrt(2)

mu = shape*scale

sigma = scale*np.sqrt(shape)

s = np.random.gamma(shape, scale, 1000000)

2.) Sample 10,000 values from the population

# sample 10000 sample

rs = random.choices(s, k=10000)

3.) Count the sample that has a distance from the expected value larger than k standard deviation and use the count to calculate the probabilities. I want to depict a trend of probabilities when k is increasing, so I use a range of k from 0.1 to 3.

# set k

ks = [0.1,0.5,1.0,1.5,2.0,2.5,3.0]

# prob list

probs = [] # for each k

for k in ks:

# start count

c = 0

# for each data sample

for i in rs:

# count if far from mean in k standard deviation

if abs(i - mu) > k * sigma :

c += 1

# count divided by number of sample

probs.append(c/10000)

4.) Plot and print the result.
Bar Chart Race?. My curiosity starts with this type of…
By: Zoe Z
Tags: Data Visualization, Matplotlib

My curiosity starts with this type of “Bar Chart Race” that is so popular on reddit/datasibeautiful, youtube/wawamustats.

There is an even easier way that you can input your parameters and make your own bar chart race. https://app.flourish.studio/@flourish/bar-chart-race. It looks like this:

There are so many models and methods that data scientists love to apply, but when it comes to explaining to people with various levels of technical skills. Graphs grasp the attention and make a better narrative.

To practice and show an example, I get the dataset from FiveThirtyEight github repo: https://github.com/fivethirtyeight/data/ or if you are a R user, there is a direct package that you can install and have fun with. My practice dataset is a dataset on: https://github.com/fivethirtyeight/data/tree/master/endorsements-june-30. The dataset is about the endorsement data before the primary election. It included the data indicating the endorsement points on the candidates before June 30 from 2000- 2012.

If you are interested in how I get to graph a racing bar graph, you can read my notes on this tutorial on how to start with a simple matplotlib animation using the matplotlib.animation module.

Let’s step through this and see what’s going on. After importing the required pieces of modules. An easy way to get a FiveThirtyEight style is to simply use style.use(). Matplotlib also has an animation module that can do most of the work for you.

Step 1: Import libraries

Let use try to graph the first bar chart that will show up at the end of the animation, which is the last frame the audience should see.

Step 2: Layout basic bar chart

Step 3: Add colors and group data

I format the color using the python color constants module:

Step 4: Put them into a function

Step 5: Using animation.FucAnimation Function to finalize the picture

You can also exported as gif or mov file. Simply change the speed parameter fps(frequency per second) and dpi to adjust the resolution, in the .save function.
Ridge Plots with Python’s Seaborn
By: Thiago Carvalho
Tags: Data Visualization, Matplotlib, Python, Seaborn, Ridgeline Plot

KDE Plots and Facet Grids

Now let’s get to the example. We’ll use this dataset from Kaggle, which contains Netflix's original productions and their IMDB scores.

import matplotlib.pyplot as plt

import seaborn as sns

import pandas as pd df = pd.read_csv('../data/netflixoriginals.csv') languages = ['English', 'Hindi', 'Spanish',

'French', 'Italian', 'Portuguese'] df_filtered = df[df['Language'].isin(languages)] df_filtered

Filtered data frame — Image by the author

We’ll need Seaborns FacetGrid to create a plot for each category systematically. The function is straightforward; we only need the data frame and the field's name to group the values.

sns.set_theme(style="white") g = sns.FacetGrid(df_filtered, row="Language") g.map_dataframe(sns.kdeplot, x="IMDB Score")

KDE plots — Image by the author

With this default configuration, it’s hard to see and compare all the distributions. One of the main advantages of Ridge plots is to make the chart compact while still informative.

Of course, there are many different solutions for this issue, using the columns, changing plot sizes, or using another visualization.

Let’s try making the charts broader and shorter.

sns.set_theme(style="white") g = sns.FacetGrid(df_filtered, row="Language", aspect=9, height=1.2) g.map_dataframe(sns.kdeplot, x="IMDB Score")

KDE plots — Image by the author

That solves the problem, but there isn’t anything special about this visualization. The simple fact that Ridge plots are unconventional makes them more attractive.

Kernel Density Estimation plots are not known for representing the data with precision; they’re great for simple tasks such as visualizing the modality or central tendency.

That means we want the user to focus on the shape of the data, and the Y-ticks aren’t needed here.

sns.set_theme(style="white") g = sns.FacetGrid(df_filtered, row="Language", aspect=9, height=1.2) g.map_dataframe(sns.kdeplot, x="IMDB Score") g.set_titles("")

g.set(yticks=[])

g.despine(left=True)

KDE plots — Image by the author

We’re ready to turn our density plots into a Ridge plot.
A year of spending
By: Annie Zhang
Tags: Data Visualization, Homework, Matplotlib

Overall breakdown — Pie Chart

The pie chart provides a snapshot of the situation at a given point in time. In this case, that “point in time” is today, where I take stock of my overall expenditure for the past year. From the chart below, I have spent the most on Food & Drink (43%), Shopping (15%), and Transport (13%). I am a student who lives with my parents. Hence, a large portion of my expenses is being taken care of by my parents which include my phone bills, utility bills, my hostel fees, and most meals on weekends. With these out of the equation, my expenditure consists mostly of daily expenses.

Time trend — Line Graph

The line graph reveals trends. In this case, I use the line graph to show a time trend — how each category of expenses changed over the year.

In May and August, my shopping expenses (orange) increase as I buy officewear for my internships. Shopping expenses were also high during the Christmas period (Nov — Jan).

In August 2018, entertainment expenses (yellow) rose sharply because I watched a concert by one of my favourite mandopop artists JJ Lin.

In August 2018 and March 2019, healthcare expenses (light blue) rose sharply as I went for my semiannual dental appointments.

In March, I also signed up for a python class outside of school which explains the increased education expenses (brown).

Most frequent buys — Wordcloud

A word cloud, in this case, shows me my most frequent purchases. I record the transaction information under “notes” so there is clarity on what I am paying for. The word cloud of my transactions, generated with https://www.wordclouds.com/, looks like this:

Clearly, I eat a lot of chicken, fish, and drink a lot of bubble tea (see the huge Gongcha and Koi). It seems that my diet isn’t particularly healthy either…
Getting back into NumPy, Day 10
By: Pär Persson Mattsson
Tags: Numerical Methods, Numpy, Matplotlib, Python Programming, Python

I’m so ready to throw this problem away and start working on something else. Another day of debugging and finding the error, and right now I don’t know if the error is with me not remembering the algorithm properly, or that something is wrong with my operators.

Currently, the algorithm gets stuck after 1 iteration. It takes the start guess, applies the projection onto the first set, onto the second set, and then it doesn’t change. The error between the true image and my guess stays the same, and nothing seems to happen. I need to take a look in my collection of papers on this matter before this drives me crazy.

One other thing that makes me confused is that the phase of the FFTs are very blocky. I’m not sure where this comes from, and I don’t know how much this influences the result. It shouldn’t create too much trouble (I think), so I don’t know how important this is for the convergence.

Well, I’ve learned a lot, though, even if the progress was minimal. Fancy indexing is very logical, and I’m starting to get the hang of matrix handling in NumPy.

And to keep today’s post from being horribly boring, here’s a picture showing the current state of my program. I have to say, the guess seems to be closer than the guess I was able to produce yesterday… :-)
Using Chinese Characters in Matplotlib
By: Kelvin Ng
Tags: Unicode, Matplotlib

When you try to display Chinese or other non-ascii characters in matplotlib, your characters may not be displayed properly, like following figure:

Error displaying Chinese characters in matplotlib

It is because the fonts used by matplotlib couldn’t decode the characters properly. To solve it, we should add the appropriate fonts and update matplotlib font cache.

locate the matplotlib fonts folder:

import matplotlib

print(matplotlib.matplotlib_fname())

this is the location of matplotlib config file, you will get something like …/matplotlib/mpl-data/matplotlibrc

The font folder is …/matplotlib/mpl-data/fonts/ttf , put your ttf file there.

2. Get ttf from ttc file (skip if you have ttf file already)

For macOS, the system Chinese fonts is Heiti, which is embed in a ttc file (ttc is a collection of multiple ttf files). Get the system ttc file in /System/Library/Fonts/STHeiti Medium.ttc Copy this file out and convert it to ttf. Here is an online ttc converter:

https://transfonter.org/ttc-unpack

3. Rebuild the Matplotlib Cache

import matplotlib.font_manager

matplotlib.font_manager._rebuild()

restart your Jupyter / ipython kernel, then test if matplotlib can load your font or not by the ttflist function of the font manager

[f for f in matplotlib.font_manager.fontManager.ttflist if 'Heiti' in f.name]

change ‘Heiti’ to your own font name. If you see your fonts object above. You are ready to use the new fonts.

4. Using Appropriate Fonts in Matplotlib

matplotlib.rcParams['font.family'] = ['Heiti TC']

This will tell matplotlib to use the specific font as default, result as follow~
Saving Python Data Visualizations for Presentations
By: Sharon Kwak
Tags: Data Visualization, Data Science, Matplotlib, Dataframes, Seaborn

Saving without any parts cutting off

After you save your plot, make sure it shows all of the parts. Even if you see a complete plot when you run the code, the .png file might not show all of the parts. You wouldn’t want to display an incomplete graph like the one below.

To prevent this from happening, you can change the formatting of the plot and figure space through different methods.

One method is to change the figure size, increasing the width and/or length to include the part that was cut off (default figure size is 6.8, 4.8).

plt.figure(figsize=(10, 4.8))

The method below keeps the original figure size but shifts the positioning and changes the size of the plot to fit everything in the original figure space.

plt.subplots_adjust(left=0.2)

You can change the number and the direction (left, right, bottom, top) depending on the amount of space your graph needs (the number represents the distance of the plot from the boundary of the figure).

The above two methods involve testing out different numbers, whether it be the size of the figure space or the margin distance. The following options automatically adjust the positioning of the plot to fit within the figure space, default or assigned, without extra whitespace on the edges.

#changes the layout first before saving the plot plt.tight_layout()

plt.savefig("test.png")

or
How to Pick the Optimal Image Size for Training Convolution Neural Network?
By: Aravind Ramalingam
Tags: Machine Learning, Visualization, Computer Vision, Matplotlib, Python

How to Pick the Optimal Image Size for Training Convolution Neural Network?

When training a Convolution Neural Network on a custom dataset, picking the right image is crucial. This will impact the training time & performance of the model. Also, we will learn how to identify if there are any issues with the dataset. Aravind Ramalingam Follow Jun 23 · 3 min read

Photo by Szabolcs Varnai on Unsplash

Why?

Before we jump to how part, Let us discuss the negative consequence of choosing the wrong size. After we pick a fixed width and height, the standard procedure is to resize all the images to this fixed size. So, now every image falls into one of the two buckets.

Downscaling: Bigger images will be down scaled, this makes it harder for CNN to learn the features required for classification or detection as the number of pixels where the vital feature will be present is significantly reduced.

Upscaling: When small images are upscaled and padded with zero, then NN has to learn that the padded portion has no impact on classification. Larger images are also slower to train and might require more VRAM.

So we have to pick our poison, the closer to optimal image size we are, the better it is.

Optimal Image Size

We all know that choosing the right size depends on the dataset, but the question is how to do it? Visualize the image size.

Dataset: The Oxford-IIIT Pet Dataset

Image Meta Data

This dataset has more than 7000 images with varying size and resolution.
Data Analysis — Only 20 % of IPL (Indian Premier League) matches are thrillers
By: Ashay Maheshwari
Tags: Pandas, Jovian Ml, Data Science, Matplotlib, Python

Overview

Data Science is new and yet growing older. With days passing by, the skill is more and more in demand and there are various studies and reports available on search engines to support the statement. Still, the skill is niche and the demand is much more than the supply of various roles related to data, which is Data Engineers, Data Analysts, Data Scientists, etc.

As a part of this post, we will use an opensource dataset of IPL matches available on Kaggle — A superb community of Machine Learning and Data Science enthusiasts .

Who should read this post ?

This post could benefit following individuals -

individuals interesting in the field of Data Science and would like to see a working case of analyzing data

individuals who know Python programming, Pandas library and would like to see some interesting cases to try doing simple data analysis activities

active readers reading technology every time

individuals looking for a superb source to learn basics of Data Science using Python and Pandas library

Acknowledgement

I would like to acknowledge and appreciate the great work done by Jovian team in putting up some interesting and interactive courses on Data Science and Machine Learning and many of them are free of cost with a certificate you can earn after completion of course assignments. I enrolled to a free course named Data Analysis with Python: Zero to Pandas which is a 60 hour course for beginners and I recommend it. This post has one of assignment which is also posted on my jovian profile and links to which will be shared in references section of this post.

Let’s figure out what we need to answer

Before we try to pick a dataset and jump into analysis, we need to get a clarity on What are we trying to achieve out of the analysis ?, What are the questions we are trying to answer out of the analysis and many other business critical decisions we might want to make based on the analysis. Let us list down a few of our questions we need an answer for —

Which city hosted the maximum number of IPL Matches so far ?

Which team played the maximum number of IPL Matches so far ?

Out of all the matches played, how many matches were tied and what is the probability of an upcoming match getting tied ?

Out of all the matches played, how many matches were abandoned and what is the probability of an upcoming match getting abandoned ?

How many matches were won batting first compared to batting second ?

What is the probability of getting a close match where a close match can be defined as a team winning a match by less than 10 runs of margin ?

The above given questions are just basic one and there is certainly no limit to what can be achieved and concluded out of data. For some more realistic scenario, one might want to predict the performance of a player in upcoming IPL seasons or the performance of a team or predict if a team should bat first or second given the venue and the opponent. We will not go to that level and will keep it simple.
AISaturdaysOgbomoso Cohort 2, WEEK 3: Matplotlib.
By: Lautech Datascience
Tags: Machine Learning, Artificial Intelligence, Data Science, Matplotlib, Python

Matplotlib is the “grandfather” library of data visualization with Python. It was created by John Hunter. He created it to try to replicate MatLab’s (another programming language) plotting capabilities in Python. So if you happen to be familiar with matlab, matplotlib will feel natural to you.

It is an excellent 2D and 3D graphics library for generating scientific figures.

Some of the major Pros of Matplotlib are:

Generally easy to get started for simple plots.

Support for custom labels and texts.

Great control of every element in a figure.

High-quality output in many formats.

Very customizable in general.

Matplotlib allows you to create reproducible figures programmatically. Let’s learn how to use it! Before you continue reading, I encourage you just to explore the official Matplotlib web page: http://matplotlib.org/

The class was really fun at first since we were still working with just plotting a simple line plots.

We all started with making sure we had matplotlib on our workspace by importing pyplot function in matplotlib.

After importing and confirming that matplotlib was installed since almost everyone installed anaconda which has the matplotlib library out of the box, we moved on to creating our first visualization, which was a line graph.

In the Command above we explained the use of %matplotlib inline and command plt.show(). Then we checked into Subplot: plotting more than one plot on a canvas and we didn’t make use of the plt.show() because of the %matplotlib inline which helps to print the plot without the plt.show() function in Jupyter Notebook.

Then we introduced ourselves to figure function( plt.figure() ) which helps to define the size of the canvas and how to position the plots in different templates.

Below is an example of 3 plots on the canvas with different size and shapes(with the use of axes).

We further talked about changing the colours of our graphs, the plot types, setting Legends and also label title and above all how to save our figures for use in presentation and visualization.

If we must confess though, it was a bit challenging but a good way to learn is by facing up to challenges😄. They were given another notebook for personal studies and from the feedback we got, some of them dug further into visualization 😍.

Thanks to our ambassador 0basa Samuel temitope for writing this and Daniel Ajisafe and Tejumade Afonja for guiding us.

AISaturdayOgbomoso wouldn’t have happened without our fellow ambassadors and coaches, our partners Intel .

follow us on https://twitter.com/LautechDataSci
Plotting the Batman Equation in Python using Numpy and Matplotlib
By: Subarno Pal
Tags: Python3, Data Visualization, Batman, Matplotlib, Python

Batman has been my favorite superhero since childhood. Plotting the Batman equation was quite fun. In this tutorial blog we will be showing how to plot the Batman equation from scratch with NumPy and Matplotlib. The only prerequisite is minimum introduction to basic python. If you are expert in python and matplotlib you are advised to skip to the final part.

We will be describing from very basic matplotlib scatter plot. First of all we will show how to plot some basics points in matplotlib using scatterplot. The code shown below plots 4 co-ordinates (1,1), (2,4), (3,9) and (4,16) in their respective x and y co-ordinates using scatter plot.

import matplotlib.pyplot as plt

import numpy as np x = [1, 2, 3 , 4]

y = [1, 4, 9, 16]

plt.plot(x,y, 'yo')

plt.show()

By plotting a series of points in a curve we can represent the function of the curve.

Y = np.arange(-10,10,.05)

X = np.zeros((0))

for y in Y:

X = np.append(X,5*y**3+50)

plt.plot(Y,X, 'yo')

plt.show()

Two different functions can be combined into a single plot just by appending those points and plotting them together.

Y1 = np.arange(-10,10,.05)

X1 = np.zeros((0))

for y in Y1:

X1 = np.append(X1,-5*y**2)



Y2 = np.arange(-10,10,.05)

X2 = np.zeros((0))

for y in Y2:

X2 = np.append(X2,(5*y**2))



X = np.append(X1,X2)

Y = np.append(Y1,Y2)

plt.plot(Y,X, 'yo')

plt.show()

All we need to do is combine 7 different curves into a plot to demonstrate Batman’s equation. The curves has been taken from this link.

import math Y = np.arange(-4,4,.005)

X = np.zeros((0))

for y in Y:

X = np.append(X,abs(y/2)- 0.09137*y**2 + math.sqrt(1-(abs(abs(y)-2)-1)**2) -3)



Y1 = np.append(np.arange(-7,-3,.01), np.arange(3,7,.01))

X1 = np.zeros((0))

for y in Y1:

X1 = np.append(X1, 3*math.sqrt(-(y/7)**2+1))

X = np.append(X,X1)

Y = np.append(Y, Y1) Y1 = np.append(np.arange(-7.,-4,.01), np.arange(4,7.01,.01))

X1 = np.zeros((0))

for y in Y1:

X1 = np.append(X1, -3*math.sqrt(-(y/7)**2+1))

X = np.append(X,X1)

Y = np.append(Y, Y1) Y1 = np.append(np.arange(-1,-.8,.01), np.arange(.8, 1,.01))

X1 = np.zeros((0))

for y in Y1:

X1 = np.append(X1, 9-8*abs(y))

X = np.append(X,X1)

Y = np.append(Y, Y1) Y1 = np.arange(-.5,.5,.05)

X1 = np.zeros((0))

for y in Y1:

X1 = np.append(X1,2)

X = np.append(X,X1)

Y = np.append(Y, Y1) Y1 = np.append(np.arange(-2.9,-1,.01), np.arange(1, 2.9,.01))

X1 = np.zeros((0))

for y in Y1:

X1 = np.append(X1, 1.5 - .5*abs(y) - 1.89736*(math.sqrt(3-y**2+2*abs(y))-2) )

X = np.append(X,X1)

Y = np.append(Y, Y1) Y1 = np.append(np.arange(-.7,-.45,.01), np.arange(.45, .7,.01))

X1 = np.zeros((0))

for y in Y1:

X1 = np.append(X1, 3*abs(y)+.75)

X = np.append(X,X1)

Y = np.append(Y, Y1) plt.plot(Y,X, 'yo')

plt.grid()

plt.show()

Thus we can see the Batman equation has been plotted. Finally some graphical modifications.

ax = plt.gca()

ax.set_facecolor((0, 0, 0))

plt.plot(Y,X, 'yo')

ax.set_yticklabels([])

ax.set_xticklabels([])

plt.show()

The complete code is available in this link.
How to Use Colormaps with Matplotlib to Create Colorful Plots in Python
By: Elizabeth Ter Sahakyan
Tags: Pandas, Programming, Data Science, Matplotlib, Python

How to Use Colormaps with Matplotlib to Create Colorful Plots in Python

Overview

Data scientists are visual storytellers, and to bring these stories to life, color plays an important role in accomplishing that. Data can be easily visualized using the popular Python library matplotlib. Matplotlib is a 2D visualization tool that allows one to create scatterplots, bar charts, histograms, and so much more. Matplotlib works very well with pandas, another popular library in Python used for data analysis. Pandas is very useful for structuring data to be then plotted with matplotlib.

Once you have a plot created with these tools, you can easily bring them to life with colors using the predefined colormaps—sets of RGBA colors that are built into matplotlib. You can even create your own matplotlib colormaps!

An important application of matplotlib colormaps is using it to make your work more accessible for people with color vision deficiencies. The most common color vision deficiency is the inability to distinguish between red and green, so avoiding placing those colors together in general is a good idea.

In this article, I will show you how to transform your charts into something more visually appealing, and accessible, using matplotlib colormaps.
Histogram Plot with Math and Python code For Data Visualization
By: Md Jahid Hasan
Tags: Data Visualization, Pandas, Matplotlib, Plot, Histograms

In this article, I will discuss the Histogram plot for data visualization. Now going to simple statistics math for more details.

Suppose, your family member is 20 and their ages are different. In this section, we will randomly generate 20 member’s age by python code.

>>> import random

>>> age =[]

>>> for x in range(20):

age.append(random.randint(1,80))

>>> print(age)

[77, 46, 5, 73, 55, 48, 32, 7, 80, 69, 40, 51, 13, 77, 44, 49, 39, 11, 72, 33]

To develop a histogram from a continuous variable you first need to split the data into intervals, that’s called bins. Age has been split into bins, with each split represent to a 10 year period starting at 20 years.

Make a Histogram Plot for age’s data.

Visualizing Histograms with Matplotlib and Pandas

Now that you’ve seen how to build a histogram in Python. Matplotlib provides the functionality to visualize Python histograms out of the box.

# Matplotlib

>>> import matplotlib.pyplot as plt

>>> age = [77, 46, 5, 73, 55, 48, 32, 7, 80, 69, 40, 51, 13, 77, 44, 49, 39, 11, 72, 33]

>>> plt.hist(age, bins=[1,11,21,31,41,51,61,71,80])

>>> plt.show()
DOM from Python, Example Plots, Testing Infrastructure
By: Madhur Tandon
Tags: JavaScript, Gsoc 2019, Matplotlib, Python, Mozilla

Testing Infrastructure

Well, we have been successful in implementing a new canvas based renderer for the matplotlib library (not everything is supported yet, but again, we have just begun, haven’t we?) which can be used on the web — matplotlib now runs in natively in the browser folks! **pats my own back**

Except that there is one thing we need to take care of. **Oh god, let this not be testing**, but it really is testing :(

It turns out that visually inspecting each and every plot is time consuming and inefficient. We need a testing infrastructure in place that can automatically test if the plots we are generating are indeed the plots we SHOULD be generating.

Pyodide does most of its testing using selenium — which makes sense since the whole project is related to running Python within the browser. This gives us a hint that the testing infrastructure should also be using selenium in some sense. In a nutshell, this is the approach (thanks Mike! for all the guidance) that should be adequate to set this up:

1. Run the Firefox Web-driver or the Chrome Web-driver in a headless state (without GUI). Make the selenium based browser to load a simple HTML file which also contains the script to load the local copy of Pyodide. 2. Plot a sample figure using matplotlib (with our own custom <canvas> based renderer) from Pyodide. Once the plot has been generated, that means an actual <canvas> tag now exists within the DOM. 3. Save the content of the <canvas> tag in a PNG file. This file is our reference image which we can use for a visual inspection. 4. Get the underlying pixel data from the <canvas> element, this can be saved in the form of a numpy array. These are raw pixel values which cannot be used for visual inspection. 5. Make a request to get the underlying data of the reference PNG image we stored earlier. This can be done using XMLHttpRequest and within the same Python script (thanks to Pyodide!) 6. Compare the data obtained from the PNG vs the data directly obtained from the <canvas> tag. These should be quite similar i.e. if the mean of absolute deviations for all pixels is within a certain threshold, we say that the 2 images are similar. Further, saving of the PNGs allows an external user to also verify the plots visually.

That wraps up the part for testing (which still isn’t merged by the way). But, at least we could figure out a way to automate this stuff.
Handy Data Visualization Functions in matplotlib & Seaborn to Speed Up Your EDA
By: Manu Kalia
Tags: Data Visualization, Eda, Data Science, Matplotlib, Python

For ease of demonstration, I’ll start out by fetching one of the datasets available via Sklearn, the California housing prices dataset. This happens to be a dataset that lends itself to a regression type problem, but the graphing functions I describe below work just as well for categoricals and classification-oriented exercises, either as they are or with minor modifications. If and when you use the plotting functions, you should, of course, use the dataframe that you have set up and ignore this particular dataset.

First, the basic imports:

fig. 1 … basic imports

Next, fetch the demonstration dataset and stick it in a pandas dataframe. There are 20,648 observations in this Sklearn dataset of California housing prices. The eight feature columns reside in the dataset.data structure.

fig. 2 … fetch dataset

Finally, create the target column price (prices in $000,000) which resides in the dataset.target structure. It happens to be placed into a multilevel column index, so I will need to flatten that to one level, as below. I also verified that there were no null values, just to be sure.
Python 資料視覺化筆記(三) — 使用Matplotlib繪圖
By: Yanwei Liu
Tags: Machine Learning, Visualization, Data Visualization, Matplotlib, Python

初始化

import pandas as pd

import numpy as np

import matplotlib.pyplot as plt

%matplotlib inline

%matplotlib notebook #在notebook裡與Matplotlib互動

圖片顯示中文

from pylab import mpl

mpl.rcParams['font.sans-serif'] = ['Microsoft YaHei']

mpl.rcParams['axes.unicode_minus'] = False

繪圖

plt.plot(x,y)

plt.scatter(x,y)

plt.bar(x,y)

plt.barh(x,y)

plt.hist(x,y)

plt.subplot(列,行,圖形編號)

plt.axhline(y) #水平線

plt.axvline(x) #垂直線

plt.fill_between #陰影

plt.hlines() #水平線圖表

plt.vlines() #垂直線圖表 #散佈圖

plt.scatter(X, Y, s=75, c=T, alpha=.5)

#輸入X和Y作為location，size=75，顏色為T，透明度alpha為50%

#柱狀圖

plt.bar(X, Y, facecolor='#9999ff', edgecolor='white')

#facecolor設置主體顏色，edgecolor設置邊框顏色為白色

#直方圖

plt.hist(X, Y)

#水平線

plt.axhline(y = avg_pts)

#垂直線

plt.axvline( y = avg_pts )

#水平線圖表

plt.hlines(y,xmin,xmax) #陰影

plt.fill_between(pp_stats.index, avg_pts, pp_stats["pts"],

where=pp_stats["pts"] >= avg_pts, color="gray",

alpha=0.5, interpolate=True) #一張圖繪製多個數據集

x = np.linspace(0, 2 * np.pi, 50)

plt.plot(x, np.sin(x),

x, np.sin(2 * x))

#一個視窗多張圖(本例為4張)

x = np.linspace(0, 10, 100)

plt.subplot(2,2,1) #plt.subplot(總行數,總列數,活躍區)

plt.plot(x, np.sin(x), 'r')

plt.subplot(2,2,2)

plt.plot(x, np.cos(x), 'g')

plt.subplot(2,2,3)

plt.plot(x, np.tan(x), 'b')

plt.subplot(2,2,4)

plt.plot(x, np.tan(x), 'n')

設定圖名、標題、註解、圖例、節點

plt.xlabel(‘XXX’) #X軸名稱

plt.ylabel(‘YYY’) #Y軸名稱

plt.title(‘TTT’) #標題

plt.subtitle('AAA') #副標題

plt.xlim((1,100)) #X軸範圍

plt.ylim((1,100)) #y軸範圍

plt.xticks([1,2,]) #X軸刻度

plt.xticks([1,2,]) #Y軸刻度

plt.legend() #顯示圖例

plt.plot(x, y, marker='o') #顯示節點 #針對特定數值進行註解

plt.annotate('Toyota Corolla', xy=(19,33.9), xytext = (21,35),

arrowprops=dict(facecolor='black', shrink=0.05)) #添加注釋文字

plt.text(-3.7, 3, r'$This\ is\ the\ some\ text. \mu\ \sigma_i\ \alpha_t$',

fontdict={'size': 16, 'color': 'r'})

# 其中(-3.7,3)是選取text的位置, 空格需要用到字符\ ,fontdict設置文本字體. #設置Y軸刻度及名稱

plt.yticks([-2, -1.8, -1, 1.22, 3],[r'$really\ bad$', r'$bad$', r'$normal$', r'$good$', r'$really\ good$']) #使用plt.yticks設置y軸刻度以及名稱：刻度為[-2, -1.8, -1, 1.22, 3]；對應刻度的名稱為[‘really bad’,’bad’,’normal’,’good’, ‘really good’] #設置圖例

l1, = plt.plot(x, y1, label='linear line')

l2, = plt.plot(x, y2, color='red', linewidth=1.0, linestyle='--', label='square line')

plt.legend(loc='upper right')

#legend顯示的訊息來自於上面代碼中的label

#參數'upper right'表示圖例將添加在圖中的右上角 #移除上方與右方邊框

plt.gca().spines['top'].set_visible(False)

plt.gca().spines['right'].set_visible(False) #編輯順序(前景與背景)

plt.scatter(..., zorder=1) #background

plt.plot(..., zorder=2) #foreground

使用不同主題

plt.style.available #列出可用的風格 plt.style.use('fivethirtyeight')

plt.style.use('dark_background')

plt.style.use('ggplot')

plt.style.use('bmh') plt.plot() #繪圖 #不同顏色的圖

plt.plot(x,y,'r')(紅色)

plt.plot(x,y,'k')(黑色)

顏色有以下參數：'b','g','r','c','m','y','k'

#線條風格

plt.plot(x,y,'o')

風格有以下參數：'o','^','s'

#結合不同顏色與線條風格

plt.plot(x,y,'ro')

儲存圖片檔