fabiochiu/medium-articles huggingface dataset:
What you need to know about Decision Trees
By: Surohit Tandon
Tags: Data Science, Supervised Learning, Machine Learning, Decision Tree, Sklearn

This post is going to be a little more technical than the last so lets put those old thinking caps on.

Right off the bat, we need to make a distinction. While most data nerds consider decision trees as a strict classification tool, there is also a regression variant. Even though regression trees split data at “nodes” (more on these later) it fits a regression model to each independent variable and the target variable, then it splits on the smallest SSE (sum of squared errors).

What we are most concerned with, and what most prediction gurus are concerned with, is good ol’ fashioned decision tree classifiers. We won't be going over any code here but I have attached a link to scikit learn’s decision tree classifier page where you can find some pseudo code on how to use the libraries decision tree class to create your own object in python and predict cool stuff.

Many people that are not yet baptised in the data world assume decision trees to be a simple case of if/else statements. While this does touch on the branch and leaf idea, it still misses a lot of what's under the hood with these models. What we WILL be going over is entropy, information gain and some of the more important hyperparameters that can be used when tuning a model.
Basics of machine learning
By: Deepesh Gupta
Tags: Reinforcement Learning, Data Science, Supervised Learning, Machine Learning, Unsupervised Learning

Basics of machine learning

Machine learning: A computer program that learn from its experience E, for a task T and gives performance P.if its performance at task in T as measured by P ,improves with experience E

Types of machine learning:

Supervised machine learning Unsupervised machine learning Reinforcement Machine learning

Supervised macine learning : we are given a dataset and know the target value of corresponding input feature so we can say that there is a relationship between input and output

Supervised learning algo is further divided into two parts:

Regression [multi class for ex:(age prediction, salary etc etc) ]

2. Classification [ binary class for ex:(gender prediction , dog vs cat , survival state etc )

Some of Supervised learning algorithms are :

Linear Regression Naive Bayes Decision Tree Random Forest Support Vector machine

Unsupervised machine learning: We are given a input feature (Data),And learn from the behaviour of input data

Unsupervised learning algorithm grouped into two problems :

Clustering [ you have n-nos of balls so you can make cluster based on color, type to identify the balls ] Association [recommended analysis for ex. ( in shopping mall market basket analysis, if you watched some X movie then you may like Y movie )

Some of Supervised learning algorithms are :

K-means clustering K-medoid clustering Hierarchical clustering K-NN (Nearest neighbour) DB-SCAN

Ex. of Unsupervised learning -

In healthcare, a hospital might cluster patients based on their tumor size so that patients with different tumor sizes can be treated differently. Suppose we are having an image of dogs and cats but the machine has no idea about the features of dogs and cats so we cants categorize it in dogs and cats but it can categorize them according to there similarities, patterns and differences. i.e we can easily categorize the picture into two parts.

Step 1 : formation of clusters based on input data

Step 2: finding distance of every clusters from target data

Step 3: choose shortest distance

Step 4: fit target into that cluster

Step 5: Predict target

Reinforcement Learning if we will talk about Supervised and unsupervised learning , learner learns from the model but in Reinforcement learning learner learns from the Agent based on Environmental rules

Learner → Agent

What learner learn ? Best policy

Brain of agent → Q — — learning equation

Q-learning equations :

Bellman equations MDP(Markov Decision Process)

Agent perform diff diff actions in environment

For every action , environment gives rewards(+ve,-ve,neutral rewards)

Based on received rewards, agent always always tries to get the maximum reward .on such way knowledge will be updated by Q-learning equation

Before training knowledge of agent is 0

Environment has 3 spaces

State space

Action space

Reward space

Ex-

thanks for reading this article .
Introduction to Machine Learning
By: Chahat Bindra
Tags: Libraries, Python Programming, Supervised Learning, Machine Learning, Algorithms

“Learning is any process by which a system improves performance from experience.”

“Machine Learning is concerned with computer programs that automatically improve their performance through experience” — Herbert Alexander Simon Alan Turing rightly questioned in his paper “Can machines do what we (as thinking entities) can do?” , all of this is becoming possible due to Artificial Intelligence and Machine Learning (AI/ML).

In this blog I would like to walk you through various aspects of Machine Learning .

So, what is Machine Learning?

Machine learning is a major application of artificial intelligence (AI) that provides systems the ability to automatically learn and improve from experience without being explicitly programmed. Machine learning focuses on the development of computer programs that can access data from the user and use it to learn for themselves.

Where do we use Machine Learning or what are its applications?

Applications of Machine Learning

Anomaly Detection

Bioinformatics

Computer Vision

Collaborative Filtering

Diagnosis

E-Commerce

Face Detection

Fraud Detection

Information Retrieval

Gaming

Gene expression

Handwriting Recognition

Human Interaction

Manufacturing

Marketing

Multimedia

Natural language Processing

Neural Network

Security

Search Engine

Sentiment Analysis

Spam

Speech Recognition

Text Summarization

Almost in each field Machine Learning is extensively used and folks have gotten benefit from it. From biometric authentication technology that is employed for security proposes and on social media platforms to assist users tag and share photos of friends. Or in Optical Character Recognition (OCR) that directly converts image of text into movable sorts which finds applications in various exams and departments to read important documents . Not to forget the Movie/Shows Recommender Engines, Self-driving cars, Spam classifiers all work on the principle of Machine Learning.
Supervised v/s Unsupervised
By: Sai Mounica Gudimella
Tags: Predictor, Supervised Learning, Unsupervised Learning, Classification, Outcomes

Supervised Learning :

Using labeled datasets to train algorithm in order to classify data or predict outcomes.

Output Y — Also called outcome, response, target, dependent variable

— Also called outcome, response, target, dependent variable Vector of inputs X — Also called as predictor, covariates, independent variable, regressors, features

— Also called as predictor, covariates, independent variable, regressors, features If it is a regression problem, then Y is quantitative

problem, then Y is If it is a classification problem, the Y is continuous / unordered set like 0/1, different species of iris-flowers etc.

In case of supervised algorithm, we have a training set to obtain the following :

Understand which inputs affect the outcome and how

Accurately predict the unseen test cases

Assess the quality of prediction and inferences derived from algorithm

Example — Predict the housing prices in a city or classify a mail as spam or not

Supervised v/s unsupervised

Unsupervised Learning :

Unlabeled data is given to the learning algorithm to derive patterns based on the input variables. So, if the dataset is labeled it is a supervised problem, and if the dataset is unlabeled then it is an unsupervised problem.

Vector of inputs X — Also called as predictor, covariates, independent variable, regressors, features

Example — handwriting pattern recognition
Performance measures for Supverised classification problem.
By: Lakshmi Sruthi
Tags: Supervised Learning, Machine Learning, Performance Metrics, Precision, Accuracy And Precision

Performance measures for Supverised classification problem.

To find a solution for our problem based on actual data and the predicted data which helps you to choose the right metric based on ur problem of the data.

Before choosing the right metric for our problem we need to know few terminologies of the measures of the classification problem. The Confusion matrix helps us to solve the problem by choosing the right metric for the problem.

Confusion Matrix

Note:

0 indicates the Absences (or) Negative

1 indicates the Presences (or) Positive

1. True Negative (TN):

An outcome where the model correctly predicts the Negative class. For example, You don't have a covid and the ML models predict correctly as a not a covid negative.

2. False Positive (FP):

An outcome where the model incorrectly predicts the positive class. For example, You don't have a covid and the ML predicts incorrectly as a covid positive. This type of error is called Type 1 error.

3. False Negative (FN):

An outcome where the model incorrectly predicts the negative class. For example, You have a covid but the ML predicts as you don’t have a covid negative. This type of error is called a Type 2 error.

4. True Positive (TP):

An outcome where the model correctly predicts the positive class. For example, You have a covid and the ML predicts you have a covid positive then it is True Positive.
K Nearest Neighbors
By: Nisa Soylu
Tags: K Nearest Neighbours, Machine Learning, Supervised Learning, AI

K Nearest Neighbor algorithm is one of the easy-to-implement supervised learning algorithms. It can be used to solve both classification and regression problems. KNN algorithms were proposed by T. M. Cover and P. E. Hart in 1967. The algorithm is used by making use of data from a sample set whose classes are known. The distance of the new data to be included in the sample data set is calculated according to the existing data and k number of close neighborhoods are checked. Generally, 3 types of distance functions are used for distance calculations:

“Euclidean” Distance

“Manhattan” Distance

“Minkowski

“ Distance

KNN algorithm is an algorithm that classifies a data to be classified according to its proximity with other data. For example, If we take k = 3, the distances of the new incoming data to the old data are measured and the closest 3 of them are determined. As can be seen in the figure below, 2 of the closest data are in A class and 1 is in B class. In this case, the algorithm decides that the new incoming data is in Class A.

KNN; It is one of the most popular machine learning algorithms because it is resistant to old, simple and noisy training data. However, it also has a disadvantage. For example, it requires a large amount of memory space when used for large data, since it stores all states while calculating distance.

Steps of the KNN algorithm:
GridSearch VS RandomizedSearch VS BayesSearch
By: Jay Hui
Tags: Data Science, Supervised Learning, Machine Learning, Optimization Algorithms, Bayesian Machine Learning

2. Result and theory

So based on the result, you can see BayesSearch provides the best result, even it only needs 60 iterations, while GridSearch and RandomizedSearch are searching for 120 iterations, but BayesSearch needs much more time.

So, what can we learn from this simple experiment?

RandomizedSearch defeats GridSearch

For GridSearch and RandomizedSearch, you should always use RandomSearch as RandomSearch performs better (search time should be similar). The reason is that RandomSearch can easily and efficiently find the optimized values across important parameters.

The illustration of GridSearch and RandomizedSearch. Uploaded by the author.

To illustrate it, consider the above picture with 1 important parameter and 1 unimportant parameter. With 3x3 = 9 combinations of GridSearch, actually, it only searches 3 different values for the important parameter in 9 iterations. However, for Randomized Search, it can search 9 different values for the 9 iterations. As a result, it is much easier for RandomizedSearch to search for the important parameters.

2. BayesSearch VS RandomizedSearch

As RandomizedSearch searches for the parameters randomly, what if we search it intentionally and directionally with the idea similar to Gradient Descent? So what we can further improve it is to consider the conditional probability (bases’ rule) to search for the parameters more wisely.

Since BasesSearch includes few computations for the conditional probability, for a small dataset, BayesSearch will take more time than the RandomizedSearch. Still, with a huge dataset, BayesSearch should have better performance than the RandomizedSearch given complex parameter search distributions and enough iterations.

Let us do one more experiment with a larger dataset:

best score of Randomized Search over 10 iterations: 0.91944

CPU times: user 22.9 s, sys: 157 ms, total: 23.1 s

best score of Bayes Search over 10 iterations: 0.95512

CPU times: user 1min 28s, sys: 749 ms, total: 1min 28s

In the first experiment with the small dataset, the time ratio between BayesSearch and RandomizedSearch is 129/0.9 = 143x. For the second experiment with the large dataset, the ratio now reduces to 88/23.1 = 3.81x.

In conclusion, based on the theory and computation experiments, we suggest the following:
Machine Learning can be divided into 3 categorizations: Supervised, Unsupervised and Reinforcement…
By: Zina Youhan
Tags: Learning, Reinforcement Learning, Supervised Learning, Unsupervised Learning, Machıne

Machine Learning Part_2

Machine Learning can be divided into 3 categorizations: Supervised, Unsupervised and Reinforcement Learning.

Let see the difference between Supervised, Unsupervised and Reinforcement in simply 😊.

Supervised Machine Learning

The below table shows a data set which is consist with rows of data. When we working with machine learning we need datasets for different purposes.

Let see some terminologies we are going to use in Machine Learning.

We call a row as an observation, an example or a tuple.

Features: Features are the input variables that describing our data.

Labels: A Label is the target that we’ve trying to predict.

Then what is Supervised Machine Learning???

First, we are training the model using Features and Labels. Then we are learning to create models that combine inputs, to produce useful predictions even on previously unseen data. We can divide supervised machine learning into two categories; Classification and regression.

What is classification???

If we can select one of known label according to given feature; we known it as classification.

Classification can be divided into two categories; Binary Class and Multiclass

Binary class- If we can select one of two known labels, we say it belongs to binary class.

As an example: — Labels like ‘yes’ or ‘no’

Multi class- if we can select one of the labels out of multiple known labels, we say it belongs to multi class.

As an example: — We can categorize the types of songs (Labels) as below.

Rock, Ballads, Soul, Novelty, and Folk etc.

What is Regression???

If we can’t predicate the label accurately. It used to predict a continuous value. Predicting prices of a house given the features of the house like size, price etc.

What is Unsupervised learning???

Unsupervised learning is a machine technique, where you do not need to supervise the model. Instead, you need to allow the model to work on its to discover information. It mainly deals with the unlabeled data.

Example:

Let’s, take the case of a baby and her family dog.

She knows and identifies this dog. Fes weeks later a family friend brings along a dog and tries to play with the baby.

Baby has not seen this dog earlier. But it recognizes many features (2 ears, eyes, walking on 4 legs) are like her pet dog. She identifies the new animal as a dog. This is the unsupervised learning.

Let’s look at what is Reinforcement Learning

Reinforcement learning is all about making decisions sequentially. In simple words we can say that the output depends on the state of the current input and next input depends on the output of the previous input. In Reinforcement learning decision is dependent, so we give labels to sequences of dependent decisions.

Ex — Chess

Let’s learn about some architectures related to machine learning in chapter 😊.
What are Regression Algorithms in Supervised Machine Learning?
By: Micro Learning
Tags: Supervised Learning, Machine Learning, Regression

Regression algorithms are a subset of Supervised Machine Learning technique which are used to predict output values based on the input labeled data.

Opposite to classification technique which is used to make predictions where the output variable is a category (yes/no, spam/not spam), regression models are used to predict a continuous value (real numbered output — temperature, stock price, weight, etc). So, regression is one of the most widely used statistics and machine learning tools for deriving intelligence from data.

Which are the main applications of regression models in machine learning?

to predict real estate values based on location, size, price, and other factors;

financial forecasting

trend analysis

marketing

time series predictions

Which are the most popular types of regression algorithm?

Simple Linear Regression; Lasso Regression; Logistic Regression; Support Vector Machines; Decision Tree Regression Random Forest Regression Multivariate Regression algorithm; Multiple Regression Algorithm; Polynomial Regression.

Resources:
Types of Machine Learning Techniques
By: Kashish Chugh
Tags: Semi Supervised Learning, Reinforcement Learning, Supervised Learning, Machine Learning, Unsupervised Learning

Machine Learning is a powerful technology having the potential to find practical solutions to problems in diverse fields. In machine learning, there are many techniques used to train the data and get the output.

Supervised Learning

In Supervised Learning, an algorithm learns from a dataset. We give the data as well as the output corresponding to each input. The algorithm learns from the given data and next time if we enter the data, it will give the output in accordance with the given data. The data provided can be of two types:

Discrete: In discrete data, we use classification. Classification is the process where incoming data is labeled based on past data samples and manually trains the algorithm to recognize certain types of objects and categorize them accordingly.

In discrete data, we use classification. Classification is the process where incoming data is labeled based on past data samples and manually trains the algorithm to recognize certain types of objects and categorize them accordingly. Continuous: For continuous data, we use regression. Regression is the process of identifying patterns and calculating the predictions of continuous outcomes.

Some widely used supervised learning algorithms are:
Patterns in Self-Supervised Learning
By: Nishant Sinha
Tags: Deep Learning, Supervised Learning, Machine Learning, Data Augmentation, Few Shot Learning

Photo by Kevin Gent on Unsplash

Probing the Self for Fun and Profit

Self-Supervision is in the air (and the talks). Explaining the difference between self-, un-, weakly-, semi-, distantly-, and fully-supervised learning (and of course, RL) just got exponentially tougher. :) Nevertheless, we are gonna try.

The problem, in context, is to encode an object (a word, sentence, image, video, audio, …) into a general-enough representation (blobs of numbers) which is useful (preserves enough object features) for solving multiple tasks, e.g., find sentiment of a sentence, translate it into another language, locate things in an image, make it higher-resolution, detect text being spoken, identify speaker switches, and so on.

Given how diverse images or videos or speech can be, we must often make do with representations tied to a few tasks (or even a single one), which break down if we encounter new examples or new tasks. Learning more, and repeatedly, and continuously, from new examples (inputs labeled with expected outputs) is our go-to strategy (supervised learning). We’ve secretly (and ambitiously) wished that this tiresome, repeated learning process will eventually go away and we’d learn good universal representations for these objects. Learn once, reuse forever. But, the so-called unsupervised learning paradigm (only-input-no-labels) hasn’t delivered much (mild exceptions like GANs and learn-to-cluster models).
Machine Learning and Algorithms
By: Deepak Kumar
Tags: Artificial Intelligence, Supervised Learning, Machine Learning, Algorithms, Python

Supervised Learning: Supervised learning is the machine learning task of learning a function that maps an input to an output based on example input-output pairs. It infers a function from labeled and classified data.

In supervised learning, we are given the data sets and already know what our correct output should look like.

big-datamadesimple

Supervised Learning can be further divided into Regression and Classification Algorithms:

Regression : In regression we fit the training data into the continuous function and try to predict results within a continuous output, meaning that we are trying to map input variables to some continuous function. eg: Predicting the price of a house according to the size. Here the price is a function of the size of the house which is the continuous output. so this is a regression problem.

: In regression we fit the training data into the continuous function and try to predict results within a continuous output, meaning that we are trying to map input variables to some continuous function. eg: Predicting the price of a house according to the size. Here the price is a function of the size of the house which is the continuous output. so this is a regression problem. Classification: In classification, outputs are predicted in discrete value such as yes or no, true or false,0 or 1, diabetes or not, male or female, positive or negative, etc. eg: In given health data predicting a person has diabetes or not is classification.

Some popular examples of supervised machine learning algorithms are:
Supervised Learning in Machine Learning
By: Rayudu Yarlagadda
Tags: Supervised Learning, Machine Learning

supervised learning is a learning in which we train the machine using data which is well labeled (i.e the questions with answers). After that, the machine is provided with a new set of questions(data) so that supervised learning algorithm analyses the training data(set of training examples) and produces a correct outcome from labeled data.

For instance, suppose you are given an basket filled with different kinds of fruits. Now the first step is to train the machine with all different fruits one by one like this:

If shape of object is rounded and depression at top having color Red then it will be labelled as – Apple .

. If shape of object is long curving cylinder having color Green-Yellow then it will be labelled as –Banana.

Now suppose after training the data, you have given a new separate fruit say Banana from basket and asked to identify it

Since the machine has already learned the things from previous data and this time have to use it wisely. It will first classify the fruit with its shape and color and would confirm the fruit name as BANANA and put it in Banana category. Thus the machine learns the things from training data(basket containing fruits) and then apply the knowledge to test data(new fruit).

Supervised learning classified into two categories of algorithms:
AAAI 2020: Unsupervised Deep Learning and AI that can reason
By: Vivek Khetan
Tags: AI, Artificial Intelligence, NLP, Supervised Learning, Self Supervised Learning

Geoffrey E. Hinton: Capsule Network using Autoencoders

Don Hinton’s presentation was about the limitations of CNN and how to overcome them. In his opinion, CNNs do not recognize the object the same way we do. People use coordinate frame and always assign intrinsic coordinate frame when they perceive shapes — this is a crucial aspect of human perception. In contrast, CNNs depend on coincidence — the coincidence between the value of two sides of the equation as well as coincidences making high dimensional filters to recognize images.

We want neural networks to generalize to new viewpoints effortlessly, and it will happen only when they can cope up the rotation and scale as good they are doing with viewpoint translation.

He has been developing capsule networks recently. The idea is to represent the segments of a given image using a group of neurons and then representing the image using these learned sub representations to identify the image. He first introduces Capsule networks in NIPS 2017 and the most recent one in NeurIPS 2019.

The current version of the capsule network is unsupervised and is totally different from the previous two. They are called Stacked Capsule Autoencoders and based on generative modeling techniques.

Don LeCun started his presentation by defining deep-learning and stating all the limitations of Supervised Learning are sometimes mistakenly seen as intrinsic limitations of DL. In his opinion, supervised learning and reinforcement learning does nest when we have much data or can simulate the environment for too many trials to learn anything, respectively. However, in real-world scenarios, obtained good labeled data is a problem as well as we cannot run real-world faster than real-time to try various scenarios — many times, the task looks simple, but a small mistake can kill us as well.

To achieve this intellectual level of prediction, we need to come up with methods to solve the three main challenges in front of deep learning. His research group at Facebook, as well as at NYU, is working on self-supervised learning to overcome this challenge. Prediction is the essence of Intelligence, and we humans learn models of the world by predicting part of the input from other parts, the future from the past, the masked from the visible, and any occluded part from all available parts.
K-Means Vs kNN
By: Sravan Roy
Tags: Knn, K Means, Supervised Learning, Machine Learning, Clustering

What’s the contrast of ‘ k ’ in k-Means and kNN ?

Often times, k-Means and kNN algorithms are interpreted in same manner although there is a distinct difference between the two. Today, we look into the major contrasts in implementing these algorithms.

k-Means

k-means clustering algorithm is one of the popular unsupervised machine learning algorithm where similar data points are grouped together to discover underlying patterns with no fixed labelled outcomes.

k in k-Means

We define a target number k, which refers to the number of centroids we need in the dataset. k-means identifies that fixed number (k) of clusters in a dataset by minimizing the within-cluster variances.

Every data point in the dataset is allocated to it’s nearest cluster while keeping the centroids as small as possible.

‘K ‘in k-means is the number of clusters the algorithm is trying to learn from the given data

The best value for k is chosen from elbow and silhouette methods

kNN

k-nearest neighbors is a supervised classification/regression algorithm where a bunch of labelled points are used to determine the class of other points.

‘k’ in k-NN is the number of nearest neighbors used to classify (or predict in case of continuous variable) a test observation sample

In k-NN classification, the output class is determined by the plurality vote of it’s neighbors, with the object being assigned to the class most common among it’s k nearest neighbors In k-NN regression, the output is the average of the values of k nearest neighbors

In the above figure, the green dot could be assigned to either red class or blue class depending on whether the value of k is 3 or 5 respectively.

The optimal value of k is chosen through various hyperparameter optimization techniques

For high dimensional data, feature extraction and dimensionality reduction techniques such as PCA, LDA or CCA are used as a pre-processing step, followed by k-NN in the reduced-dimension space

Wrapping up

k-means is commonly used in scenarios like understanding population demographics, market segmentation, social media trends, anomaly detection, etc.. where the clusters are unknown.

k-NN, on the other hand, is used to classify and predict data where the target variable is usually known before hand.

Cheers! Now we understood the difference between k-Means and k-NN !!
Interactive Plots with Plotly by a Beginner for Beginners — Part I
By: Yazan Shannak
Tags: Data Visualization, Data, Data Science, Plotly, Python

Step by step code

Installation

First let’s start by installing Plotly on your environment

You can use pip or conda

pip install plotly conda install plotly

Import the library

Now create a new notebook or use an existing one and let’s import common libraries and Plotly

Data Preparation

For the sake of this post, we will generate some dummy data to visualize, so let’s do an example of different cities with the different car brands that people have in addition to other features we will add later to imply different types of plots later . Let’s go with the famous cities of New York, Berlin and London, and my favourite car brands of Mercedes, Dodge, and Mitsubishi.

Pie Chart

Now let’s start creating some good looking interactive charts, starting with the distribution of car brands in New York only .

As we mentioned earlier, we’ll first create the trace which is a pie chart and as we mentioned earlier pie charts uses values and labels, so our labels are the brands’ names and the values are the count of each brand in New York.

So a three step procedure would be:

Create a pie trace defining values and labels. Create a layout Create the plot object and pass it to the plotting function

Note this, when defining the Figure variable, make sure to pass the data parameter as a list, otherwise it’ll throw an error.

and we should get this plot:

Now since we explored the first step, let’s try and edit the trace’s style, remember this is to be defined while creating the trace itself. Let’s do the following:

Change the info on the chart to view both the label and percentage Change the info when you hover to view the numeric value solely. Change the colors to ugly ones since I’ve always been terrible at picking them.

So what I would do is pay the official documentation a visit and explore all the possibilities and write the following code:

As you noted, the colors are defined through a ‘marker’ parameter which has a colors key, and the valid color codes are HEX color codes, the other two variables are defined by separate parameters. This all can be tricky to digest at first, but you’ll get used to the common variables and parameters after doing a single project.

One more thing before we can view the chart, if the pie sections show the labels, why would we need a legend?. The legend is part of the overall layout of the plot, to erase it we’ll define it through the layout variable.

and voila here’s your ugly coloured pie chart!

This is it for part one, hope it gave you a fare and brief understanding of plotly, in the next posts we will try other features and see how plotly can be powerful when integrated with the one and only Pandas library.

Happy Visualizing! :)
Plotly Experiments — Column and Line plots
By: Naren Santhanam
Tags: Data Visualization, Data Exploration, Bar Chart, Data Science, Plotly

Bar plots

In my previous post, I had explained how to create scatterplots using Plotly with examples from the King County housing dataset. Another popular type of plot is the column plot or bar plot. Unlike a scatterplot, which is used to compare two numerical variables against each other and examine relationships, bar / column plots are useful to investigate one or more numerical variables across different categories. There are different types of bar plots — individual, clustered, stacked, etc.

The dataset

I will be using the Bay Area Bike Share dataset from Kaggle for this post. The Bay Area Bike Share enables quick, easy, and affordable bike trips around the San Francisco Bay Area. They make regular open data releases (this dataset is a transformed version of the data from this link), plus maintain a real-time API. The dataset has various files that contain information on the different stations in the Bay Area, the trips taken each day along with the weather info. I will be using the stations and trips datasets for this data exploration exercise.

Stations

Let us take a look at the stations dataset:
Plotly Experiments — Scatterplots
By: Naren Santhanam
Tags: Data Visualization, Data Exploration, Scatter Plots, Data Science, Plotly

Plotly has a wide variety of plots and offers users a high amount of control on the various parameters to customize the plots. As I learn more about this package, I would like to walk through some of my experiments here as a way for me to practice and also serve as a tutorial for anyone who wants to learn.

Notes about Plotly

Plotly charts have two major components: data and layout.

Data — this represents the data that we are trying to plot. This informs the Plotly’s plotting function of the type of plots that need to be drawn. It is basically a list of plots that should be part of the chart. Each plot within the chart is referred to as a ‘trace’.

Layout — this represents everything in the chart that is not data. This means the background, grids, axes, titles, fonts, etc. We can even add shapes on top of the chart and annotations to highlight certain points to the user.

The data and layout are then passed to a “figure” object, which is in turn passed to the plot function in Plotly.
From Punchcards to Plotly
By: Eric Langowski
Tags: Japanese American, Data Science, Plotly

Last time, I wrote about how I set out to tackle an arcane form of data in the form of 1940s administrative punchcards. After many struggles (and processing hundreds of literally mispunched values), I was able to convert the NARA webpages to a pandas dataframe and begin to visualize the data using plotly.

The birth years of Japanese Americans in the data set. As expected, the data shows the stark generational divides between the Issei (first generation immigrants) and Nisei (second generation American born). Due to changes in immigration laws and labor conditions, immigrating from Japan to Hawaii or California became largely impossible after the 1920s.

My web scraper only captured the punched values — so the first step was to create a data dictionary. NARA provides the original documentation, but…

It is in a less than helpful format. Fortunately, NARA has also provided an individual CSV file for each entry field. A simple loop using requests and pandas.to_dict() later…and I still had hundreds of values which could not be successfully coded. I wrote a helper function to try to see if there was a pattern.

def bad_values(df, values_dict):

for col in df.columns:

col_values = df[col].unique()

valid_values = values_dict[col]

if valid_values:

bad_values = set(col_values) - set(valid_values)

if bad_values:

print(col, ' has bad values')

for value in bad_values:

count = df[df[col] == value].shape[0]

if count > 5:

print(value, 'appears ', count)

By comparing the set of unique values in a column to valid values in the NARA dictionary, I was able to find “bad” values in the data and printed out only those which appeared more than five times for simplicity. A lot of the missing values were things coded as 39 instead of 039 or 4–2 instead of 4X2. One set of “bad” values suggested a mistake in NARA’s coding. The field “Total Years of Schooling in Japan” was coded A = 0 (for 0 years) to G = ’16 YEARS, ETC.’ but the values H, I, J, K, L had no codes (I suspect these codes might represent 17, 18, 19, 20, and 20+ years).

After filling in the majority of missing codes, I wrote a simple loop which mapped the coded values to their descriptive names.

for col in col_names:

if values_dict[col]:

df[col] = df[col].map(values_dict[col]).astype('category')

Over the coming weeks, I will explore visualizing and analyzing categorical columns in the dataset and linking this data with other data sets.

Feel free to follow along!
4 Technical Duh! Lessons I Learned from My Latest Data Science Project
By: Orhan G. Yalçın
Tags: Data Visualization, Dash, Data Science, Plotly, Python

The initial dataset was provided in the form of a Google Sheet. The file consists of 58 separate sheets and more than 150k US government contracts. There are 26 columns, and all the sheets store the same info in different business areas. I worked with millions of data points before, and I thought like:

I am just going to download this as an XLSX file, list all the sheet names, run a for loop to create a data frame for each sheet, and finally pd.concat() them.

Boy, was I wrong?

It took 58 minutes to read 150k rows, and after a quick search, I realized that pandas perform terribly in XLSX format. I’ve never used XLSX format for a big data(ish) project before and had no idea!

Note that I actually considered downloading the file as a CSV file, but there is no easy and automated way to save every sheet as a separate CSV file. Also, the CSV format can only have a single sheet.

Seeing how terrible pandas performed with the XLSX format, I reached out to the client and asked for CSV originals. Reading 470k observations from 108 files took under 1 minute!

After completing the project, I found this solution:

2 — If you want full customization, go with Plotly Graph Objects, not Plotly Express

As a person on the clock, I always try to find the quickest way to reach the desired outcome. Our desired outcome was to represent each company’s aggregated contract info. I cleaned and processed the data nicely and wanted to create an interactive bubble chart. Well, if you are using Python, your best bet is Plotly. Now, Plotly is a very advanced but also a very messy library. It basically offers several modules to convert your data into HTML visuals using the JavaScript D3.js library. The idea is revolutionary, but the documentation is a mess. Why? Let me explain.

Plotly has a module called Graph Objects. It’s the main Plotly module that developers can create beautiful interactive charts. You have full independence to design your chart however you want. I am talking about placing images, changing colors, dealing with legends, and much more. But, Plotly offers another module called Plotly Express. It’s also great, and you can also create great visuals, almost identical. But almost…

Well, since I wanted to deliver fast and within the budget, I opted for Plotly Express. Everything went perfectly until I realized that I could not really change the legend style. When you visit the documentation, there are complex methods to achieve it; whatever I try, it returns with a warning ⚠️ and doesn’t do the trick.
How To Use Plotly.js With Angular 8
By: Tushar Sadana
Tags: Angular, JavaScript, Plotly, Programming, D3js

Getting Started

We are going to use the Vanilla JavaScript version of Plotly in Angular to make it work faster and smoother with their WebGL version.

This tutorial assumes that you have an application on Angular.

We will first include Plotly.js in our Angular project.

Download their plotly-latest.min.js (1.48.3 was the version when I was writing this tutorial) from the download section. (You can use the CDN but I wanted to keep my application totally internet-independent).

If you are using the CDN, simply add the script tag in the index.html .

If you downloaded the file, copy the file to the assets folder.

Add this line in angular.json in the scripts tag:

"scripts": [ "src/assets/plotly-latest.min.js" ]

We have successfully included the library in our project, now we are going to use it.

Let’s assume we have a component graph where we are going to use this.

In graph.component.ts :

declare var Plotly: any;

export class GraphComponent implements OnInit {

...

...

}

We create a div where the graph should be plotted in the HTML file, you can add your CSS and HTML tags accordingly. I’ll provide mine in the end which helped me keep my project responsive.

graph.component.html

<div id="Graph" #Graph></div>

Going back to the graph.component.ts .

We use @ViewChild to find the element from the DOM.

The reasons to use ViewChild instead of document.getElementById are:

It is TypeScript. If the div is not visible (i.e. it may be in some tab which is to be rendered when you go to that tab, or its visibility is hidden for some reason in the application), the document.getElementById will return an error; it will not find it.

Now, we get the data. We can get it from any GET request, a file, or whatever source you might have.

I am going to use a GET request.

Next, we need to format the data in a particular way for this graphing library to work:

We’ve formatted the data and it is time to define the layout of the plot and finally plot it.

We use the function, Plotly.newPlot() .

Now, serve the application and check.
The geography of Pubs in the UK: Visualizing Geographical Information with Plotly
By: Juan Felipe Alvarez Jaramillo
Tags: Data Wrangling, Maps, Data Visualization, Plotly

In my previous blog, I performed some Text Mining techniques to learn more about the names of the Pubs in UK. Since the source of the information contained also the location of each Pub, I am writing this second blog to learn something different about the drinking culture of this country.

Plotting a map is an excellent way of making sense of datasets that contain information about location in the physical world. To understand where the Pubs are geographically concentrated, I would like to start with a basic Scatterplot using the coordinates (latitude and longitude) provided in the dataset.

To do so, I will use Plotly, which is a very advanced and flexible data visualization library. The chart below uses an integration with Mapbox in order to get the base layer of the map and then plot the dots on top of that. We can see that the largest concentration of Pubs happen in the Greater London region, the Liverpool-Manchester-Leeds corridor and also in the region surrounding Newcastle.

If you wish to do so, you can zoom in to get a more detailed look and also hover the mouse to see the pubs name and coordinate. Click here to see a zoomable version of the map.
How to plot a grouped stacked bar chart in plotly
By: Moritz Körber
Tags: Visualization, Python, Bar Chart, Plotly

plotly makes it easy to create an interactive stacked or grouped bar chart in Python by assigning the desired type to the layout attribute barmode. Unfortunately, barmode only takes either stack or group but not both as an argument. It seems like they are currently working on this option, but there is a workaround for now: subcategory axes!

Let’s say you compare two design layouts of your online store — modern and classic — in a small A/B test. You collect data for three consecutive weeks and count the number of conversions and exits per layout and week. Let’s convert this information into a data frame:

To plot this, we need a categorical x-axis that shows the week, stacked bars to show the number of conversions and exits, and those bars grouped into modern and classic. Since barmode cannot be stack and group at the same time, the grouping must be done by sub-categories. Simply pass a list in the form of [category, sub-category] to x , leave barmode at "stack" and add a trace per response category (here: conversion and exit).

And here it is:
Perfect Data Visualization Tool in Python (Not matplotlib) (Plotly套件介紹)
Tags: Data Science Tools, Data Visualization, Towards Data Science, Plotly, Python

Hello, if you do not read Chinese version article, you could move to my github or nbviewer. That would give you as same as the article in here but in English version. (I would upload these links as soon as possible)

Hi, 大家最近過的好嗎? 今天我要來介紹一個新的套件,想必各位如果有在持續增進自己python的視覺化能力,你會發現,可能必須花費好幾個小時去研究自己想要的功能如何展現?如何製作註解?如何將在圖表上看的特性標記出來?你常常要花好幾行的code才能呈現你所想要的功能,為了解決這些問題,我將使用plotly來取代原本你所使用matplotlib,它能夠更好的解決你所面臨的麻煩,也是我今天想要撰寫這篇文章的目的

這篇文章所有的內容都會包含在我的github裡,可以去看看

plotly

那什麼是plotly呢? plotly是一個能讓你畫出互動式圖表的一個開源套件,其基本操作是免費的,如果你想要使用網頁的版本,可以使用更多plotly的進階功能的話,就必須付費

plotly+cufflinks

cufflinks這個套件是用來結合pandas和plotly的,你可以使用dataframe的同時,直接加iplot就可以產生比matplotlib還要更fancy的圖表

pip install plotly

pip install cufflinks

現在要先導入要使用的套件

#可以製作互動式圖表的套件

import plotly.plotly as py

import plotly.graph_objs as go

from plotly.offline import iplot,init_notebook_mode #為可以將pandas和plotly結合的套件

import cufflinks

cufflinks.go_offline(connected=True)

init_notebook_mode(connected=True)

Data

這次的資料集我依舊會使用之前第一篇文章所用來分析Lebron James常規賽的比賽數據當作我這次視覺化專案的工具

之後我也將根據圖的不同一種一種的做介紹

可以使用在cufflinks下的圖表有這一些

scatter, bar, box, spread,

ratio, heatmap, surface, histogram,

bubble, bubble3d,scatter3d, scattergeo,

ohlc, candle, pie, choroplet

iplot裡面有的參數,我將針對其中的幾項進行探索與教學,你可以根據你想要的圖表進行分別給予

DataFrame.iplot(

["kind='scatter'", 'data=None', 'layout=None', "filename=''", 'sharing=None', "title=''", "xTitle=''", "yTitle=''", "zTitle=''", 'theme=None', 'colors=None', 'colorscale=None', 'fill=False', 'width=None', "dash='solid'", "mode='lines'", "interpolation='linear'", "symbol='circle'", 'size=12', "barmode=''", 'sortbars=False', 'bargap=None', 'bargroupgap=None', 'bins=None', "histnorm=''", "histfunc='count'", "orientation='v'", 'boxpoints=False', 'annotations=None', 'keys=False', 'bestfit=False', 'bestfit_colors=None', 'mean=False', 'mean_colors=None', "categories=''", "x=''", "y=''", "z=''", "text=''", 'gridcolor=None', 'zerolinecolor=None', 'margin=None', 'labels=None', 'values=None', "secondary_y=''", "secondary_y_title=''", 'subplots=False', 'shape=None', 'error_x=None', 'error_y=None', "error_type='data'", 'locations=None', 'lon=None', 'lat=None', 'asFrame=False', 'asDates=False', 'asFigure=False', 'asImage=False', 'dimensions=None', 'asPlot=False', 'asUrl=False', 'online=None', '**kwargs'],

)

histogram

給予單一變數進行畫圖,表示x軸為該變數,而y軸在為其計數
Deploy Plotly Dash app with Dash DAQ components on AWS cloud
By: Ruslan Korniichuk
Tags: Plotly, Deployment, Dashboard, AWS, Python

To share a Dash app, you need to “deploy” your Dash app to a server. The guide below works for Dash application with and without Dash DAQ components. We’ll deploy app on AWS in easiest way.

Dash “Hello, World!”

Let’s create new directory (e.g. medium ). Create new application.py file with code below:

Note: Line app.scripts.config.serve_locally = True is required for Dash DAQ components.

Python dependencies

Navigate to project directory (e.g. medium )and create new requirements.txt file with code below:

AWS Elastic Beanstalk

AWS Elastic Beanstalk is an easy-to-use service for deploying and scaling web applications and services developed with Python.

Install the Elastic Beanstalk Command Line Interface (EB CLI):

$ sudo pip install awsebcli

If you want install EB CLI on Amazon Linux, use instruction below:

$ sudo yum update $ sudo yum install git

$ sudo yum groupinstall "Development Tools"

$ sudo yum install libffi-devel openssl-devel $ git clone https://github.com/aws/aws-elastic-beanstalk-cli-setup.git

$ ./aws-elastic-beanstalk-cli-setup/scripts/bundled_installer $ echo 'export PATH="/home/ec2-user/.ebcli-virtual-env/executables:$PATH"' >> ~/.bash_profile && source ~/.bash_profile $ rm -rf aws-elastic-beanstalk-cli-setup

Initiate EB

Navigate to project directory (e.g. medium )and initializes your directory with the EB CLI:

$ eb init

Select a default region (e.g. eu-west-1 ). Enter application name (e.g. medium ).

It appears you are using Python. Is this correct?

(Y/n):

Enter Y .

Select Python version (e.g. Python 3.6 ).

Do you want to set up SSH for your instances?

(Y/n):

Enter n .

Deploy Dash app

Create a new environment:

$ eb create

Enter environment name (e.g. medium-dev ). Enter DNS CNAME prefix (e.g. medium-dev ). Select a load balancer type (e.g. application ).

Open the application URL in a browser:

$ eb open

Update Dash app

Let’s update Dash app. Change:

html.H1('Hello, World!')

line to:

html.H1('Hello, Medium!')

Deploy new source code to the environment:

$ eb deploy

Open the application URL in a browser:

$ eb open

Delete Dash app from AWS cloud

Terminate the environment: