fabiochiu/medium-articles huggingface dataset:
Exploratory Data Analysis
By: Vinoth Kumar Chandar
Tags: Data Science, Greatlearning, Data Analysis, Exploratory Data Analysis, Business Analytics

Exploratory data analysis(EDA) is an approach to analyze the data to summarize their main characteristics, often using statistical graphics and other data visualization methods. EDA is all about making sense of data in hand, before getting them dirty with it. EDA is similar to the art of storytelling, a story that data is trying to tell.

‘Data science is the field of study that combines domain expertise, programming skills, and knowledge of mathematics and statistics to extract meaningful insights from data.’

By conducting EDA, we can achieve the major role of Data Science as like providing meaningful insight into the data as to how the variables are correlated to each other and how it is impacting our Business. EDA gives a better understanding of the Variables and the Relationship between them along with the Pattern Identification of the Distribution. From which we can understand the spread of the data and how it is been distributed. Even with the high level of domain knowledge, after performing the EDA, we will be able to understand the Business much better on the basis of how each variable is correlated to each other.

The Main Components of exploring data

· Understanding the variables in the dataset

· Cleaning and Identifying the Patterns in the dataset

· Analyzing relationships between variables in our dataset

By going through the exploratory data analysis, we’ll have a much better understanding of our data along with the business process, which will make it easier to choose our model, our attributes, and refine it overall. This piece of work(EDA), will make a huge change in our Data Modelling which will also return to a more accurate and appropriate solution for our Business.

Thanks! :)
Alteryx vs Python: Data Cleaning, and EDA
By: Varun Ganti
Tags: Data Science, Python Programming, Alteryx, Data Cleaning, Exploratory Data Analysis

This article aims to highlight the differences between using python and alteryx for data analysis. Python is a open source, high level object oriented programming language. Python provides specific functionality to deal with high level data science application. Specifically, the pandas package in python is great for reading in data , cleaning the data, and exploring the data.

Alternatively , alteryx provides data analytic solutions by implementing a easy to use graphical user interface for the ETL(Extract, Transform, Load)process. Alteryx is great at visualizing the end to end workflow for data analytics and provides drop down user interface for the non-programmer. In my opinion, both python and alteryx can be used in conjunction, to maximize efficiency, in the data science process.

Below is an illustration of a sample workflow in Alteryx

Sample Workflow Alteryx

To test my assumption, that the best way to perform data cleaning and eda is to use python and alteryx in conjunction, I analyzed the framingham heart disease data set.

Data Cleaning

Below, is a table that illustrates some of the key data cleaning functionalities and tools used in both python and alteryx

Data Cleaning: Alteryx vs Python

The table, above, illustrates the technical tools, used in both python and alteryx, to perform efficient data cleaning. It is important to note that python requires some sort of basic programming knowledge, whereas alteryx is essentially a drag and drop software.

Winner

Tie

Exploratory Data Analysis

Below, is a table that illustrates some of the key EDA functionalities and tools used in both python and alteryx.

EDA: Alteryx vs Python

I really enjoy using python’s seaborn and matplotlib packages to build more customization on visuals for EDA. Alternatively, alteryx is much easier to navigate and use when it comes to EDA. I have yet to dive deeper into the visual capabilities in Alteryx, but from my understanding it seems that the most efficient way to build visuals is to clean the data in Alteryx, and use either python or tableau, to finish the job.

Winner

Tie

Conclusion:

To conclude, after using both alteryx and python for data cleaning and EDA, I believe the best way to analyze data is to use both in conjunction. Alteryx has a nice easy user interface that allows the average non-programmer complete complex data analysis that cannot be done in excel. Alternatively, python is a language with more customization at the hands of the user with built in data analysis packages. Alteryx has a built in tool that enables the user to add python scripts in the workflow. This extra functionality in alteryx allows for both alteryx and python to be used in conjunction for strong data analysis. Next week, I will be comparing the modeling capabilities of both python and alteryx.
A better EDA with Pandas-profiling
By: Thomas Gey
Tags: Data Science, Python, Machine Learning, Pandas Profiling, Exploratory Data Analysis

A world full of bad data

Mostly in Data Science, data we collect aren’t as clean and complete as available datasets on internet, designed and prepared for the application of machine learning models. Mostly, data comes from external sources, is not well structured or is simply incomplete, and it’s your job to make this data welcoming and usable!

While data scientists are aware of the importance of data quality, this is not necessarily the case for business leaders. They focus more on adopting technology as early as possible to maintain a competitive advantage and often have little understanding of the practical aspects. This means that people use everything in their datasets and hope for the best, under the pretext that quantity can compensate for quality.

Because of this difference of mindset, a data scientist spends 80% of his time preparing data sets before modelling.

Why 80% of his time? There’s missing values, unbalanced data, column names without any meaning, deduped data, outdated data and other reasons. Awareness of the importance of data is recent. As a result, even if the trend is reversed, the data is rarely easily accessible and well structured.

The consequence of a bad dataset is simple and logic: predictive models made with these datasets will have low accuracy and low efficiency. Train a model with incorrect data will create a bias and your results will be far from reality. A bad model is a model unable to be deployed, so its a loss of money for the company developing the model.

A car built with rusted pieces will not drive correctly or at worst, not drive at all… and nobody wants to use or buy a rusted car!

So, your dataset is an essential key to your project. Its technical and economic success depends on the quality of data you selected. During a machine learning project, skipping this data evaluation step can lead to a considerable waste of time and you will have to start your project over again from the beginning. It seems logic now data scientists spend so much time investigating and preparing datasets!
EXPLORATORY DATA ANALYSIS AND REGRESSION
By: Kaushal Gupta
Tags: Artificial Intelligence, Data Science, Exploratory Data Analysis, Machine Learning, Regression

In this article we will talk about what is Exploratory data analysis ,why we need to do this and how to implement this using python programming. And also use regression model .

Introduction

EDA (Exploratory Data Analysis) is a data analysis technique by which we try to understand the given data and the relationship among those data and also try to find out features , insights and anomalies from the data using statistics and graphical representation.

Regression is a Supervised learning which predicts a numeric values based on previous observed data . Regression analysis is a statistical process of estimating the relationship among variables .

PHASE 1: UNDERSTANDING THE VARIABLES AND RELATIONSHIP AMONG THEM

Step 1 : Importing necessary Libraries and the data set.

Step 2: Understanding the variables in the data set.

Step 3 : Now Check the dimension of the given data set.

Step 4 : Dropping Duplicates rows in the columns if any

we observe that there is one duplicate column in the data set .

Step 5 : Using describe method to have information about the variables

Step 6 : Using value_counts() method on region and sex to identify the count of each category in that column.

Step 7 : Using pairplot method to understand the distribution of each variable with other

Step 8 : Plotting heat map to understand the correlation of variables .

Step 9 : Understanding the distribution of the charges .

Step 10 : Understanding the distribution of the age .

Now We have enough knowledge of about our data set . So, lets find some features from this data set.

PHASE 2 : FEATURE ENGINEERING

Step 1 : How many females and males are smokers

Step 2 : Smoker who are below or equal to 18 years

Step 3 : Number of smokers from different regions

Step 4 : Number of children of smokers patients

Step 5 : Number of children of smokers patients

Step 6 : Smokers on the basis of age

Step 7 : Non-Smokers on the basis of the age

PHASE 3 : APPLYING REGRESSION MODEL

Step 1 : Splitting the data set into input and output

Step 2 : Encoding the Categorical variables

Step 3 : Splitting the data set into training and testing data set .

Step 4 : Applying regression model on training set

Step 5 : Predicting the testing set

Step 6 : Calculating the accuracy of the model

So, we are done with EDA and regression using python. The github link for the code and the data set is here :- https://github.com/kaushal-gupta/Machine_learning_Real_world/tree/master/regression/Medical_cost_prediction

If you like it give a star on github and follow me there for more updates and give claps too.

Thanks for reading !!!
The Basics of EDA (with candy)
By: Rachel Koenig
Tags: Data Science, Candyhierarchy, Pandas Dataframe, Statscandy, Exploratory Data Analysis

The Basics of EDA (with candy)

“The goal is to turn data into information, and information into insight.”

Carly Fiorina, former CEO of Hewlett-Packard

As a data scientist, it is said that we will spend about 80% of our time on EDA. Therefore, as tedious as it may seem, it is a good idea to master the process before moving on to the fun stuff. The process which includes cleaning, sorting, and checking feature correlations to weed out the noise and hopefully gain some insight to what pieces are most useful.

The dataset for this example is the candy hierarchy for 2017 from the Science Creative Quarterly website. You can find and download the csv here if you want to follow along. I’m using Python 3.7 in Jupyter notebook.

Some of the steps I share here will be personal preference and you can modify as you like, however, these first few steps are pretty important to ALWAYS do.

Import:

Pandas — data analysis library, necessary for almost all data manipulation

Numpy — for linear algebra functions, good to have even if you’re not expecting to need it.

Matplotlib — visualization library for plotting and graphs

Seaborn — for more customizable data visualization

Read in your data:

Using pandas pd.read_csv function, enter your data source as a string and save it as a variable which will be the name of your DataFrame going forward. Many people use df, I like to use something more descriptive so i’ll go with candy this time.

Next you’ll want to check the first few rows of the data to get a peak at what you’ll be working with.
Descriptive Statistics: Expectations vs. Reality (Exploratory Data Analysis)
By: Gonçalo Guimarães Gomes
Tags: Data Science, Descriptive Statistics, Statistics, Pandas, Exploratory Data Analysis

An easy descriptive statistics approach to summarize the numeric and categoric data variables through the Measures of Central Tendency and Measures of Spread for every Exploratory Data Analysis process.

Photo by Katherine Hanlon on Unsplash

About the Exploratory Data Analysis (EDA)

EDA is the first step in the data analysis process. It allows us to understand the data we are dealing with by describing and summarizing the dataset’s main characteristics, often through visual methods like bar and pie charts, histograms, boxplots, scatterplots, heatmaps, and many more.

Why is EDA important?

Maximize insight into a dataset (be able to listen to your data)

Uncover underlying structure/patterns

Detect outliers and anomalies

Extract and select important variables

Increase computational effenciency

Test underlying assumptions (e.g. business intuiton)

Moreover, to be capable of exploring and explain the dataset’s features with all its attributes getting insights and efficient numeric summaries of the data, we need help from Descriptive Statistics.

Statistics is divided into two major areas:

Descriptive statistics : describe and summarize data;

: describe and summarize data; Inferential statistics: methods for using sample data to make general conclusions (inferences) about populations.

This tutorial focuses on descriptive statistics of both numerical and categorical variables and is divided into two parts:
Exploratory Data Analysis Regarding Customer Churn in Telecom Industry
By: Earlyan A. Bernatapi
Tags: Data Analytics, Churn, Exploratory Data Analysis, Data Visualization, Pyhton

According to research by Misra et al. (2017), customer churn is affected by 8 factors: network coverage, voice quality, tariff, customer service, value-added service, fair company, friendly company, an innovation company.

Prefix analysis was conducted in addition to using the factors mentioned above by using Exploratory Data Analysis. The dataset used can be obtained freely through Kaggle. So what is Exploratory Data Analysis?

The analysis carried out depends on the type of data contained in the dataset, through 3 proposed methods, i.e., bivariate analysis, correlation analysis, and numeric features analysis, expected to provide insight into existing patterns.

In the bivariate analysis, it can be seen that the number of factors that exist can affect the level of churn. It was found that customers who did not receive technical support, used fiber optic services, and did not receive device protection tended to have high churn rates.

Result of Bivariate Analysis

For correlation analysis, the results obtained that internet service, online security, online backup, device protection, tech support, and streaming are highly correlated features. The most correlated to churn: senior citizen, partner, multiple lines, online backup, monthly charges, and total charges.

Result of Correlation Analysis

On average, those who have churned stayed less than a year (10 months), while those who stayed have been there for 3 years. 75% of those who have churned stayed less than 30 months. A quarter of loyal customers have been here for more than 5 years. It was found that loyal customers have fewer monthly charges.
What are Q-Q plots?
By: Prafful Mishra
Tags: Data Science, Mathematics, Statistics, Exploratory Data Analysis, Data Visualization

In real-world scenarios for data analysis, one of the most important parts is to draw conclusions from the available data. As data collection can be a very costly affair a lot of times, trying to fit the available handful data points as a part of a well known and analyzed distribution can be an effective method to apply pre-defined inferences.

One such method to check whether the available dataset follows a distribution is plotting Quartile-Quartile Plot, popularly known as Q-Q plots. For the ease of understanding, the following explanation will check whether the data available is Normally distributed or not.

Steps to plot a Q-Q graph

The first step towards determining if the data follows Normal Distribution or not is computing the percentiles.

Consider we have a set X with the following data points:

The first step is to make a list sorted in ascending order for the values of x as:

After we arranged the list of all the data points in an ascending ordered list, we need to compute the percentile values for the above list and obtain percentile values for the same(in this scenario, we took 100 percentile values).

Now, we need to get a set which is Normally Distributed (for the sake of this example) as follows and compute the percentiles for this list similar to the set X.

such that the mean for this distribution is 0 and the variance is 1.

We have obtained the list of 100 values for both the sets(ordered X percentiles and theoretical quantiles from Y). Considering the first list as the list of x coordinates and the second as y coordinate, we will plot these points on a graph. This graph is known as Q-Q graph.

Drawing Inference

If the points plotted on the graph lie on a straight line, then we can easily conclude that both the sets of data have similar distribution (In this case we can say that X is Normally Distributed)

Limitation

One of the major limitations for this method is that we need to have huge data points, as making a conclusion on fewer data would not be a wise decision.

Conclusion

This concept can be extended to test any data by replacing the Y set with the desired distribution. The recognition of the distribution can be very useful for drawing already studied inferences about the data.
A Comprehensive Data Analysis on a WhatsApp Group Chat
By: Faaez Razeen
Tags: Data Science, Visualization, Data Analysis, Exploratory Data Analysis, WhatsApp

A Comprehensive Data Analysis on a WhatsApp Group Chat

Photo by Yucel Moran on Unsplash

While thinking of what to do for my next project, an idea suddenly popped up in my head. Why not do data analysis on a WhatsApp group chat of college students and find out interesting insights like the most used emoji, the sentiment score of each person, who swears the most, the most actives times of the day, or does the group use phones during college teaching hours? These would be some interesting insights for sure, more for me than for you since the people in this chat are people I know personally.

Note: I represent each of my friends with two letters, an abbreviation of their name as a way to maintain anonymity.

Data Retrieval & Preprocessing

The first step was to gather the data. WhatsApp allows you to export your chats through a .txt format. Opening this file up, you get messages in a format that looks like this:

Telling lies? No, Papa!

Since WhatsApp texts are multi-line, you cannot just read the file line by line and get each message that you want. Instead, you need a way to identify if a line is a new message or part of an old message. You could do this use regular expressions, but I went forward with a more simple method. I created a function called vali_date(), which returns True if the argument passed is a valid date.

I am extremely proud of how I named this function.

While reading each line, I split it based on a comma and take the first item returned from the split() function. If the line is a new message, the first item would be a valid date, and it will be appended as a new message to the list of messages. If it’s not, the message is part of the previous message, and hence, will be appended to the end of the previous message as one continuous message.

From here it’s just a matter of extracting the necessary details from each line: the date sent, the time sent, the sender, and the message itself. This is just a matter of using some simple string processing functions, mainly split(), and making a DataFrame out of it. If you’re using the same method then please make sure to specify the date and time formats correctly.
COVID-19 Data Visualization using Python
By: Jaskeerat Singh Bhatia
Tags: Data Science, Coronavirus, Covid 19, Exploratory Data Analysis, Data Visualization

Data Visualization is the first step towards getting an insight into a large data set in every data science project. Once the data has been acquired and preprocessed (cleaned and deduplicated), the next step in the Data Science Life Cycle is Exploratory Data Analysis which kicks off with visualization of the data. The aim here is to extract useful information from the data.

I have used Python and its few powerful libraries to achieve the task. Also, I have used Google Colab Notebooks to write the code so as to avoid the hassle of installing any IDE or packages in case you wish to follow along.

Let’s get started

The first step is to open a new Google Colab ipython notebook and import the libraries we require.

import pandas as pd import numpy as np import matplotlib.pyplot as plt import seaborn as sns import plotly.express as px ### for plotting the data on world map

Loading the data

These visualizations are based on data as of May 25, 2020. I have used the daily report data published by John Hopkins University for May 25, 2020. The next part of the code deals with loading the .csv data to our project.

path = 'https://raw.githubusercontent.com/CSSEGISandData/COVID-19/master/csse_covid_19_data/csse_covid_19_daily_reports/05-25-2020.csv' df = pd.read_csv(path) df.info() df.head()

In just two lines in code we have our data loaded and ready for use as a Pandas Dataframe. the next two lines display the information about the data (metadata) i.e., a total of 3409 rows of data and 11 columns. It also gives us a preview of the first five rows.

Preprocessing the data

Now since our data has loaded successfully, the next step is to preprocess the data before using it for plotting. It will include :

Removing superfluous columns like ‘FIPS’, ‘Admin2', ‘Last_Update’ (since all the data is for single-day — 25th May).

Removing columns ‘Province_State’ and ‘Combined_Key’ since statewide data is not available for all the countries.

Grouping together data by ‘Country_Region’ and rename the column to ‘Country’

df.drop(['FIPS', 'Admin2','Last_Update','Province_State', 'Combined_Key'], axis=1, inplace=True) df.rename(columns={'Country_Region': "Country"}, inplace=True) df.head()

The data can be grouped together by the ‘groupby’ function of the dataframe. It is similar to the GROUPBY statement in SQL.

world = df.groupby("Country")['Confirmed','Active','Recovered','Deaths'].sum().reset_index() world.head()

Finally our data is cleaned and ready to use.
Data Science Workflow Terminology
By: Chandler O'Neal
Tags: Data Science Workflow, Data Wrangling, Data Mining, Exploratory Data Analysis, Preprocessing

Words to clarify:

Let’s take a look at the potluck, word cloud, what have you, of the various terms that apply to transforming data from a messy state into a more organized form used for analyzing and generating predictions from the data.

Data Mining, Data Collecting, Exploratory Data Analysis, Data Preparation, Preprocessing, Feature Engineering.

Data Mining

Data mining is the overarching process of collecting, analysing, and applying machine learning algorithms to data.

Note:

The visual above is defined as an “industry-proven way to guide your data mining efforts”. The CRISP-DM is the same idea as the visual below — an organized process of data mining.

Data Wrangling / or / Preprocessing / or / Cleaning

Data wrangling is the process of cleaning, structuring, and enriching raw data into a more usable and understandable format for decision-making in less time.

Essentially the restructuring and reformatting of data to increase its overall quality (usability).

EDA (Exploratory Data Analysis)

Exploratory Data Analysis is useful for scoping the dataset as a whole — useful to follow data cleaning with EDA to evaluate better organized data.

Useful to:
A Function That Makes Your Initial EDA a Breeze
By: Katie Sylvia
Tags: Data Science, Python, Graphics, Exploratory Data Analysis, Seaborn

Photo by Joshua Aragon on Unsplash

EDA (Exploratory Data Analysis) is a crucial part of the Data Science journey — it allows us to understand the given data and guides us in our modeling decisions. Getting a “feel” for your data before making any assumptions can be critical in formulating valuable insights. The bad news? It can often be time-consuming.

My last project used the data provided by the Austin Animal Shelter from the City of Austin Open Data Portal. A predictive model was used to determine if an animal were to be adopted based off of their intake information (animal type, age, breed, color, month of intake, etc.). With over 150,000 animals in the shelter, it was important to explore these different distributions in order to get an understanding of the intakes in Austin.

As I started exploring and creating bar plots, I found myself writing the same code over and over again. Often when this happens I ask myself, “Should I create a function instead?” Ultimately, creating this function saved me a lot of time. Work smarter, not harder. Here’s the function I created:

def initial_eda(df):

# List of categorical columns

cat_cols = df.select_dtypes('object').columns



for col in cat_cols:

# Formatting

column_name = col.title().replace('_', ' ')

title= 'Distribution of ' + column_name



# Unique values <= 12 to avoid overcrowding

if len(df[col].value_counts())<=12:

plt.figure(figsize = (8, 6))

sns.countplot(x=df[col],

data=df,

palette="Paired",

order = df[col].value_counts().index)

plt.title(title, fontsize = 18, pad = 12)

plt.xlabel(column_name, fontsize = 15)

plt.xticks(rotation=20)

plt.ylabel("Frequency",fontsize = 15)

plt.show();

else:

print(f'{column_name} has {len(df[col].value_counts())} unique values. Alternative EDA should be considered.')

return

When running the function on my data, here was the output:
Making Sense of 33 Million Rows of Instacart Data: Exploratory Data Analysis Using Plotly
By: Vijay Vankayalapati
Tags: Data Science, Python, Plotly, Exploratory Data Analysis, Data Visualization

Making Sense of 33 Million Rows of Instacart Data: Exploratory Data Analysis Using Plotly Vijay Vankayalapati Follow Aug 22 · 6 min read

Source: Zdnet.com

Instacart is one of the hottest Silicon Valley startups of the last decade and is now valued at whopping $39 billion. In 2017 they hosted a Kaggle competition challenging data scientists to predict which products will be reordered for a given user. For this purpose they open sourced a dataset comprising anonymised information about 3 million+ orders from their 200,000+ users.

The first step in understanding data is to perform exploratory data analysis on the dataset and have a cursory insight of it. We shall use python’s plotly library for data visualisation and also datapane library to make our plotly charts interactive at some places here.

The dataset in question can be downloaded from Kaggle website. It contains 6 CSV files holding data pertaining to orders, products, departments and aisles.

Please note that I chopped off parts of plotly code in github gists that is related to embellishing the charts. This is to reduce the size of this page (it is already too long). Complete code can be found on my github.
Simple exploratory data analysis on Marijuana Dataset part -1
By: Das Showmik
Tags: Exploratory Data Analysis, Visualization, Seaborn

Greetings . Undoubtedly Marijuana has significant impact of human behavior analysis . It will be a great study to see how cannabis effects human behavior and mood as well as which strains has the most impact on which type of mood / feeling of different user . The data set which I found is a very basic

First , let us look at the data set . The data set contains six columns which are respectively “Strain” ( the name of strain), “Type”(3 major type of cannabis ), “Rating”(User Rating) , “Effects” ( self explanatory), “ Flavor”(Taste ) , “Description”(Origin of the strain) .

In this part I will work only with the Type , effects and flavor column just to see some general relationships.

From the “Effects” column , I try to separate the individual effect and split it into different columns . With a simple count plot with the mighty seaborn in python , we try to capture the primary effects induced by cannabis .

Primary effects of marijuana described by the user

We can see that the most 4 common effects described by the cannabis user are : Relaxed , Happy ,Euphoric, Sleepy ( Though I would go for Hungry at first (!) ) As different cannabis strain have different effects on human , it will be nice to see strain wise effects on user.

So what about the flavor . The flavor column has 33 unique type of flavors and the most common ones are as followed :

Countplot on common cannabis strain flavor

We can see from the plot that the most common flavors are : Earthy, Sweet, Citrus , Berry , etc .

The next thing which can be interesting to see is how the spectrum changes with respect to different cannabis strains . This time also we invoke seaborn to the rescue!

Sativa effects spectrum count plot

Sativa makes people Happy and Uplifted . Also people have described that sativa worked as source of energy , focus and creativity . Though there are cases that Sativa also induces paranoia (!) but according to the dataset these are the results.

Indica effects spectrum count plot

Indica has some different results . Using Indica strains make people more relaxed , euphoric and sleepy . Also people tends to get more hungry compared to sativa users. We can also see that focus is depreciated as well as creativity . This is an interesting insight , is not it ?

From the hybrid strains , well the effects are also a mixture of sativa and Indica effects . For example hybrid takes the most common effects of cannabis , happy and uplifted merging with relaxed and euphoric feeling from Indica.

It was intended for a short write up and also this is not the bible for exploratory data analysis . On the next part , I would like to get some insights from the origin of the marijuana strains using natural language processing.

Data source : https://github.com/kushyapp/cannabis-dataset/
The Data Analysis with A Dataset
By: Md Shakib Mondal
Tags: Data Science, Eda, Data Analysis, Exploratory Data Analysis, Dataframes

The word, “DATA ANALYSIS” is a common word in the world of Data Science. If you ask what is data analysis?… The Formal definition will be….

Data analysis is a process of inspecting, cleansing, transforming and modeling data with the goal of discovering useful information, informing conclusions and supporting decision-making

if you are interested more about “The Data Analysis” process, you can go through my blog on the above topic. link:-

https://medium.com/@shakibmondal/the-data-analysis-process-6bf501ad47a8

All the coding of this blog is done in a python notebook file. Are you interested in that?

here is the GitHub project link for you:-

https://github.com/SkeyRahaman/Data_Analysis_on_Startups_of_India

Basically, there are 5 major steps in the process of Data Analysis and they are:-

1. Asking the Right Question. 2. Data Wrangling or Data Munging. 3. Exploratory Data Analysis. 4. Drawing Conclusions. 5. Communicating Results.

We will be going deep into these processes, let’s start from the first part.

1. Asking the Right Questions.

This is the process where we set our aim for this Data analysis. After seeing the dataset, some of the question that i may ask the dataset are.

What is the relation between Funding ecosystem and time?

What are the roles of cities in Indian Startups?

What is the top favored Industry for funding?

Who are the major investors?

What are the types of funding StartUps get in India?

The average amount that a startup get in India city wise?

Which Startup has got the Highest Funding in India?

2. Data Wrangling or Data Munging.

This step is also known as “Data Pre-processing”. This Step can also be divided into three major part as:-
Starbucks Customer Analysis and Best Offers
By: Adham Elkhouly
Tags: Starbucks, Data Science, Predictive Analytics, Machine Learning, Exploratory Data Analysis

II. Exploratory Data Analysis

1. Offers Sent out

Offers sent out are of 3 types: BOGO (buy one, get one), informational, and discount.

We can see that discount offers are the least offered by Starbucks at 20%. While the other 2 come equally at 40%. Informational offers do not require an action from the user side, while the other 2 do.

2. Age, Income and Gender Distribution of Customers

There were some very interesting findings here that I will attempt to wrangle and explain.

Simplest one being the gender distribution, where most rewards customers tend to be male at about 58%, females coming second at about 40%, and other genders at 2%.

The next 2 plots will be analyzed simultaneously.

Interestingly, most Starbucks rewards customers lie in the 50–59 age group (I personally thought it would be in the 20–39 range!). However, if we look at the plot above that, we can see the income levels also increasing with each age group.

Therefore, the explanation here can be actually that as income increases (tends to increase with age), customers enroll in the rewards program.

3. Distribution of Offers Received by Customers

There are a total of 10 offers sent out by Starbucks, and it can be seen that they are equally distributed.

This is especially crucial later, so that our predictive model does not skew towards any certain offer, just because it is sent out more.

4. Completed and Viewed Offers

Offer events in the data set are of three types as explained above. Now we will look at the relation between viewing/clicking on an offer and completing .

We can notice that are highly correlated. Highly completed offers, are those that are also highly viewed.

Offers 2 and 7 are the exceptions, since they are informational offers, and cannot technically be completed, just viewed.
Geektrust update - we’ll build your product for you. Interested?
By: Krishnan Nair
Tags: Startup, Entrepreneurship, Software Development

So, COVID-19 huh?

For a bootstrapped company like us that plans with a 3 month runway, this is a deathknell. However in the larger scheme of things with the loss of lives and livelihoods all around us, we are grateful for our health and our homes. Businesses can always be rebuilt.

COVID effect. The number of interviews is our north star metric. That has plummeted of late.

But enough of our worries! Here we are, offering a world-class tech team to help build and take your product to the market for you. We are committing a time period of upto 6 months to work with you.

What’s the team?

We’ve spent majority of our working lives at ThoughtWorks. 4 of us have spent close to 10 years there. We’ve built world-class software across domains and tech stacks. Go, NodeJS, Java, Python, AngularJS are what we’ve worked on mostly, but we see new tech as something that can be picked up fairly quickly.

We have a 5-member team available — 3 developers, and 2 of us who are good with product and marketing.

Dhanush Gopinath, CTO at Geektrust who loves & lives to code. Ex-ESPN & Altair. Big time proponent of Go (we use Go at Geektrust). Has worked on Java, NodeJS, Python, Groovy and Angular.

Alok Simha, Principal Developer, ex-ThoughtWorks, ex-Scripbox. Has worked on Java, C#, Go, NodeJS, Angular. Worked at ThoughtWorks locations across the world.

Gaurav Aditya, Developer, IIT-Bombay post grad, ex-eClerx, ex-Brillio. Joined Geektrust a year ago. Has worked on NodeJS, Go, and Angular.

Sneha Jain, CPO at Geektrust. Product manager & project manager at ThoughtWorks before Geektrust. Managed products for clients in India, UK, Singapore and the US.

Krishnan Nair, CEO at Geektrust, ex Head of Delivery at ThoughtWorks. Worked at Bangalore, London, San Francisco, San Antonio as agile consultant for ThoughtWorks in the past.

In the not-so-distant past, we’ve helped a leading Indian fintech startup build some key parts of their product. If you’re interested, you can reach us at hello@geektrust.in. When you write in, do write in detail about yourself, your company and the product. Thank you!
New for Agents: ListPacks®, Premium Shareables, ListReports LIVE, and more!
By: Ryan Terrigno
Tags: Real Estate, Startup, Tech, Technology

New for Agents: ListPacks®, Premium Shareables, ListReports LIVE, and more!

From ListPacks® to ListReports LIVE and more, you won’t want to miss this update on our newest features for agents.

ListPacks®

Instantly share curated, customizable collections of listings on social media or directly with your buyers.

Capturing new leads has never been easier, or more delightful. With just a few clicks, agents can now share beautiful “packs” of listings with their buyer community.

Choose from pre-made categories such as “Homes with pools” and“Single-level homes”, OR set simple preferences to create your own custom ListPacks.

Everybody enjoys looking at homes — serious buyers and lookie-loos alike. ListPacks® provide consumers a truly differentiated experience with unique features including our interactive neighborhood infographics. Even better, each listing prominently displays your contact info, keeping viewers tightly connected to you when they’re ready to express interest in a particular property.

Ready to get started? Click here to start sharing ListPacks® now!
Is watching Netflix halal or haram?
By: Dr. Saad
Tags: Marketing, Netflix, Startup, Religion, Business

In general, watching Netflix is not haram.

The/a specific content could/can be haram. There are tons of content on Netflix. There are haram contents available and also halal contents available. It depends on you which you pick, halal or haram. You should avoid watching haram contents and you are allowed to watch halal contents.

The Prophet Muhammad (peace be upon him) said:

“Verily, the lawful (Halal) is clear and the unlawful (Haram) is clear.”-(Sahih Bukhari 52, Sahih Muslim 1599).

The Quran said:

‘Say, “My Lord has only forbidden immoralities — what is apparent of them and what is concealed — and sin, and oppression without right, and that you associate with Allah that for which He has not sent down authority, and what you say about Allah that which you do not know.”-(Quran 7:33).

Halal is an Arabic word. Halal is something which is lawful or permissible according to Islam. Haram is also an Arabic word. Haram is something which is unlawful or impermissible according to Islam.

A person should do/have etc which is halal and he/she should avoid/ignore everything that is haram. Many people believe that Islam made haram most of the things of the world, which is totally wrong concept. Everything in/of the world is halal. Yes, I am repeating again, everything in/of the world is halal. Only a hand-counted few things are haram that are/is immoral or harmful to/for us.

Note: I am not promoting Netflix, nor any product of Netflix.
Programming History: The Influence of Algol on Modern Programming Languages (Part 1)
By: Mike Mcmillan
Tags: Python, History, JavaScript, Startup, Programming

A Brief History of Programming Languages Up to 1960

The first attempts at creating high-level computer programming languages — languages that significantly simplified the act of programming from what it took to write code in machine language (or even assembly language) — began in earnest in the late 1940s and early 1950s. If you’re interested in reading more about these early attempts to create high-level languages, you should read the paper, “Early Development of Programming Languages,” (EDPL) written by Donald Knuth and Luis Trabb Pardo.

In that paper, Knuth and Pardo introduced the TPK algorithm, which they used to compare the early high-level languages. I’m going to use this algorithm to describe some of the features of both the Algol language and its most popular precursor, Fortran, so here is the algorithm in JavaScript (using the Mozilla Spidermonkey version of JavaScript):

The first high-level language to make a true mark on computing (and a language that’s still in use today) was Fortran. Here is the TPK algorithm in Fortran (the version introduced about 1957, according to Knuth in the EDPL paper):

FUNF(T) = SQRTF(BASF(T))+5.0*T**3

DIMENSION A(11)

1 FORMAT(6F12.4)

READ 1, A

DO 10 J = 1, 11

I = 11 – J

Y = FUNF(A(I+1))

IF (400.0 – Y) 4, 8, 8

4 PRINT 5, I

5 FORMAT(I10, 10H TOO LARGE)

GOTO 10

8 PRINT 9, I, Y

9 FORMAT(I10, 10H TOO LARGE)

10 CONTINUE

STOP 52525

This program does a good job of demonstrating the state of computer programming at that time. It includes a function definition but it’s written as a line of the program. The program uses arrays, as well as formatted input and output. Fortran has one type of loop, the DO loop. There is an if statement that uses a computed GOTO when deciding which program line to execute. And there’s an infamous, explicit GOTO that helps the loop iterate by skipping over a few statements that don’t always need to execute.
Lenna Lim, A New Hire, A New Milestone for the Content Marketing Team!
By: Content Marketing Insights
Tags: Content Marketing, HR, Culture, Startup, Digital Pr

By: Jeremy Chew

Many times, people ask, what do you do in the content marketing team?

There are many ways of answering it. Some would say that we provide value to the business from an external SEO manner, others would say we function like a public relations or communications department.

But in our team, our goal in mind is to be Innovative Leaders of Creative Digital Content Marketing and to Inspire others in the region with our work!

This is a truly amazing goal and an exciting journey for us.

While we are all in this together, I was tasked to find a new hire that will fill a position in our team for the HK & Chinese market.

And boy, it has not been an easy task.

Why so? The requirements of this role extend beyond the standard qualification, industry experience or mere intelligence.

To find people with the potential to become an Innovative Leader, it is imperative the candidate displays evidence of grit, empathy, humility, passionate about continuous learning, creativity, openness in receiving feedback & confidence in providing feedback.

Many candidates fit the bill on paper, but very few showed potential as an Innovative Leader.

And one day, Lenna Lim showed up at our door (figuratively speaking) and approached our People Ops hiring team asking if we have a job vacancy.

The hiring team contacted me, and the rest was history!

As our first Content Marketer who can both read & write in Mandarin, the addition of Lenna Lim marks a new milestone for our team & opens the door to a brand-new world of new types of content!

The team has been highly impressed with her passion in storytelling, a positive attitude & a strong desire in continuous learning.

With this simple short write up, the Content Marketing team is excited to have Lenna Lim join our team as we brave through the world of creative content together!

Welcome, Lenna! Let us embark on this thrilling content marketing journey together!

By the time this write up is published, Lenna has been with us for about three weeks. Here are her thoughts during her time with us so far.

“Time flies! I can’t believe that I’ve been with iPrice for 3 weeks already! Although the Movement Control Order (MCO) in Malaysia has restricted our meeting & work, it did not stop the enthusiasm of my team members. I could fully feel the positive attitude, respectful communication & group commitment from my team members!”

“What impressed me most were the onboarding training classes, where they put in utmost effort in teaching me skills & knowledge for me to be successful in my role. Whenever I face problems, they were immediate in giving me a helping hand. I believe talent can help you win games, but teamwork & intelligence helps you win championships! I’m looking forward to accomplishing both team & company missions under our leader’s guidance in the Content Marketing team! Glad to be an iPricer!”
LinkedIn Posts Best-Practices Guide for 2020
By: Darragh Grove-White
Tags: Linkedin Best Practices, Entrepreneur, Growth Hacking, Startup, B2B

This LinkedIn posts best-practices guide will show you how to make high ROI LinkedIn posts with your existing content.

The LinkedIn posts best-practices guide for 2020 is an excerpt from the free LinkedIn pods training course.

Why quality engagement pods need quality posts

Attention-grabbing headlines with examples

Attention grabbing headlines cheat sheet supplemental in the link below:

Making strategic use of the first 3 lines

APP: Agree, promise, preview what’s to come

Encourage dreams, justify failures, allay fears, throw rocks at enemies, confirm suspicions

Use short sentences and paragraphs for your posts

Grade 8 reading level

Supplemental Writing Apps

Hemmingway app[FREE]

Pro-writing aid [PAID]

Give a lot of value up front for free

Show your human side

How to use emojis on LinkedIn

Address your prospect’s pain-points and aspirations in their words and language

How to increase engagement by asking questions in posts

Trade engagement for gated content

LinkedIn posts best-practices for hashtags

Tag your company in posts (and other people when appropriate.)

The next email will be your link to take the quiz and become an official community member!

If you have not received the email with your quiz link, please reply to a previous email you’ve received and we’ll get you access to it.
We’ve got IDEAS!
By: Andrei Cosmin Munteanu
Tags: Startup, Ideas, Team

The mentors have decided. HERE is the full list of ideas and team leads for each of them:

1. ArtOn, Atila Gombos (idea Owner)

2. AutismVoiceLine, Nicoleta Mariana Orlea (idea Owner)

3. Bioteq, Cosmin Poieana (idea Owner)

4. Bit Triage (temporary), Daniel Stefan Visan (idea Owner)

5. BuddyUp, Miruna Toader (idea Owner)

6. BYEOVID, andreea vasilcin (idea Owner)

7. Career education chatbot, Madalina Partoaca (idea Owner)

8. Citizen scientists agains corona, Dragos Bordescu (idea Owner)

9. Conference at court, mihail.adc@gmail.com (idea Owner)

10. Covid-19 Destination Form and Traking, Apostol Stefan Mihai (idea Owner)

11. DeepCrisis, Yacine Bouchrika (idea Owner)

12. Defend-wear, Razvan Valceanu (idea Owner)

13. Defense Viruses, Petre Iulian Timis (idea Owner)

14. Digital school for doctors, Cucos Florin (idea Owner)

15. Epidemologic Tracker, Pantilimon Andrei Lucian (idea Owner)

16. FiiLocal, Ana-Arina Raileanu (idea Owner)

17. Help for Hire, Tushar Advani (idea Owner)

18. iDareYou, Meder Robert (idea Owner)

19. InhlAid, Razvan Mihai Cretu (idea Owner)

20. Life e-Learning, Anna Mera (idea Owner)

21. LOVESCOPE, Anda Tepei (idea Owner)

22. Mask Radar web platform, Tudor Vlad Beleuta (idea Owner)

23. MediNav, Adina Wanda Fiat (idea Owner)

24. MHFA ( Mental Health First Aid), Cristina Loredana Nina (idea Owner)

25. News AI (name still pending), Jimmy Nastase (idea Owner)

26. Notary Online, Alexandru Motoonu (idea Owner)

27. Oldie but Goldie, Nemeș Silvia Amalia (idea Owner)

28. One Time Jobs Platform, Diana Ioana Sterea (idea Owner)

29. Online Education for Everyone — remove the veil of inequality, Mariana Ciuca (idea Owner)

30. Pedal tap, David-Florentin Chirita (idea Owner)

31. Plasmatch, Tiberius Pircalabu (idea Owner)

32. quarantiMe, stefan filip (idea Owner)

33. Rebels with a cause., Dean Smith (idea Owner)

34. Scarecity, Sebastian Macovei (idea Owner)

35. SUPPLIED, Munteanu Iulian-Alexandru (idea Owner)

36. SusTrack, Alex Iuonut (idea Owner)

37. Tech2Grow, Adrian Cioroianu (idea Owner)

38. Teleradiologie, ANDREEA LEFTER (idea Owner)

39. TeleVaccine, VizeAurelian Dorin (idea Owner)

40. The internet of gardens, Cindea Laurentiu Bogdan (idea Owner)

41. Tooting, Madalina Marinescu (idea Owner)

42. UV-C Disinfection/sterilization device, Andrut Alin Liviu (idea Owner)

43. Virtual Medical Congress, Ion-Gheorghe PETROVAI (idea Owner)

44. WaaS — wellbeing as a service, Adrian BALASKO (idea Owner)

45. Wastely, Liviu Tudoran (idea Owner)

The participants overdid themselves. We’ve received over 120 wonderful submissions and the 19 mentors had a hard time selecting. The mentors have selected 45 ideas based on:

The uniqueness of the idea,

How impactful the idea is during COVID-19.

Some ideas weren’t selected, but we believe that nothing goes to waste. This is an opportunity to meet new people and make new connections. Who knows? Maybe they’ll meet their new best friend, and you, your future team.

It’s time to team up!