Exploratory Data Analysis (EDA) — Hands-on NYC Airbnb Dataset
By: Rajvi Shah
Tags: Data Analysis, Kaggle, Data Science, Data Visualization, Machine Learning

Dataset

Download the dataset from: NYC Airbnb data

REMEMBER: The goal is to predict rental prices for given data.

Note: I have just explained some steps for EDA in NYC Airbnb to get the intuition of approaching EDA for any problem. To know more about this dataset, check on Github.

Try to understand the data.

For example: Ask questions like;

How many columns are there in the dataset?

What are the datatypes of those columns?

How many data-points contain some null values? i.e. missing values

How to replace these missing values?

How many data-points are there in the dataset?

What is the range of all columns with numbers?

What is the mean, median, mode, percentile, etc. values of features?

How many values are there for this particular feature in some range?

Try to answer or verify these questions or similar kind of questions as follows:

Now, check the output & note down the observation. For example:

The output of data.dtypes would be as follows;

Reference: Author

And the observation can be:

some columns contain text so maybe there would be a need to use BoW, tfidf, w2vec, etc. for data modeling.

Similarly, the output of data.isnull().sum()is;

Reference: Author

And the observation can be:

There are 4 columns that have null values, have to replace them as it might affect later while considering the remaining values of the same column.

So here, the next is to replace these null values as shown in the above code snippet.

Now, to know that the range of data-points for a particular feature, univariate plotting using Histogram is a good option.

Output:

Reference: Author

Observation:

Most of the prices are less than 1000.

Now,

Reference: Author

Observation:

In all, there are 239 data-points with a rental price > 1000.

These are either super lavish listings or there can an error during input. Nonetheless, since these records are skewing our data, we can treat them as outliers and drop them.

Output:

Reference: Author

Output:

Reference: Author

Observations:

This is a kind of Gaussian distribution for rental values < 250.

Considering threshold = 250 is a good option.

Likewise, trying to know different features of the dataset.

Output:

Reference: Author

Observation:

Almost 90% of the data is covered.

Output:

Reference: Author

Observation:

There are 5 groups & 85% are covered by Manhattan & Brooklyn.

Output:

Reference: Author

Here we can note that Brooklyn and Manhattan tend to have more listings with price > 150. Also, most listings above price > 100 are entire home types followed by private rooms and shared rooms which is the cheapest.

Following to exploration of features, it is good to drop features that are not important for prediction.

Now,

Also, checking the correlation between features is important;

Exploring room type feature;

Output:

Reference: Author

Mostly, the room type is either an entire home or a private room.

Detecting outliers is also a crucial thing during EDA. Now, exploring, minimum_nights column;

Output:

Reference: Author

It’s quite weird, the range is between 1 night to 1250 nights. From the graph, most of the nights are under 100.

Output:

Reference: Author

Only 747 data points have a number of minimum_nights > 30, so replacing all of those with 30.

Now, plotting correlation graph for all features;

Output:

Reference: Author

This graph can depict a lot about important features to achieve the goal to solve the problem. Now, the data can be used to build a simple linear regression model, and then one can decide the approach for data modeling.

You can find the source code from Github.