Is there life outside Excel Spreadsheets?
By: Christian Eloysio
Tags: Analysis, Data, San Francisco, Data Science, Airbnb

Data extraction

All analyzed data was extracted from Inside Airbnb Portal.

For this initial exploratory data analysis, I chose the following file:

listing.csv - Summary information and metrics for listings in San Francisco (good for visualizations).

Data Analysis

At first, as we would do in excel, we will try to broaden the situational awareness around the dataset. I will leave a dictionary of variables for better understanding.

Dictionary

id - Property unique identifier

- Property unique identifier name - Property name/description

- Property name/description host_id - Host unique identifier

- Host unique identifier host_name - Host name

- Host name neighbourhood_group - this column has no valid entries and will be deleted

- this column has no valid entries and will be deleted neighbourhood - Propety's neighbourhood name

- Propety's neighbourhood name latitude - Property's latitude

- Property's latitude longitude - Property's longitude

- Property's longitude room_type - Type of accomodation that is offered (Entire home/apt, Private room, etc)

- Type of accomodation that is offered (Entire home/apt, Private room, etc) price - property's price per night

- property's price per night minimum_nights - Property's minimum nights

- Property's minimum nights number_of_reviews - Property's reviews count

- Property's reviews count last_review - Last review date

- Last review date reviews_per_month - Number of reviwes per month

- Number of reviwes per month calculated_host_listings_count - Number of propeties from the same host

- Number of propeties from the same host availability_365 - Available days in the last 365 days

Let’s try to answer some questions regarding the dataset:

Q1. How many variables and entries do our dataset contains? What are the types of the variables?

Entries: 6611

Variables: 16 id int64

name object

host_id int64

host_name object

neighbourhood_group float64

neighbourhood object

latitude float64

longitude float64

room_type object

price int64

minimum_nights int64

number_of_reviews int64

last_review object

reviews_per_month float64

calculated_host_listings_count int64

availability_365 int64

dtype: object

As we can see above, our dataset contains 6611 entries spread through 16 different columns (variables).

Regarding the type of each variable, it is important to check if the numeric columns have the proper corresponding type. Otherwise, calculations won’t be possible.

Q2. What is the missing value percentage in each variable?

Using the missingno library, we can first visualize the missing values in each variable. Each white line represents a missing value.

Missing value visually represented

Afterward, we can check the percentage of missing values in each column:

neighbourhood_group 1.000000

reviews_per_month 0.232189

last_review 0.232189

availability_365 0.000000 calculated_host_listings_count 0.000000

number_of_reviews 0.000000

minimum_nights 0.000000

price 0.000000

room_type 0.000000

longitude 0.000000

latitude 0.000000

neighbourhood 0.000000

host_name 0.000000

host_id 0.000000

name 0.000000

id 0.000000

dtype: float64

The whole neighbourhood_group variable is missing, while the reviews_per_month and last_review variables both have 23% of their values missing.

Q3. What is the distribution of the variables?

To check the distribution of the variables, I will plot the histogram for the numeric variables:

Histogram for numeric variables

As it is shown, some variables as minimum_nights and price have uneven distributions. That is probably due to the outliers in the dataset. Hence, we will delete those in the next step.

Q4. Are there Outliers on the Dataset?

An Outlier is an observation that lies at an abnormal distance from other values in a random sample from a population. In other words, it´s a piece of data that is so different from the rest that can unbalance the results of our analysis. Thus, to have a more precise dataset, we will seek to remove the outliers.

Let´s first take a look at a statistical summary for the dataset:

Dataset Statistical Summary

Let´s take a closer look at the price variable:

Although 75% of the entries are under 245, the maximum value is 25000.

The minimum value is 0, which is impossible if someone is renting a property.

Related to the minimum_nights variable:

Although 75% of the entries are below 30, the maximum value is 1125.

The median (50%) is equal to the 75th percentile (75%). There’s probably a strong concentration of data among this value.

To visualize the outliers in the dataset, we can plot the “boxplot”.

Long story short, the black dots represent outliers.

Plotting the boxplot for price :

Boxplot for price

Plotting the boxplot for minimum_nights :

Boxplot for minimun_nights

Reaffirming what we saw in the statistical summary, in both cases ( price and minimum_nights ) there are values that lie at an abnormal distance from other values, and will be considered as outliers in this analysis.

There are different methods to mathematically identify the outliers. Here, we will identify them using the IQR Score, explained in the image below:

Statistical method to define Outliers

In a nutshell:

IQR = 75th Percentile — 25th Percentile

Superior limit = 75th Percentile + 1,5*IQR

Inferior limit = 25th Percentile — 1,5*IQR

The outliers are the values above the superior limit or below the inferior limit. These are the ones that should be deleted

Identifying the Outliers

Calculating IQR, superior limit, and inferior limit for the price variable:

IQR for price: 150.0

Superior limit for price: 470.0

Inferior limit for price: -130.0

As we can’t have negative values for price, nor values that are equal to 0, only values greater than zero and below the superior limit should be considered.

Now, let´s check the percentage of values that are equal to 0, or above the superior limit:

% Equal to 0: 0.03%

% Greater than superior limit: 7.01%

Deleting the outliers from the dataset:

Initial number of entries: 6611

Entries after cleaning price: 6145

Running the same process for minimum_nights :

IQR for minimum_nights: 28.0

Superior limit for minimun_nights: 72.0

Inferior limit for minimun_nights: -40.0

% Greater than superior limit: 1.6% Entries after cleaning minimum_nights and price: 6042

Finally, let´s look at the distribution for price and minimum_nights before and after cleaning the outliers.

price before cleaning outliers:

Histogram for price before deleting outliers

price after cleaning outliers:

Histogram for price after deleting outliers

minimum_nights before cleaninig outliers:

Histogram for minimum_nights before deleting outliers

minimum_nights after cleaning outliers:

Histogram for minimum_nights after deleing outliers

To keep the dataset cleaner, I will also delete the neighbourhood_group variable since it has no valid values.

After treating the dataset, this is the new statistical summary:

Statistical Summary After Deleting outliers in price and minimum_nights

Some highlights:

In price , the mean is way closer to the median than before.

, the mean is way closer to the median than before. The new maximum price value is 470.

value is 470. The new maximum minimum_nights value is 62.

value is 62. Without the outliers, we can have a deeper understanding of the real distribution of data in both price and minimum_nights .

As we initial stated:

There were outliers in price affecting our analysis.

affecting our analysis. There were outliers in minimum_nights affecting our analysis.

affecting our analysis. There is a concentration of minimum_nights data among 30 days. A hypothesis for this would be that people in San Francisco generally rent their houses for a specific season that is a month-long. That can be analyzed with further research, which is not in the scope of this study.

Q4. What is the correlation between variables?

A correlation implies that there is a relation between two variables in the dataset.

This relation can be measured, and we will be doing that by:

Creating a correlation matrix.

Creating a heatmap using the seaborn library.

Creating the correlation matrix:

Correlation matrix for the dataset

Plotting the heatmap:

Data by Christian Eloysio, 2021

As it can be observed, there is no strong correlation between the variables that were listed above.

Q5. What is the most common room type?

If we were going to rent a property in San Francisco, it would be useful to know which type is the most common one.

We can run an analysis in the room_type variable to address this question.

First, checking the total number of entries in all the types of room available:

Entire home/apt 3660

Private room 2135

Shared room 171

Hotel room 76

Name: room_type, dtype: int64

And checking the percentage of each type of room in the Dataset:

Entire home/apt 60,57%

Private room 35,33%

Shared room 28,30%

Hotel room 12,57%

Name: room_type, dtype: float64

Running this analysis, we could notice that 60% of the rentals in San Francisco are Entire home/apt, which might explain the reason why the minimum nights for a great part of the locations are 30 days: people are generally looking for a place to live and pay a monthly rent.

Even though the majority of locations offered are Entire home/apt, that does not mean that this is the one that is mostly rented.

We can investigate that by checking the median value for availabiblty_365 in each room type:

room_type

Private room 124

Entire home/apt 140

Hotel room 194

Shared room 209

Name: availability_365, dtype: int64

While Entire home/apt is the most common type offered, private room is the least available one. That may indicate that there is greater demand for this type of room in San Francisco.

Q6. What is the most expensive neighborhood in San Francisco?

When renting a place to stay, price is generally one of the top concerns.

Let´s check the mean price per neighbourhood:

neighbourhood

Twin Peaks 215.065217

Marina 199.708333

Castro/Upper Market 198.070866

Potrero Hill 197.846154

Glen Park 192.234043

Presidio 191.857143

Chinatown 186.187050

South of Market 185.183771

Golden Gate Park 183.750000

Noe Valley 179.712329

Name: price, dtype: float64

We can also visualize the properties based on their longitude and latitude, colored by price:

Data by Christian Eloysio, 2021

Credits to https://www.kopa.co/blog/posts/a-guide-to-the-neighborhoods-of-san-francisco

According to the analysis above, although places like Twin Peaks and Marina can be expensive and have the highest mean price, we can still find cheaper places to stay in these areas.

Conclusions

With just a few lines of code we were able to:

Import data and all the necessary libraries.

Discover that our dataset has 16 variables and more than 6000 entries.

Discover that the variable neighbourhood_group has no valid values.

has no valid values. Calculate and delete outliers for price and minimum_nights .

and . Understand the real distribution for price and minimum_nights .

and . Check the correlation between variables.

Discover that while the most common room_type is Entire room/apt, the least available one is Private room.

is Entire room/apt, the least available one is Private room. Discover that Twin Peaks has the highest price mean among the neigbourhoods in San Francisco.

This is just the tip of the Iceberg. I hope the power of an analysis using Python is clear.

If “Data is the new bacon”, you need proper silverware to taste all the flavor of it. Excel can certainly be useful in some cases, but tools like Python will help you to extract more value from your data.

Thanks for reading! You can check the complete analysis on my Github.