Use Python to Upload Your First Dataset on Kaggle— Taiwan Housing Project (1/2)
By: Peiyuan Chien
Tags: Data Science, Python, Programming, Housing, Kaggle

Step 1: Collect Data from the Open Data Platform

On the website of the Ministry of the Interior, we can download the transaction records of the real estate based on the region and timeframe.

Step 2: Observe What We Collected

What can we see from the dataset below?

The first row of data is actually English translation of the column names

Some missing values (NaN)

In the column transaction year month and day (交易年月日), the year information follows the civil calendar rather than the Gregorian calendar

Text contents are in Chinese

Original Dataset

Step 3: Preprocess Data

According to our observation, we will start to preprocess the dataset step by step.

Drop the First Row

As mentioned, the first row is the record of column names in English, so drop it.

df.drop(0, axis=0)

Rename Column Names

To make the dataset easier to use, we translate all the columns in English.

COL_NAME = ['district', 'transaction_type', 'address', 'land_shift_area', 'urban_land_use', 'non_urban_use', 'non_urban_use_code', 'transaction_date', 'transaction_number', 'shift_level', 'total_levels', 'building_state', 'main_use', 'main_building_material', 'complete_year', 'building_shift_total_area', 'num_room', 'num_hall', 'num_toilet', 'num_partition', 'management_org', 'total_ntd', 'unit_ntd', 'carpark_category', 'carpark_shift_area', 'carpark_ntd', 'note', 'serial_no'] df.columns = COL_NAME

Drop Useless Columns

Some columns have high missing values (above 90% of data) or useless information, thus, drop them.

DROPED_COLUMNS = ['non_urban_use', 'non_urban_use_code', 'note', 'serial_no'] df = df.drop(DROPED_COLUMNS, axis=1)

Transform Data Types

By checking df.info(), we can transform data types into reasonable ones.

df['land_shift_area'] = df['land_shift_area'].astype(float)

Deal with Missing Values

For each column, there are different ways to deal with missing values. Here let’s look at the simplest example, fill missing values of total levels with 0 because there is no building for the transaction of land or parking lot. Other columns might reference additional information such as related columns to infer the missing values. The detail could be found in my Gitlab.

df['total_levels'] = df['total_levels'].fillna(0) # land and car park transaction have no shifting levels

Generate Additional Features

Column transactin_number “土地1建物0車位0” includes information telling us how many lands, buildings, and car parks are there in a transaction. By using the regular expression, we can extract key information out as a new feature.

df['number_of_land'] = df['transaction_number'].apply(lambda x: int(re.findall('土地\d+', x)[0][2:])) df['number_of_building'] = df['transaction_number'].apply(lambda x: int(re.findall('建物\d+', x)[0][2:])) df['number_of_carpark'] = df['transaction_number'].apply(lambda x: int(re.findall('車位\d+', x)[0][2:]))

Another column transaction_date helps us extract information of the transaction year. After adding 1,911 to the year of the civil calendar, we generalize our dataset in a more standard way.

df[‘transaction_year’] = df[‘transaction_date’].apply(lambda x: str(1911 + int(x[:-4]))) # year should be categorical value

Do Translation

To make this dataset more widely used, we translate the text content into English. Since there is a limit of API calls with translation package, we do not translate text row by row but translate the unique word in each column first and then map them back to the original dataset.

# 0. fields to translate

COL_TO_TRANSLATE = ['transaction_type', 'urban_land_use', 'main_use', 'main_building_material', 'carpark_category'] # 1. find unique words and do translation

dic_translation = {}

from translate import Translator

translator= Translator(from_lang="zh-TW", to_lang="english") for col in COL_TO_TRANSLATE:

for word in pd.unique(df[col]).tolist():

dic_translation[word] = translator.translate(word) # 2. conduct replacement

for col in COL_TO_TRANSLATE:

df[col] = df[col].map(dic_translation)

Next, I will show how I uploaded the preprocessed data to Kaggle and document more information on it. Happy Journey on Data Science! : )

For all detailed content in this section, please check Gitlab: https://gitlab.com/chrissmart/taiwan-housing-price-prediction/-/blob/master/src/data_collection.ipynb