Predicting Used Car Prices with Machine Learning
By: Soner Yıldırım
Tags: Machine Learning, Data Science, Data Analysis, Data Visualization, Linear Regression

3. Exploratory Data Analysis

Price

It’s always good to get some insight about the target variable. The target or dependent variable is price in our case.

print(df.price.mean())

print(df.price.median())

83153.7379289853

64250.0

Mean is much higher than median which indicates there are outliers or extreme values. Let’s also check maximum and minimum values:

print(df.price.max())

print(df.price.min())

111111111.0

24.0

This values are obviously wrong. There is no wolkswagen polo for over 100 million unless it is gold coated. Similarly, the value of 24 Turkish Liras is not possible. After sorting values in price column by using sort_values() function, I detected a few more outliers and dropped them using pandas drop() function by passing indexes of the values to be dropped. Let’s check new mean and median values:

print(df.price.mean())

print(df.price.median())

print(df.price.median()) 66694.66636931311

64275.0

25000.0

Mean is still higher than median but the difference is not extreme. I also checked mode which is the value that occurs most often. Mean being higher than median indicates that the data is right or positive skewed which means we have more of lower prices and some outliers with higher values. Measures of central tendency being sorted as mean > median > mode is an indication of positive (right) skewness. We can double check with distribution plot:

x = df.price

plt.figure(figsize=(10,6))

sns.distplot(x).set_title('Frequency Distribution Plot of Prices')

It can be seen from the graph that the data is right skewed and the peak around 25000 shows us the mode. Another way of checking the distribution and outliers is boxplot:

plt.figure(figsize=(8,5))

sns.boxplot(y='price', data=df, width=0.5)

The bottom and top of the blue box represent first quartile (25%) and third quartile (75%), respectively. First quartile means 25% of data points are below this point. The line in the middle is the median (50%). The outliers are shown with dots above the maximum line.

Date

I don’t think date by itself has an effect on the price but waiting period of the ad on website is a factor to be considered. Longer waiting time might motivate owner to reduce the price. If an ad stays on the website for a long time, it might be because the price is not set properly. So I will add a column indicating the number of days ad has been on the website. Data was scraped on 18.01.2020.

df['ad_duration'] = pd.to_datetime('2020-01-18') - df['ad_date']

Ad_duration must be a numerical data so ‘days’ next to numbers need to be removed. I used pandas replace() function to remove ‘days’.

Let’s check the distribution of ad duration data:

print(df.ad_duration.mean())

print(df.ad_duration.median())

12.641540291406482

10.0

Mean is higher than the median and there are many outliers. Data is right skewed. To get a better understanding, I also plotted data points less than 50:

Location

There are 81 different cities but 62% of all ads are listed in top 10 cities with Istanbul having 23% of all ads.

a = df.city.value_counts()[:10]

df_location = pd.DataFrame({"city": a , "share": a/6726}) df_location.share.sum()

0.6216176033303599

Top 10 cities in terms of number of ads

Color

It seems like the optimal choice of color is white for wolkswagen polo. More than half of the cars are white followed by red and black. Top 3 colors cover 72% of all cars.

Beyaz:White, Kırmızı:Red, Siyah:Black

Year

The age of the car definitely effects the prices. However, instead of the model year of the car, it makes more sense to use is as age. So I substituted ‘year’ column from current year.

df['age'] = 2020 - df['year']

According to the distribution, most of the cars are less than 10 years old. There is a huge drop at 10 followed by an increasing trend.

Km

Km value shows how much the car has ben driven so it is definitely an important factor determining the price. Km data has approximately a normal distribution.

print(df.km.mean())

print(df.km.median())

141011.5676479334

137000.0

Ad title

Ad title is kind of a caption of the ad. Sellers try to attract possible buyers with a limited number of characters. Once an ad is clicked on, another page with pictures and more detailed information opens up. However, the first step is to get people to click on your ad so ad title plays a critical role in selling process.

Let’s check what people usually write in the title. I used wordcloud for this task.

#import dependencies

from wordcloud import WordCloud, STOPWORDS

The only required parameter for WordCloud is a text. You can check the docstring by typing “?WordCloud” for other optional parameters. We cannot input a list to wordcloud so I created a text by concatenating all the titles in ad_title column:

text_list = list(df.ad_title)

text = '-'.join(text_list)

Then used this text to generate a wordcloud:

#generate wordcloud

wordcloud = WordCloud(background_color='white').generate(text) #plot wordcloud

plt.figure(figsize=(10,6))

plt.imshow(wordcloud, interpolation='bilinear')

plt.axis("off")

plt.show()

wordcloud without stopwords

The idea of a wordcloud is pretty simple. The more frequent words are shown bigger. It is an informative and easy-to-understand tool for text analysis. However, the wordcloud above does not tell us much because the words “vw”, “volkswagen” and “polo” are not what we are looking for. They show the brand we are analyzing. In this case, we should use stopwords parameter of wordcloud to list the words that need to be excluded.

stopwords = ['VW', 'VOLKSWAGEN', 'POLO', 'MODEL', 'KM']

wordcloud = WordCloud(stopwords=stopwords).generate(text) plt.figure(figsize=(10,6))

plt.imshow(wordcloud, interpolation='bilinear')

plt.axis("off")

plt.show()

wordcloud with stopwords

I did not use background_color parameter this time just to show the difference. The words are in Turkish so I will give a brief explanation:

“Hatasız” : Without any problem/issue “Sahibinden”: From the owner (this is important because people tend to buy from the owner rather than a dealer). “Otomatik”: Automatic transmission “Boyasız”: No paint (no part of the car painted due to a crack, scratch or a repair)

The other words are mainly about being clean, not having any previous repairment.

Model

Model column includes three different kinds of information: engine size, fuel type and variant. After checking the values, I found out that only engine size information is complete for all cells. Fuel type and variant are missing for most of the cells so I created a separate column for engine size.

The first three characters after spaces represent engine size. I first removed spaces and extracted the first three characters from model column:

#remove spaces

df.model = df.model.replace(' ','',regex=True) engine = [x[:3] for x in df.model]

df['engine'] = engine

Let’s check how price changes with different engine sizes:

df.engine.value_counts()

1.4 3172

1.6 1916

1.2 1205

1.0 409

1.9 20

1.3 4

Name: engine, dtype: int64 df[['engine','price']].groupby(['engine']).mean().sort_values(by='price', ascending=False)

It seems like average price decreases with increasing engine size. 1.3 can be ignored since there are only 4 cars with 1.3 engine. There is a big gap with 1.0 and the other engine sizes because 1.0 is a newer model. As you can see on the chart below, cars with 1.0 engine size have both lowest age and km on average which shows that they are newer models.