#!/usr/bin/env python
# coding: utf-8

# In[1]:


import pandas as pd
import numpy as np
import sys
sys.path.append('../')


# # Wine Quality

# ## Introduction

# The dataset is taken from [Here](https://archive.ics.uci.edu/ml/datasets/Wine+Quality). Two datasets are included in this project, the red and white vinho verde wine samples, from the north of Portugal. The goal is to model wine quality based on physicochemical tests (see Cortez et al., 2009, [Web Link](http://www3.dsi.uminho.pt/pcortez/wine/)).

# ## Attribute Information

# - **fixed acidity**
# - **volatile acidity**
# - **citric acid**
# - **residual sugar**
# - **chlorides**
# - **free sulfur dioxide**
# - **total sulfur dioxide**
# - **density**
# - **pH**
# - **sulphates**
# - **alcohol**
# - **quality** (score between 0 and 10)

# ## Load the Data

# In[2]:


red_data = pd.read_csv('datasets/winequality-red.csv', sep=';')
white_data = pd.read_csv('datasets/winequality-white.csv', sep=';')


# In[3]:


red_data.head()


# In[4]:


white_data.head()


# ### Binding

# In[5]:


red_data['variant'] = 'red'
white_data['variant'] = 'white'
data = pd.concat([red_data, white_data], axis=0)
data = data.reset_index().drop('index', axis=1)


# In[6]:


data.sample(5)


# I bind these 2 datasets because they have the **same structure**. I'm curious if the variant contributes to the quality of the wine.

# ## Exploratory Data Analysis

# In[7]:


from fd_lib.overview import Overview
import matplotlib.pyplot as plt
import seaborn as sns
sns.set()


# ### General Information

# In[8]:


data.describe()


# In[9]:


data.info()


# In[10]:


data.isnull().sum().sum()


# Observation shows that:
# - There are some features that are very high and very far from the mean.
# - All columns are numeric features, except variant.
# - There is no value missing value.

# ### Correlation

# In[11]:


corr = data.corr()
mask = np.triu(np.ones_like(corr, dtype=bool))
plt.figure(figsize=(11,9))
sns.heatmap(corr, cmap='Greens', mask=mask, annot=True);


# **Alcohol** appears to be a feature that is highly correlated with wine quality.

# ### Outliers and Distribution of _Quality_

# In[12]:


fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(15,5))

sns.boxplot(ax=ax1, x='quality', data=data)
ax1.set_title('Outliers', size=20, pad=15)
ax1.set_xlabel('')

sns.distplot(ax=ax2, a=data['quality'], hist=False)
ax2.set_title('Distribution', size=20, pad=15)
ax2.set_xlabel('')

plt.show()


# In[13]:


cleaned_data = data[(data['quality'] >= 4) & (data['quality'] <= 7)]


# In[14]:


cleaned_data.sample(5)


# There are many more normal wines sample than excellent or poor ones. Removing the minor class will make the model **performs** better. But removing these will make the model fail in predicting that minority because it will **always predict normal quality**. The quality is normally distributed.

# ### Insight

# In[15]:


low_quality_wine = Overview(data, data['quality'] == 3, only_outliers=True)


# In[16]:


first_index = low_quality_wine.get_index(0)
low_quality_wine.get_frame(first_index)


# In[17]:


high_quality_wine = Overview(data, data['quality'] == 9, only_outliers=False)


# In[18]:


first_index = high_quality_wine.get_index(0)
high_quality_wine.get_frame(first_index)


# ## Modeling

# In[19]:


# Model
from sklearn.linear_model import Ridge
from sklearn.linear_model import Lasso
from sklearn.svm import SVR
from sklearn.tree import DecisionTreeRegressor
from sklearn.ensemble import RandomForestRegressor
from xgboost import XGBRegressor
# Selection
from sklearn.model_selection import train_test_split
from sklearn.model_selection import KFold
from sklearn.model_selection import cross_val_score
import time as t
# Feature Engineering
from fd_lib.feature_engineering import BestPair
# Pipeline
from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import OneHotEncoder
# Scoring
from sklearn.metrics import mean_absolute_error as MAE


# The dataset has a lot of features and outliers but I have removed some of the extreme outliers. It's interesting to use a model that can handle multiple features and outliers. These are the models I am considering:
# - **Ridge**, multicollinearity is present in the dataset
# - **Lasso**, automatically does feature selection.
# - **SVR**, performs well in high-dimensional feature space and robust to outliers.
# - **DecisionTreeRegressor**, can handle non-linear data and robust to outliers.
# - **RandomForestRegressor**, ensemble version of Decision Tree.
# - **XGBRegressor**, sophisticated version of Random Forest.

# ### Train Test Split

# In[20]:


X = cleaned_data.drop('quality', axis=1)
y = cleaned_data['quality']

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)


# ### Features

# #### Correlation

# In[21]:


BestPair(X, y, how='all').mold()


# The calculated pairs **don't have a higher correlation** than the original, I wouldn't use them.

# #### Pipeline

# In[22]:


features = Pipeline([
    ('encode', ColumnTransformer([
        ('onehot', OneHotEncoder(), ['variant'])
    ], remainder='passthrough'))
])


# ### Model Selection

# In[23]:


estimators = {
    'Ridge': Ridge(),
    'Lasso': Lasso(),
    'SVR': SVR(),
    'DT': DecisionTreeRegressor(),
    'RF': RandomForestRegressor(),
    'XGB': XGBRegressor()
}


# In[24]:


cv = KFold(n_splits=10, shuffle=True, random_state=0)
Xt = features.fit_transform(X)
for name, estimator in estimators.items():
    start = t.time()
    scores = cross_val_score(estimator, Xt, y, cv=cv, scoring='neg_mean_absolute_error', n_jobs=-1)
    finish = t.time()
    print('{}: {:.3f} in {:.3f}s'.format(name, scores.mean(), finish - start))


# **RandomForest** appears to be performing **the best**, but it takes relatively **longer time** to train. There will be a **trade off**. However, I consider to use DecisionTree in production as it is significantly smaller with only a little of performance trade-off.

# ### Fitting Model

# In[25]:


est = Pipeline([
    ('features', features),
    ('regressor', RandomForestRegressor())
])
est.fit(X_train, y_train);


# I did the hyperparameters tuning previously but it made the model's performance **worse**.

# ### Scoring

# In[26]:


pd.DataFrame({'MAE': [MAE(y_train, est.predict(X_train)),
                      MAE(y_test, est.predict(X_test))],
              'R2': [est.score(X_train, y_train),
                     est.score(X_test, y_test)]},
             index=['Train', 'Test'])


# In[27]:


pd.DataFrame({'Predict': est.predict(X_test).round(0).astype(int), 'True': y_test}).sample(5)


# The **Mean Absolute Error** is **below 0.5**, meaning that rounding will not affect the value. The predicted values are also very close to the true values, most of them even the same.

# ## Feature Importances

# In[28]:


columns = np.delete(cleaned_data.columns, [11])
scores = np.append(
    est.named_steps['regressor'].feature_importances_[2:],
    np.average(
        est.named_steps['regressor'].feature_importances_[:2],
        weights=[len(red_data), len(white_data)]
    )
)


# I did OneHotEncoding on the variant, it will produce 2 features because there are **2 classes** in the variant. I take the **weighted mean** of these values.

# In[29]:


sorted_feature_importances = sorted(zip(columns, scores), key=lambda item: -item[1])

columns = []
scores = []
for item in sorted_feature_importances:
    columns.append(item[0])
    scores.append(item[1])


# In[30]:


fig, ax = plt.subplots(figsize=(12,7))

sns.barplot(ax=ax, x=columns, y=scores)
ax.set_title('Feature Importances', size=20, pad=15)
ax.set_xlabel('Feature')
ax.set_ylabel('Score')

for index, score in enumerate(scores):
    ax.text(index, score, round(score, 3), ha='center')

plt.xticks(rotation=45)
plt.plot();


# The chart shows that **alcohol** is the feature that most determines **wine quality** followed by **volatile acidity**. **Volatile acidity** *(VA)* is a measure of the wine's gaseous acids that contributes to the smell and taste of vinegar in wine.
# 
# [Read More](https://extension.psu.edu/volatile-acidity-in-wine)

# ## Conclusion

# - **Alcohol** and **volatile acidity** are the most important factors determining **wine quality** while **variant** doesn't.
# - I chose **RandomForestRegressor** for this mini-project for the following reasons besides the drawback is it requires quite a **long time** to train.
#  - **Robust to outliers**, meaning that it is not sensitive to outliers.
#  - **Does a good job**, meaning that the predicted value is very close to the true value.
this,is,the,notebook,
# Laporan Proyek Machine Learning - Frederic Davidsen

## Domain Proyek

Pada project ini, saya menggunakan Wine Quality Dataset dari https://archive.ics.uci.edu/ml/datasets/Wine+Quality untuk melakukan prediksi kualitas anggur berdasarkan parameter fisikokimia yang terukur. Kualitas anggur sangat penting bagi produsen dan konsumen, karena hal ini dapat mempengaruhi harga dan persepsi di pasar. Dengan menggunakan machine learning, saya akan membuat model untuk memprediksi kualitas anggur melalui parameter (fitur) yang ada secara lebih objektif dan efisien dibandingkan dengan pengujian sensorial yang biasanya dilakukan.

## Business Understanding

### Problem Statements

- Pernyataan Masalah 1: Bagaimana cara memprediksi kualitas anggur berdasarkan fitur fisikokimia yang ada?
- Pernyataan Masalah 2: Fitur mana yang paling berpengaruh terhadap kualitas anggur?

### Goals

- Jawaban pernyataan masalah 1: Mengembangkan model prediktif yang dapat memberikan penilaian kualitas anggur dengan akurasi yang tinggi.
- Jawaban pernyataan masalah 2: Mengidentifikasi fitur paling penting yang mempengaruhi penilaian kualitas anggur.

#### Solution Statements

- Menggunakan algoritma machine learning berjenis regresi untuk membuat model prediktif.
- Melakukan improvement pada model baseline dengan hyperparameter tuning untuk meningkatkan akurasi.

## Data Understanding

Saya menggunakan Wine Quality Dataset yang tersedia di UCI Machine Learning Repository. Dataset ini berisi pengukuran berbagai bahan kimia dan penilaian kualitas yang dilakukan ahli.

### Variabel-variabel pada Wine Quality Dataset adalah sebagai berikut:
- fixed acidity: asam dalam anggur yang cenderung tidak menguap.
- volatile acidity: tingkat asam yang mudah menguap dan dapat menyebabkan rasa tajam.
- citric acid: terdapat dalam jumlah kecil pada anggur untuk menambah kesegaran.
- residual sugar: gula yang tersisa setelah fermentasi.
- chlorides: garam yang ada dalam anggur.
- free sulfur dioxide: SO2 dalam anggur berfungsi sebagai pengawet.
- total sulfur dioxide: total SO2 adalah jumlah total yang ada.
- density: kepadatan anggur.
- pH: tingkat keasaman/alkalinitas anggur.
- sulphates: menambah daya pertahanan anggur terhadap oksidasi.
- alcohol: persentase alkohol dalam anggur.
- quality: kelas kualitas anggur (target variabel).

## Data Preparation

Dalam data preparation, langkah-langkah yang dilakukan meliputi:

1. Categorical Encoding: Melakukan encoding pada fitur categorical agar dapat diproses oleh model.
2. Handling Missing Values: Memeriksa keberadaan data yang hilang dan menanganinya dengan metode yang sesuai.
3. Pemisahan Data: Dataset dibagi menjadi data latih dan uji dengan rasio 80:20 untuk memastikan model generalisasi yang baik.

## Modeling

Saya mencoba menggunakan berbagai jenis model regresi untuk memprediksi kualitas anggur:

1. Ridge: Terdapat multicollinearity pada dataset.
2. Lasso: Dapat melakukan "automatic feature selection".
3. SVR: Memiliki performa baik pada feature space yang berdimensi tinggi serta robust terhadap outliers.
4. DecisionTreeRegressor: Dapat menangani fitur non-linear dalam data and robust terhadap outliers.
5. RandomForestRegressor: Version ensemble dari Decision Tree.
6. XGBRegressor: Versi boosting yang layak dicoba.

### Improvement
- Saya mencoba melakukan hyperparameter tuning untuk mengoptimalkan jumlah pohon di Random Forest dan parameter learning rate di Gradient Boosting, namun hasil yang didapat kurang baik. Saya mendapati bahwa hyperparameter default memiliki performa yang lebih baik.

## Evaluation

Saya menggunakan metrik evaluasi Mean Absolute Error (MAE) dan R2 Score untuk mengukur kinerja model:

- Mean Absolute Error (MAE): Pengukuran rata-rata absolut error, metrik ini tidak memberikan bobot lebih apapun meskipun pada nilai error yang lebih besar.
- R2 Score: Menunjukkan seberapa baik model dilengkapi dengan data yang sesuai. R2 mendekati 1 menunjukkan model yang bagus.

Formula:
- MSE = (1/n) |actual - prediction|
- R² = 1 - (Σ(actual - prediction)² / Σ(actual - mean)²)

Hasil pada test set:
- Random Forest: MAE = 0.39, R² = 0.53
this,is,the,markdown,