Repository: ploomber/soorgeon (language: python)

Issue:
making the CI more reliable
We have a bunch of integration tests that fetch data and notebooks using Kaggle's API. However, they fail sometimes (probably because the Kaggle API isn't very reliable). See https://github.com/ploomber/soorgeon/issues/58

One solution is to use the Kaggle API once and then upload the files to an S3 bucket, then have the CI download them from S3 instead of Kaggle.


Discussion / hints:
@edublancas what about including them in one of the repos?
including the files? the problem are the data files, some of them are a few MBs in size, I think we should store them somewhere else
Ok I think store in S3 bucket would be good.  I am going to list my thoughts here:
1. create a S3 bucket for notebook and data
2. write a script to download from kaggle and then upload to s3
3. run manually once to store what we have now in index.yaml to s3
4. change the CI to retrieve data from S3 instead of downloading again
5. Add another CI triggered by change in index.yaml  or in regular CI just add one check if index.yaml has been changed to run the download and upload script again.

You guys think this is a viable approach? 
Yeah it is, I don't think you need 2, just do it manually since there are
6-7 notebooks. We have an automated script in the CI but it fails since the
kaggle API isn't stable.
I didn't get step #5 thought, elaborate?

You'd probably also need to make this bucket public and we shouldn't have
anything in there besides those notebooks.

I'm also thinking of just opening a git repo just for those and consuming it, that way we don't need to pay for S3 get requests every time (try this approach first).

On Wed, Jun 8, 2022 at 4:54 PM Xilin ***@***.***> wrote:

> Ok I think store in S3 bucket would be good. I am going to list my
> thoughts here:
>
>    1. create a S3 bucket for notebook and data
>    2. write a script to download from kaggle and then upload to s3
>    3. run manually once to store what we have now in index.yaml to s3
>    4. change the CI to retrieve data from S3 instead of downloading again
>    5. Add another CI triggered by change in index.yaml or in regular CI
>    just add one check if index.yaml has been changed to run the download and
>    upload script again.
>
> You guys think this is a viable approach?
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/ploomber/soorgeon/issues/59#issuecomment-1150404905>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ACYPJOMSOJOEA6XIM54YZWDVOEB6TANCNFSM5X6U55AQ>
> .
> You are receiving this because you commented.Message ID:
> ***@***.***>
>

For number 5
If someone contributed by adding a new url in [soorgeon](https://github.com/ploomber/soorgeon)/[_kaggle](https://github.com/ploomber/soorgeon/tree/main/_kaggle)/index.yaml, then we need to upload it to S3 right. So we can of course manually do it. But I am thinking maybe write a workflow triggered by changes in index.yaml. (which i dont know if it is possible to achieve) Then it runs the script to upload  newly downloaded notebook.


 
If we do this manually, i think all we need to change is the test right? Instead of downloading again from kaggle, we read directly from our storage repo. And if in the future someone added new urls, we also manually put it into our storage repo?

This works? 
Yes, that's correct!
I think let's skip 5 for now, and if we see the number of notebooks scales
to double/triple then we can do that.

On Wed, Jun 8, 2022 at 5:18 PM Xilin ***@***.***> wrote:

> If we do this manually, i think we can all we need to change is the test
> right? Instead of downloading again from kaggle, we read directly from our
> storage repo. And if in the future someone added new urls, we also manually
> put it into our storage repo?
>
> This works?
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/ploomber/soorgeon/issues/59#issuecomment-1150424594>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ACYPJOJNQZ7ZF7PNLBDJW6DVOEEZ3ANCNFSM5X6U55AQ>
> .
> You are receiving this because you commented.Message ID:
> ***@***.***>
>

Let's try this line of thoughts:
[Try retrying](https://github.com/litl/backoff) the steps up to 3 times, if that keeps on breaking,
Try via git - sample data
Then if that fails, full data on S3
👍

Fix patch:
diff --git a/.github/workflows/ci.yaml b/.github/workflows/ci.yaml
index 526432b..f7fece7 100644
--- a/.github/workflows/ci.yaml
+++ b/.github/workflows/ci.yaml
@@ -19,7 +19,6 @@ jobs:
           python-version: ${{ matrix.python-version }}
       - name: Lint with flake8
         run: |
-          python -m pip install "pip<22"
           pip install flake8
           flake8
       - name: Install dependencies
@@ -49,7 +48,6 @@ jobs:
           python-version: ${{ matrix.python-version }}
       - name: Lint with flake8
         run: |
-          python -m pip install "pip<22"
           pip install flake8
           flake8
       - name: Install dependencies
@@ -60,8 +58,6 @@ jobs:
           pip install -r _kaggle/requirements.lock.txt
       - name: Integration tests
         env:
-          KAGGLE_KEY: ${{ secrets.KAGGLE_KEY }}
-          KAGGLE_USERNAME: ${{ secrets.KAGGLE_USERNAME }}
           PLOOMBER_STATS_ENABLED: false
         run: |
           pytest "tests/test_sample_notebooks.py::test_notebooks[${{ matrix.notebook }}]"
diff --git a/.gitignore b/.gitignore
index 9dd5087..e85a8fb 100644
--- a/.gitignore
+++ b/.gitignore
@@ -47,6 +47,7 @@ share/python-wheels/
 .installed.cfg
 *.egg
 MANIFEST
+.DS_Store
 
 # PyInstaller
 #  Usually these files are written by a python script from a template
diff --git a/_kaggle/customer-segmentation-clustering/nb.py b/_kaggle/customer-segmentation-clustering/nb.py
deleted file mode 100644
index a22f50c..0000000
--- a/_kaggle/customer-segmentation-clustering/nb.py
+++ /dev/null
@@ -1,584 +0,0 @@
-# ---
-# jupyter:
-#   jupytext:
-#     text_representation:
-#       extension: .py
-#       format_name: percent
-#       format_version: '1.3'
-#       jupytext_version: 1.13.1
-#   kernelspec:
-#     display_name: Python 3
-#     language: python
-#     name: python3
-# ---
-
-# %% _kg_hide-input=true _kg_hide-output=true papermill={"duration": 0.059258, "end_time": "2021-10-08T04:14:39.074653", "exception": false, "start_time": "2021-10-08T04:14:39.015395", "status": "completed"} tags=[]
-# This Python 3 environment comes with many helpful analytics libraries installed
-# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python
-# For example, here's several helpful packages to load
-
-import numpy as np  # linear algebra
-import pandas as pd  # data processing, CSV file I/O (e.g. pd.read_csv)
-
-# Input data files are available in the read-only "../input/" directory
-# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory
-
-import os
-for dirname, _, filenames in os.walk('.'):
-    for filename in filenames:
-        print(os.path.join(dirname, filename))
-
-# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using "Save & Run All"
-# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session
-
-# %% [markdown] papermill={"duration": 0.027866, "end_time": "2021-10-08T04:14:39.135536", "exception": false, "start_time": "2021-10-08T04:14:39.107670", "status": "completed"} tags=[]
-# ## Customer Segmentation
-#
-# <img src="https://github.com/KarnikaKapoor/Files/blob/main/Colorful%20Handwritten%20About%20Me%20Blank%20Education%20Presentation.gif?raw=true">
-#
-# In this project, I will be performing an unsupervised clustering of data on the customer's records from a groceries firm's database. Customer segmentation is the practice of separating customers into groups that reflect similarities among customers in each cluster. I will divide customers into segments to optimize the significance of each customer to the business. To modify products according to distinct needs and behaviours of the customers. It also helps the business to cater to the concerns of different types of customers.
-#
-#
-#    <a id='top'></a>
-# <div class="list-group" id="list-tab" role="tablist">
-## TABLE OF CONTENTS
-#
-# * [1. IMPORTING LIBRARIES](#1)
-#
-# * [2. LOADING DATA](#2)
-#
-# * [3. DATA CLEANING](#3)
-#
-# * [4. DATA PREPROCESSING](#4)
-#
-# * [5. DIMENSIONALITY REDUCTION](#5)
-#
-# * [6. CLUSTERING](#6)
-#
-# * [7. EVALUATING MODELS](#7)
-#
-# * [8. PROFILING](#8)
-#
-# * [9. CONCLUSION](#9)
-#
-# * [10. END](#10)
-#
-
-# %% [markdown] papermill={"duration": 0.028812, "end_time": "2021-10-08T04:14:39.192253", "exception": false, "start_time": "2021-10-08T04:14:39.163441", "status": "completed"} tags=[]
-# <a id="1"></a>
-# ## IMPORTING LIBRARIES
-
-# %% papermill={"duration": 1.53841, "end_time": "2021-10-08T04:14:40.759736", "exception": false, "start_time": "2021-10-08T04:14:39.221326", "status": "completed"} tags=[]
-#Importing the Libraries
-import numpy as np
-import pandas as pd
-import datetime
-import matplotlib
-import matplotlib.pyplot as plt
-from matplotlib import colors
-import seaborn as sns
-from sklearn.preprocessing import LabelEncoder
-from sklearn.preprocessing import StandardScaler
-from sklearn.decomposition import PCA
-from yellowbrick.cluster import KElbowVisualizer
-from sklearn.cluster import KMeans
-import matplotlib.pyplot as plt, numpy as np
-from mpl_toolkits.mplot3d import Axes3D
-from sklearn.cluster import AgglomerativeClustering
-from matplotlib.colors import ListedColormap
-from sklearn import metrics
-import warnings
-import sys
-if not sys.warnoptions:
-    warnings.simplefilter("ignore")
-np.random.seed(42)
-
-# %% [markdown] papermill={"duration": 0.029156, "end_time": "2021-10-08T04:14:40.817772", "exception": false, "start_time": "2021-10-08T04:14:40.788616", "status": "completed"} tags=[]
-# <a id="2"></a>
-# ## LOADING DATA
-
-# %% papermill={"duration": 0.087752, "end_time": "2021-10-08T04:14:40.933340", "exception": false, "start_time": "2021-10-08T04:14:40.845588", "status": "completed"} tags=[]
-#Loading the dataset
-data = pd.read_csv("input/marketing_campaign.csv", sep="\t")
-print("Number of datapoints:", len(data))
-data.head()
-
-# %% [markdown] papermill={"duration": 0.028, "end_time": "2021-10-08T04:14:40.989700", "exception": false, "start_time": "2021-10-08T04:14:40.961700", "status": "completed"} tags=[]
-# <img src="https://github.com/KarnikaKapoor/Files/blob/main/Colorful%20Handwritten%20About%20Me%20Blank%20Education%20Presentation.png?raw=true">
-#
-# For more information on the attributes visit [here](https://www.kaggle.com/imakash3011/customer-personality-analysis).
-#
-# <a id="3"></a>
-# ## DATA CLEANING
-#
-#
-# **In this section**
-# * Data Cleaning
-# * Feature Engineering
-#
-# In order to, get a full grasp of what steps should I be taking to clean the dataset.
-# Let us have a look at the information in data.
-#
-
-# %% papermill={"duration": 0.058629, "end_time": "2021-10-08T04:14:41.076744", "exception": false, "start_time": "2021-10-08T04:14:41.018115", "status": "completed"} tags=[]
-#Information on features
-data.info()
-
-# %% [markdown] papermill={"duration": 0.028603, "end_time": "2021-10-08T04:14:41.136192", "exception": false, "start_time": "2021-10-08T04:14:41.107589", "status": "completed"} tags=[]
-# **From the above output, we can conclude and note that:**
-#
-# * There are missing values in income
-# * Dt_Customer that indicates the date a customer joined the database is not parsed as DateTime
-# * There are some categorical features in our data frame; as there are some features in dtype: object). So we will need to encode them into numeric forms later.
-#
-# First of all, for the missing values, I am simply going to drop the rows that have missing income values.
-
-# %% papermill={"duration": 0.042312, "end_time": "2021-10-08T04:14:41.207425", "exception": false, "start_time": "2021-10-08T04:14:41.165113", "status": "completed"} tags=[]
-#To remove the NA values
-data = data.dropna()
-print(
-    "The total number of data-points after removing the rows with missing values are:",
-    len(data))
-
-# %% [markdown] papermill={"duration": 0.028861, "end_time": "2021-10-08T04:14:41.266284", "exception": false, "start_time": "2021-10-08T04:14:41.237423", "status": "completed"} tags=[]
-# In the next step, I am going to create a feature out of **"Dt_Customer"** that indicates the number of days a customer is registered in the firm's database. However, in order to keep it simple, I am taking this value relative to the most recent customer in the record.
-#
-# Thus to get the values I must check the newest and oldest recorded dates.
-
-# %% papermill={"duration": 0.046416, "end_time": "2021-10-08T04:14:41.341737", "exception": false, "start_time": "2021-10-08T04:14:41.295321", "status": "completed"} tags=[]
-data["Dt_Customer"] = pd.to_datetime(data["Dt_Customer"])
-dates = []
-for i in data["Dt_Customer"]:
-    i = i.date()
-    dates.append(i)
-#Dates of the newest and oldest recorded customer
-print("The newest customer's enrolment date in therecords:", max(dates))
-print("The oldest customer's enrolment date in the records:", min(dates))
-
-# %% [markdown] papermill={"duration": 0.029401, "end_time": "2021-10-08T04:14:41.400477", "exception": false, "start_time": "2021-10-08T04:14:41.371076", "status": "completed"} tags=[]
-# Creating a feature **("Customer_For")** of the number of days the customers started to shop in the store relative to the last recorded date
-
-# %% papermill={"duration": 0.051941, "end_time": "2021-10-08T04:14:41.482563", "exception": false, "start_time": "2021-10-08T04:14:41.430622", "status": "completed"} tags=[]
-#Created a feature "Customer_For"
-days = []
-d1 = max(dates)  #taking it to be the newest customer
-for i in dates:
-    delta = d1 - i
-    days.append(delta)
-data["Customer_For"] = days
-data["Customer_For"] = pd.to_numeric(data["Customer_For"], errors="coerce")
-
-# %% [markdown] papermill={"duration": 0.040084, "end_time": "2021-10-08T04:14:41.565744", "exception": false, "start_time": "2021-10-08T04:14:41.525660", "status": "completed"} tags=[]
-# Now we will be exploring the unique values in the categorical features to get a clear idea of the data.
-
-# %% papermill={"duration": 0.043279, "end_time": "2021-10-08T04:14:41.640142", "exception": false, "start_time": "2021-10-08T04:14:41.596863", "status": "completed"} tags=[]
-print("Total categories in the feature Marital_Status:\n",
-      data["Marital_Status"].value_counts(), "\n")
-print("Total categories in the feature Education:\n",
-      data["Education"].value_counts())
-
-# %% [markdown] papermill={"duration": 0.029496, "end_time": "2021-10-08T04:14:41.700019", "exception": false, "start_time": "2021-10-08T04:14:41.670523", "status": "completed"} tags=[]
-# **In the next bit, I will be performing the following steps to engineer some new features:**
-#
-# * Extract the **"Age"** of a customer by the **"Year_Birth"** indicating the birth year of the respective person.
-# * Create another feature **"Spent"** indicating the total amount spent by the customer in various categories over the span of two years.
-# * Create another feature **"Living_With"** out of **"Marital_Status"** to extract the living situation of couples.
-# * Create a feature **"Children"** to indicate total children in a household that is, kids and teenagers.
-# * To get further clarity of household, Creating feature indicating **"Family_Size"**
-# * Create a feature **"Is_Parent"** to indicate parenthood status
-# * Lastly, I will create three categories in the **"Education"** by simplifying its value counts.
-# * Dropping some of the redundant features
-
-# %% papermill={"duration": 0.055238, "end_time": "2021-10-08T04:14:41.784923", "exception": false, "start_time": "2021-10-08T04:14:41.729685", "status": "completed"} tags=[]
-#Feature Engineering
-#Age of customer today
-data["Age"] = 2021 - data["Year_Birth"]
-
-#Total spendings on various items
-data["Spent"] = data["MntWines"] + data["MntFruits"] + data[
-    "MntMeatProducts"] + data["MntFishProducts"] + data[
-        "MntSweetProducts"] + data["MntGoldProds"]
-
-#Deriving living situation by marital status"Alone"
-data["Living_With"] = data["Marital_Status"].replace({
-    "Married": "Partner",
-    "Together": "Partner",
-    "Absurd": "Alone",
-    "Widow": "Alone",
-    "YOLO": "Alone",
-    "Divorced": "Alone",
-    "Single": "Alone",
-})
-
-#Feature indicating total children living in the household
-data["Children"] = data["Kidhome"] + data["Teenhome"]
-
-#Feature for total members in the householde
-data["Family_Size"] = data["Living_With"].replace({
-    "Alone": 1,
-    "Partner": 2
-}) + data["Children"]
-
-#Feature pertaining parenthood
-data["Is_Parent"] = np.where(data.Children > 0, 1, 0)
-
-#Segmenting education levels in three groups
-data["Education"] = data["Education"].replace({
-    "Basic": "Undergraduate",
-    "2n Cycle": "Undergraduate",
-    "Graduation": "Graduate",
-    "Master": "Postgraduate",
-    "PhD": "Postgraduate"
-})
-
-#For clarity
-data = data.rename(
-    columns={
-        "MntWines": "Wines",
-        "MntFruits": "Fruits",
-        "MntMeatProducts": "Meat",
-        "MntFishProducts": "Fish",
-        "MntSweetProducts": "Sweets",
-        "MntGoldProds": "Gold"
-    })
-
-#Dropping some of the redundant features
-to_drop = [
-    "Marital_Status", "Dt_Customer", "Z_CostContact", "Z_Revenue",
-    "Year_Birth", "ID"
-]
-data = data.drop(to_drop, axis=1)
-
-# %% [markdown] papermill={"duration": 0.029572, "end_time": "2021-10-08T04:14:41.843846", "exception": false, "start_time": "2021-10-08T04:14:41.814274", "status": "completed"} tags=[]
-# Now that we have some new features let's have a look at the data's stats.
-
-# %% papermill={"duration": 0.109706, "end_time": "2021-10-08T04:14:41.983356", "exception": false, "start_time": "2021-10-08T04:14:41.873650", "status": "completed"} tags=[]
-data.describe()
-
-# %% [markdown] papermill={"duration": 0.030087, "end_time": "2021-10-08T04:14:42.043835", "exception": false, "start_time": "2021-10-08T04:14:42.013748", "status": "completed"} tags=[]
-# The above stats show some discrepancies in mean Income and Age and max Income and age.
-#
-# Do note that  max-age is 128 years, As I calculated the age that would be today (i.e. 2021) and the data is old.
-#
-# I must take a look at the broader view of the data.
-# I will plot some of the selected features.
-
-# %% papermill={"duration": 8.979335, "end_time": "2021-10-08T04:14:51.053279", "exception": false, "start_time": "2021-10-08T04:14:42.073944", "status": "completed"} tags=[]
-#To plot some selected features
-#Setting up colors prefrences
-sns.set(rc={"axes.facecolor": "#FFF9ED", "figure.facecolor": "#FFF9ED"})
-pallet = ["#682F2F", "#9E726F", "#D6B2B1", "#B9C0C9", "#9F8A78", "#F3AB60"]
-cmap = colors.ListedColormap(
-    ["#682F2F", "#9E726F", "#D6B2B1", "#B9C0C9", "#9F8A78", "#F3AB60"])
-#Plotting following features
-To_Plot = ["Income", "Recency", "Customer_For", "Age", "Spent", "Is_Parent"]
-print("Reletive Plot Of Some Selected Features: A Data Subset")
-plt.figure()
-sns.pairplot(data[To_Plot], hue="Is_Parent", palette=(["#682F2F", "#F3AB60"]))
-#Taking hue
-plt.show()
-
-# %% [markdown] papermill={"duration": 0.046896, "end_time": "2021-10-08T04:14:51.147774", "exception": false, "start_time": "2021-10-08T04:14:51.100878", "status": "completed"} tags=[]
-# Clearly, there are a few outliers in the Income and Age features.
-# I will be deleting the outliers in the data.
-
-# %% papermill={"duration": 0.05833, "end_time": "2021-10-08T04:14:51.252647", "exception": false, "start_time": "2021-10-08T04:14:51.194317", "status": "completed"} tags=[]
-#Dropping the outliers by setting a cap on Age and income.
-data = data[(data["Age"] < 90)]
-data = data[(data["Income"] < 600000)]
-print("The total number of data-points after removing the outliers are:",
-      len(data))
-
-# %% [markdown] papermill={"duration": 0.046761, "end_time": "2021-10-08T04:14:51.348041", "exception": false, "start_time": "2021-10-08T04:14:51.301280", "status": "completed"} tags=[]
-# Next, let us look at the correlation amongst the features.
-# (Excluding the categorical attributes at this point)
-
-# %% papermill={"duration": 4.47676, "end_time": "2021-10-08T04:14:55.873081", "exception": false, "start_time": "2021-10-08T04:14:51.396321", "status": "completed"} tags=[]
-#correlation matrix
-corrmat = data.corr()
-plt.figure(figsize=(20, 20))
-sns.heatmap(corrmat, annot=True, cmap=cmap, center=0)
-
-# %% [markdown] papermill={"duration": 0.063054, "end_time": "2021-10-08T04:14:56.005374", "exception": false, "start_time": "2021-10-08T04:14:55.942320", "status": "completed"} tags=[]
-# The data is quite clean and the new features have been included. I will proceed to the next step. That is, preprocessing the data.
-#
-# <a id="4"></a>
-# ## DATA PREPROCESSING
-#
-# In this section, I will be preprocessing the data to perform clustering operations.
-#
-# **The following steps are applied to preprocess the data:**
-#
-# * Label encoding the categorical features
-# * Scaling the features using the standard scaler
-# * Creating a subset dataframe for dimensionality reduction
-
-# %% papermill={"duration": 0.085186, "end_time": "2021-10-08T04:14:56.148485", "exception": false, "start_time": "2021-10-08T04:14:56.063299", "status": "completed"} tags=[]
-#Get list of categorical variables
-s = (data.dtypes == 'object')
-object_cols = list(s[s].index)
-
-print("Categorical variables in the dataset:", object_cols)
-
-# %% papermill={"duration": 0.078164, "end_time": "2021-10-08T04:14:56.296208", "exception": false, "start_time": "2021-10-08T04:14:56.218044", "status": "completed"} tags=[]
-#Label Encoding the object dtypes.
-LE = LabelEncoder()
-for i in object_cols:
-    data[i] = data[[i]].apply(LE.fit_transform)
-
-print("All features are now numerical")
-
-# %% papermill={"duration": 0.075769, "end_time": "2021-10-08T04:14:56.430085", "exception": false, "start_time": "2021-10-08T04:14:56.354316", "status": "completed"} tags=[]
-#Creating a copy of data
-ds = data.copy()
-# creating a subset of dataframe by dropping the features on deals accepted and promotions
-cols_del = [
-    'AcceptedCmp3', 'AcceptedCmp4', 'AcceptedCmp5', 'AcceptedCmp1',
-    'AcceptedCmp2', 'Complain', 'Response'
-]
-ds = ds.drop(cols_del, axis=1)
-#Scaling
-scaler = StandardScaler()
-scaler.fit(ds)
-scaled_ds = pd.DataFrame(scaler.transform(ds), columns=ds.columns)
-print("All features are now scaled")
-
-# %% papermill={"duration": 0.087305, "end_time": "2021-10-08T04:14:56.575069", "exception": false, "start_time": "2021-10-08T04:14:56.487764", "status": "completed"} tags=[]
-#Scaled data to be used for reducing the dimensionality
-print("Dataframe to be used for further modelling:")
-scaled_ds.head()
-
-# %% [markdown] papermill={"duration": 0.062052, "end_time": "2021-10-08T04:14:56.694594", "exception": false, "start_time": "2021-10-08T04:14:56.632542", "status": "completed"} tags=[]
-# <a id="5"></a>
-# ## DIMENSIONALITY REDUCTION
-# In this problem, there are many factors on the basis of which the final classification will be done. These factors are basically attributes or features. The higher the number of features, the harder it is to work with it. Many of these features are correlated, and hence redundant. This is why I will be performing dimensionality reduction on the selected features before putting them through a classifier.
-# *Dimensionality reduction is the process of reducing the number of random variables under consideration, by obtaining a set of principal variables.*
-#
-# **Principal component analysis (PCA)** is a technique for reducing the dimensionality of such datasets, increasing interpretability but at the same time minimizing information loss.
-#
-# **Steps in this section:**
-# * Dimensionality reduction with PCA
-# * Plotting the reduced dataframe
-#
-# **Dimensionality reduction with PCA**
-#
-# For this project, I will be reducing the dimensions to 3.
-
-# %% papermill={"duration": 0.118668, "end_time": "2021-10-08T04:14:56.870748", "exception": false, "start_time": "2021-10-08T04:14:56.752080", "status": "completed"} tags=[]
-#Initiating PCA to reduce dimentions aka features to 3
-pca = PCA(n_components=3)
-pca.fit(scaled_ds)
-PCA_ds = pd.DataFrame(pca.transform(scaled_ds),
-                      columns=(["col1", "col2", "col3"]))
-PCA_ds.describe().T
-
-# %% papermill={"duration": 0.367462, "end_time": "2021-10-08T04:14:57.337354", "exception": false, "start_time": "2021-10-08T04:14:56.969892", "status": "completed"} tags=[]
-#A 3D Projection Of Data In The Reduced Dimension
-x = PCA_ds["col1"]
-y = PCA_ds["col2"]
-z = PCA_ds["col3"]
-#To plot
-fig = plt.figure(figsize=(10, 8))
-ax = fig.add_subplot(111, projection="3d")
-ax.scatter(x, y, z, c="maroon", marker="o")
-ax.set_title("A 3D Projection Of Data In The Reduced Dimension")
-plt.show()
-
-# %% [markdown] papermill={"duration": 0.061136, "end_time": "2021-10-08T04:14:57.460284", "exception": false, "start_time": "2021-10-08T04:14:57.399148", "status": "completed"} tags=[]
-# <a id="6"></a>
-# ## CLUSTERING
-#
-# Now that I have reduced the attributes to three dimensions, I will be performing clustering via Agglomerative clustering. Agglomerative clustering is a hierarchical clustering method.  It involves merging examples until the desired number of clusters is achieved.
-#
-# **Steps involved in the Clustering**
-# * Elbow Method to determine the number of clusters to be formed
-# * Clustering via Agglomerative Clustering
-# * Examining the clusters formed via scatter plot
-
-# %% papermill={"duration": 1.74229, "end_time": "2021-10-08T04:14:59.264051", "exception": false, "start_time": "2021-10-08T04:14:57.521761", "status": "completed"} tags=[]
-# Quick examination of elbow method to find numbers of clusters to make.
-print('Elbow Method to determine the number of clusters to be formed:')
-Elbow_M = KElbowVisualizer(KMeans(), k=10)
-Elbow_M.fit(PCA_ds)
-Elbow_M.show()
-
-# %% [markdown] papermill={"duration": 0.063192, "end_time": "2021-10-08T04:14:59.390530", "exception": false, "start_time": "2021-10-08T04:14:59.327338", "status": "completed"} tags=[]
-# The above cell indicates that four will be an optimal number of clusters for this data.
-# Next, we will be fitting the Agglomerative Clustering Model to get the final clusters.
-
-# %% papermill={"duration": 0.213864, "end_time": "2021-10-08T04:14:59.670550", "exception": false, "start_time": "2021-10-08T04:14:59.456686", "status": "completed"} tags=[]
-#Initiating the Agglomerative Clustering model
-AC = AgglomerativeClustering(n_clusters=4)
-# fit model and predict clusters
-yhat_AC = AC.fit_predict(PCA_ds)
-PCA_ds["Clusters"] = yhat_AC
-#Adding the Clusters feature to the orignal dataframe.
-data["Clusters"] = yhat_AC
-
-# %% [markdown] papermill={"duration": 0.063434, "end_time": "2021-10-08T04:14:59.798310", "exception": false, "start_time": "2021-10-08T04:14:59.734876", "status": "completed"} tags=[]
-# To examine the clusters formed let's have a look at the 3-D distribution of the clusters.
-
-# %% papermill={"duration": 0.381844, "end_time": "2021-10-08T04:15:00.244379", "exception": false, "start_time": "2021-10-08T04:14:59.862535", "status": "completed"} tags=[]
-#Plotting the clusters
-fig = plt.figure(figsize=(10, 8))
-ax = plt.subplot(111, projection='3d', label="bla")
-ax.scatter(x, y, z, s=40, c=PCA_ds["Clusters"], marker='o', cmap=cmap)
-ax.set_title("The Plot Of The Clusters")
-plt.show()
-
-# %% [markdown] papermill={"duration": 0.065606, "end_time": "2021-10-08T04:15:00.376433", "exception": false, "start_time": "2021-10-08T04:15:00.310827", "status": "completed"} tags=[]
-# <a id="7"></a>
-# ## EVALUATING MODELS
-#
-# Since this is an unsupervised clustering. We do not have a tagged feature to evaluate or score our model. The purpose of this section is to study the patterns in the clusters formed and determine the nature of the clusters' patterns.
-#
-# For that, we will be having a look at the data in light of clusters via exploratory data analysis and drawing conclusions.
-#
-# **Firstly, let us have a look at the group distribution of clustring**
-
-# %% papermill={"duration": 0.470936, "end_time": "2021-10-08T04:15:00.913213", "exception": false, "start_time": "2021-10-08T04:15:00.442277", "status": "completed"} tags=[]
-#Plotting countplot of clusters
-pal = ["#682F2F", "#B9C0C9", "#9F8A78", "#F3AB60"]
-pl = sns.countplot(x=data["Clusters"], palette=pal)
-pl.set_title("Distribution Of The Clusters")
-plt.show()
-
-# %% [markdown] papermill={"duration": 0.067211, "end_time": "2021-10-08T04:15:01.051651", "exception": false, "start_time": "2021-10-08T04:15:00.984440", "status": "completed"} tags=[]
-#
-#
-
-# %% [markdown] papermill={"duration": 0.066727, "end_time": "2021-10-08T04:15:01.186736", "exception": false, "start_time": "2021-10-08T04:15:01.120009", "status": "completed"} tags=[]
-# The clusters seem to be fairly distributed.
-
-# %% papermill={"duration": 0.510132, "end_time": "2021-10-08T04:15:01.794103", "exception": false, "start_time": "2021-10-08T04:15:01.283971", "status": "completed"} tags=[]
-pl = sns.scatterplot(data=data,
-                     x=data["Spent"],
-                     y=data["Income"],
-                     hue=data["Clusters"],
-                     palette=pal)
-pl.set_title("Cluster's Profile Based On Income And Spending")
-plt.legend()
-plt.show()
-
-# %% [markdown] papermill={"duration": 0.07143, "end_time": "2021-10-08T04:15:01.938650", "exception": false, "start_time": "2021-10-08T04:15:01.867220", "status": "completed"} tags=[]
-# **Income vs  spending plot shows the clusters pattern**
-# * group 0: high spending & average income
-# * group 1: high spending & high income
-# * group 2: low spending & low income
-# * group 3: high spending & low income
-#
-# Next, I will be looking at the detailed distribution of clusters as per the various products in the data. Namely: Wines, Fruits, Meat, Fish, Sweets and Gold
-
-# %% papermill={"duration": 4.684948, "end_time": "2021-10-08T04:15:06.695522", "exception": false, "start_time": "2021-10-08T04:15:02.010574", "status": "completed"} tags=[]
-plt.figure()
-pl = sns.swarmplot(x=data["Clusters"],
-                   y=data["Spent"],
-                   color="#CBEDDD",
-                   alpha=0.5)
-pl = sns.boxenplot(x=data["Clusters"], y=data["Spent"], palette=pal)
-plt.show()
-
-# %% [markdown] papermill={"duration": 0.07193, "end_time": "2021-10-08T04:15:06.840213", "exception": false, "start_time": "2021-10-08T04:15:06.768283", "status": "completed"} tags=[]
-#
-# From the above plot, it can be clearly seen that cluster 1 is our biggest set of customers closely followed by cluster 0.
-# We can explore what each cluster is spending on for the targeted marketing strategies.
-#
-
-# %% [markdown] papermill={"duration": 0.074689, "end_time": "2021-10-08T04:15:06.988145", "exception": false, "start_time": "2021-10-08T04:15:06.913456", "status": "completed"} tags=[]
-# Let us next explore how did our campaigns do in the past.
-
-# %% papermill={"duration": 0.373835, "end_time": "2021-10-08T04:15:07.435114", "exception": false, "start_time": "2021-10-08T04:15:07.061279", "status": "completed"} tags=[]
-#Creating a feature to get a sum of accepted promotions
-data["Total_Promos"] = data["AcceptedCmp1"] + data["AcceptedCmp2"] + data[
-    "AcceptedCmp3"] + data["AcceptedCmp4"] + data["AcceptedCmp5"]
-#Plotting count of total campaign accepted.
-plt.figure()
-pl = sns.countplot(x=data["Total_Promos"], hue=data["Clusters"], palette=pal)
-pl.set_title("Count Of Promotion Accepted")
-pl.set_xlabel("Number Of Total Accepted Promotions")
-plt.show()
-
-# %% [markdown] papermill={"duration": 0.075057, "end_time": "2021-10-08T04:15:07.585345", "exception": false, "start_time": "2021-10-08T04:15:07.510288", "status": "completed"} tags=[]
-# There has not been an overwhelming response to the campaigns so far. Very few participants overall. Moreover, no one part take in all 5 of them. Perhaps better-targeted and well-planned campaigns are required to boost sales.
-#
-
-# %% papermill={"duration": 0.325298, "end_time": "2021-10-08T04:15:07.983731", "exception": false, "start_time": "2021-10-08T04:15:07.658433", "status": "completed"} tags=[]
-#Plotting the number of deals purchased
-plt.figure()
-pl = sns.boxenplot(y=data["NumDealsPurchases"],
-                   x=data["Clusters"],
-                   palette=pal)
-pl.set_title("Number of Deals Purchased")
-plt.show()
-
-# %% [markdown] papermill={"duration": 0.075506, "end_time": "2021-10-08T04:15:08.136009", "exception": false, "start_time": "2021-10-08T04:15:08.060503", "status": "completed"} tags=[]
-# Unlike campaigns, the deals offered did well. It has best outcome with cluster 0 and cluster 3.
-# However, our star customers cluster 1 are not much into the deals.
-# Nothing seems to attract cluster 2 overwhelmingly
-#
-
-# %% _kg_hide-input=true _kg_hide-output=true papermill={"duration": 3.296251, "end_time": "2021-10-08T04:15:11.506721", "exception": false, "start_time": "2021-10-08T04:15:08.210470", "status": "completed"} tags=[]
-#for more details on the purchasing style
-Places = [
-    "NumWebPurchases", "NumCatalogPurchases", "NumStorePurchases",
-    "NumWebVisitsMonth"
-]
-
-for i in Places:
-    plt.figure()
-    sns.jointplot(x=data[i],
-                  y=data["Spent"],
-                  hue=data["Clusters"],
-                  palette=pal)
-    plt.show()
-
-# %% [markdown] papermill={"duration": 0.083796, "end_time": "2021-10-08T04:15:11.674202", "exception": false, "start_time": "2021-10-08T04:15:11.590406", "status": "completed"} tags=[]
-# <a id="8"></a>
-# ## PROFILING
-#
-# Now that we have formed the clusters and looked at their purchasing habits.
-# Let us see who all are there in these clusters. For that, we will be profiling the clusters formed and come to a conclusion about who is our star customer and who needs more attention from the retail store's marketing team.
-#
-# To decide that I will be plotting some of the features that are indicative of the customer's personal traits in light of the cluster they are in.
-# On the basis of the outcomes, I will be arriving at the conclusions.
-
-# %% papermill={"duration": 25.901281, "end_time": "2021-10-08T04:15:37.659466", "exception": false, "start_time": "2021-10-08T04:15:11.758185", "status": "completed"} tags=[]
-Personal = [
-    "Kidhome", "Teenhome", "Customer_For", "Age", "Children", "Family_Size",
-    "Is_Parent", "Education", "Living_With"
-]
-
-for i in Personal:
-    plt.figure()
-    sns.jointplot(x=data[i],
-                  y=data["Spent"],
-                  hue=data["Clusters"],
-                  kind="kde",
-                  palette=pal)
-    plt.show()
-
-# %% [markdown] papermill={"duration": 0.10691, "end_time": "2021-10-08T04:15:37.873431", "exception": false, "start_time": "2021-10-08T04:15:37.766521", "status": "completed"} tags=[]
-# **Points to be noted:**
-#
-# The following information can be deduced about the customers in different clusters.
-#
-# <img src="https://github.com/KarnikaKapoor/Files/blob/main/Colorful%20Handwritten%20About%20Me%20Blank%20Education%20Presentation%20(3).png?raw=true">
-#
-
-# %% [markdown] papermill={"duration": 0.106499, "end_time": "2021-10-08T04:15:38.087648", "exception": false, "start_time": "2021-10-08T04:15:37.981149", "status": "completed"} tags=[]
-# <a id="9"></a>
-# ## CONCLUSION
-#
-# In this project, I performed unsupervised clustering.
-# I did use dimensionality reduction followed by agglomerative clustering.
-# I came up with 4 clusters and further used them in profiling customers in clusters according to their family structures and income/spending.
-# This can be used in planning better marketing strategies.
-#
-# **<span style="color:#682F2F;"> If you liked this Notebook, please do upvote.</span>**
-#
-# **<span style="color:#682F2F;">If you have any questions, feel free to comment!</span>**
-#
-# **<span style="color:#682F2F;"> Best Wishes!</span>**
-#
-# <a id="10"></a>
-# ## END
diff --git a/_kaggle/customer-segmentation-clustering/requirements.txt b/_kaggle/customer-segmentation-clustering/requirements.txt
deleted file mode 100644
index 99139e6..0000000
--- a/_kaggle/customer-segmentation-clustering/requirements.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-yellowbrick
\ No newline at end of file
diff --git a/_kaggle/feature-selection-and-data-visualization/nb.py b/_kaggle/feature-selection-and-data-visualization/nb.py
deleted file mode 100644
index 398ad1c..0000000
--- a/_kaggle/feature-selection-and-data-visualization/nb.py
+++ /dev/null
@@ -1,536 +0,0 @@
-# ---
-# jupyter:
-#   jupytext:
-#     text_representation:
-#       extension: .py
-#       format_name: percent
-#       format_version: '1.3'
-#       jupytext_version: 1.13.1
-#   kernelspec:
-#     display_name: Python 3
-#     language: python
-#     name: python3
-# ---
-
-# %% [markdown]
-# # FEATURE SELECTION & DATA VISUALIZATION
-
-# %% [markdown] _cell_guid="b72f3424-c0c2-47e8-b18b-bc89253eab3f" _execution_state="idle" _uuid="bbc2abb85c63b8c357f5c2e0140105ed195f7fb4"
-# ## What will you learn from this project?
-# ![deasd.png](attachment:2834b410-2f8d-4dc3-b461-cfa22817f85b.png)
-
-# %% [markdown]
-# ## Introduction
-# In this data analysis report, I usually focus on feature visualization and selection as a different from other kernels. Feature selection with correlation, univariate feature selection, recursive feature elimination, recursive feature elimination with cross validation and tree based feature selection methods are used with random forest classification. Apart from these, principle component analysis are used to observe number of components.
-#
-# **Enjoy your data analysis!!!**
-#
-# ![Cancer_Awareness-2019_blog_v1.0_noText.jpg](attachment:6511de54-b562-4594-bcf9-a12222fbef97.jpg)
-
-# %% [markdown] _cell_guid="8d7fdae8-1d31-4873-a021-d553e2c4087c" _execution_state="idle" _uuid="c28fa7775a99901a882aee31e890ea99fe796d91"
-# ## Data Analysis Content
-# 1. [Python Libraries](#1)
-# 1. [Data Content](#2)
-# 1. [Read and Analyse Data](#3)
-# 1. [Visualization](#4)
-# 1. [Feature Selection and Random Forest Classification](#5)
-#     1. [Feature selection with correlation and random forest classification](#6)
-#     1. [Univariate feature selection and random forest classification](#7)
-#     1. [Recursive feature elimination (RFE) with random forest](#8)
-#     1. [Recursive feature elimination with cross validation and random forest classification](#9)
-#     1. [Tree based feature selection and random forest classification](#10)
-# 1. [Feature Extraction with PCA](#11)
-# 1. [Conclusion](#12)
-
-# %% [markdown]
-# <a id='1'></a>
-# ## Python Libraries
-# * In this section, we import used libraries during this kernel.
-
-# %% _cell_guid="52942f7b-e58d-4275-86f0-ced1bcea06f9" _execution_state="idle" _uuid="d7dc365d2933b6675c57c98d438356e4cc1e6125"
-# This Python 3 environment comes with many helpful analytics libraries installed
-# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python
-# For example, here's several helpful packages to load in
-
-import numpy as np  # linear algebra
-import pandas as pd  # data processing, CSV file I/O (e.g. pd.read_csv)
-import seaborn as sns  # data visualization library
-import matplotlib.pyplot as plt
-# Input data files are available in the "../input/" directory.
-# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory
-import time
-from subprocess import check_output
-
-print(check_output(["ls", "input"]).decode("utf8"))
-#import warnings library
-import warnings
-# ignore all warnings
-warnings.filterwarnings('ignore')
-# Any results you write to the current directory are saved as output.
-
-# %% [markdown]
-# <a id='2'></a>
-# ## Data Content
-# 1. **ID number**
-# 1. **Diagnosis (M = malignant, B = benign)**
-# 1. **radius (mean of distances from center to points on the perimeter)**
-# 1. **texture (standard deviation of gray-scale values)**
-# 1. **perimeter**
-# 1. **area**
-# 1. **smoothness (local variation in radius lengths)**
-# 1. **compactness (perimeter^2 / area - 1.0)**
-# 1. **concavity (severity of concave portions of the contour)**
-# 1. **concave points (number of concave portions of the contour)**
-# 1. **symmetry**
-# 1. **fractal dimension ("coastline approximation" - 1)**
-# <br>
-# * The mean, standard error and "worst" or largest (mean of the three largest values) of these features were computed for each image, resulting in 30 features. For instance, field 3 is Mean Radius, field 13 is Radius SE, field 23 is Worst Radius.
-# * All feature values are recoded with four significant digits.
-# * Missing attribute values: none
-# * Class distribution: 357 benign, 212 malignant
-
-# %% [markdown]
-# <a id='3'></a>
-# ## Read and Analyse Data
-
-# %% _cell_guid="c9bd4680-5a5d-4ce5-8b85-1820d2e478d0" _execution_state="idle" _uuid="4a65810840012b075b5a359994931bec8acf9ab0"
-data = pd.read_csv('input/data.csv')
-
-# %% [markdown] _cell_guid="48131d2a-cb21-4c1f-8213-f0e78647287c" _execution_state="idle" _uuid="81d7851cbda2bc774e989259005e98999b84c0b2"
-# Before making anything like feature selection,feature extraction and classification, firstly we start with basic data analysis.
-# Lets look at features of data.
-
-# %% _cell_guid="d30f1486-bb97-40d7-9125-67e6f15286dc" _execution_state="idle" _uuid="3e01972c830afa1ce55025c0b7a202d4b204dd1d"
-data.head()  # head method show only first 5 rows
-
-# %% [markdown] _cell_guid="d3b8329c-20eb-4f00-b900-ffc9278cd82a" _execution_state="idle" _uuid="16fbbd5b380761a5e22478133c1f8ead52ca7abb"
-# **There are 4 things that take my attention**
-# 1) There is an **id** that cannot be used for classificaiton
-# 2) **Diagnosis** is our class label
-# 3) **Unnamed: 32** feature includes NaN so we do not need it.
-# 4) I do not have any idea about other feature names actually I do not need because machine learning is awesome **:)**
-#
-# Therefore, drop these unnecessary features. However do not forget this is not a feature selection. This is like a browse a pub, we do not choose our drink yet !!!
-
-# %% _cell_guid="60308baf-344a-41fb-8580-cef707ce5aa8" _execution_state="idle" _uuid="54600377cdbec016505dcb970bb1988afbc260a2"
-# feature names as a list
-col = data.columns  # .columns gives columns names in data
-print(col)
-
-# %% _cell_guid="8764b4cf-9963-4c1a-b449-059de8153e4c" _execution_state="idle" _uuid="94ea75618315ac7af54cf80a501c42b40e77ecbc"
-# y includes our labels and x includes our features
-y = data.diagnosis  # M or B
-list = ['Unnamed: 32', 'id', 'diagnosis']
-x = data.drop(list, axis=1)
-x.head()
-
-# %% _cell_guid="31ec8d06-ea25-4b34-84ca-c322b3d8a10f" _execution_state="idle" _uuid="71fecf26e957a2d670182d607aca5a7b92b4a3b6"
-ax = sns.countplot(y, label="Count")  # M = 212, B = 357
-B, M = y.value_counts()
-print('Number of Benign: ', B)
-print('Number of Malignant : ', M)
-
-# %% [markdown] _cell_guid="8c7d5b2b-445c-45ec-81da-5b3853617711" _execution_state="idle" _uuid="af412eff1a8fa83f1f9fb6daaeac78cf498d589a"
-# Okey, now we have features but **what does they mean** or actually **how much do we need to know about these features**
-# The answer is that we do not need to know meaning of these features however in order to imagine in our mind we should know something like variance, standart deviation, number of sample (count) or max min values.
-# These type of information helps to understand about what is going on data. For example , the question is appeared in my mind the **area_mean** feature's max value is 2500 and **smoothness_mean** features' max 0.16340. Therefore **do we need standardization or normalization before visualization, feature selection, feature extraction or classificaiton?** The answer is yes and no not surprising ha :) Anyway lets go step by step and start with visualization.
-
-# %% _cell_guid="c92292c2-d999-42f3-8618-73b231c163e6" _execution_state="idle" _uuid="7d909ed445dd83306413a72986cebc17d1814cc7"
-x.describe()
-
-# %% [markdown] _cell_guid="6179a010-0819-481e-8095-72ba1021fdcd" _execution_state="idle" _uuid="3edac4b24f82f00d32efe9d812aed40fb06fdbed"
-# <a id='4'></a>
-# ## Visualization
-# In order to visualizate data we are going to use seaborn plots that is not used in other kernels to inform you and for diversity of plots. What I use in real life is mostly violin plot and swarm plot. Do not forget we are not selecting feature, we are trying to know data like looking at the drink list at the pub door.
-
-# %% [markdown] _cell_guid="fa50c6cf-ccb6-49fa-b4bb-e7f14a3c4a09" _execution_state="idle" _uuid="d3d8066df83b9a30087610eed09782c1dec7c4cf"
-# Before violin and swarm plot we need to normalization or standirdization. Because differences between values of features are very high to observe on plot. I plot features in 3 group and each group includes 10 features to observe better.
-
-# %% _cell_guid="d58052d6-9e8c-46f4-a2d5-b9e82b247f27" _execution_state="idle" _uuid="d640909614b5ff561e35b33e555458df70b22486"
-# first ten features
-data_dia = y
-data = x
-data_n_2 = (data - data.mean()) / (data.std())  # standardization
-data = pd.concat([y, data_n_2.iloc[:, 0:10]], axis=1)
-data = pd.melt(data,
-               id_vars="diagnosis",
-               var_name="features",
-               value_name='value')
-plt.figure(figsize=(10, 10))
-sns.violinplot(x="features",
-               y="value",
-               hue="diagnosis",
-               data=data,
-               split=True,
-               inner="quart")
-plt.xticks(rotation=90)
-
-# %% [markdown] _cell_guid="8eb078f8-63a8-405d-8cdb-10c9e8a06863" _execution_state="idle" _uuid="6c3ce53008a590504b1afef35e5edff183ed0ce4"
-# Lets interpret the plot above together. For example, in **texture_mean** feature, median of the *Malignant* and *Benign* looks like separated so it can be good for classification. However, in **fractal_dimension_mean** feature,  median of the *Malignant* and *Benign* does not looks like separated so it does not gives good information for classification.
-
-# %% _cell_guid="46ee71d3-93c1-4c2d-bb00-6995f7a1c816" _execution_state="idle" _uuid="0a18301387ce26b58a68e5a2d340b39e86c1f5e0"
-# Second ten features
-data = pd.concat([y, data_n_2.iloc[:, 10:20]], axis=1)
-data = pd.melt(data,
-               id_vars="diagnosis",
-               var_name="features",
-               value_name='value')
-plt.figure(figsize=(10, 10))
-sns.violinplot(x="features",
-               y="value",
-               hue="diagnosis",
-               data=data,
-               split=True,
-               inner="quart")
-plt.xticks(rotation=90)
-
-# %% _cell_guid="58f17ef1-6530-4db8-bcd9-32691363e8a9" _execution_state="idle" _uuid="d1c4e84c3d6bda4b9ff9e284d8c790dd46980c31"
-# Second ten features
-data = pd.concat([y, data_n_2.iloc[:, 20:31]], axis=1)
-data = pd.melt(data,
-               id_vars="diagnosis",
-               var_name="features",
-               value_name='value')
-plt.figure(figsize=(10, 10))
-sns.violinplot(x="features",
-               y="value",
-               hue="diagnosis",
-               data=data,
-               split=True,
-               inner="quart")
-plt.xticks(rotation=90)
-
-# %% _uuid="54c8d1570b21a65ea707c21440d90f6701b15e2b"
-# As an alternative of violin plot, box plot can be used
-# box plots are also useful in terms of seeing outliers
-# I do not visualize all features with box plot
-# In order to show you lets have an example of box plot
-# If you want, you can visualize other features as well.
-plt.figure(figsize=(10, 10))
-sns.boxplot(x="features", y="value", hue="diagnosis", data=data)
-plt.xticks(rotation=90)
-
-# %% [markdown] _cell_guid="7e3412af-5856-4cb1-922d-6a3dd6c5f238" _execution_state="idle" _uuid="80d14a320a514d9e727135db6d64155bea3ca35b"
-# Lets interpret one more thing about plot above, variable of **concavity_worst** and **concave point_worst** looks like similar but how can we decide whether they are correlated with each other or not.
-# (Not always true but, basically if the features are correlated with each other we can drop one of them)
-
-# %% [markdown] _cell_guid="d6282b97-002d-4aeb-a09a-ce4ef138a1ca" _execution_state="idle" _uuid="f76a463fada43aa587f7bd035148698e71db6309"
-# In order to compare two features deeper, lets use joint plot. Look at this in joint plot below, it is really correlated.
-#  Pearsonr value is correlation value and 1 is the highest. Therefore, 0.86 is looks enough to say that they are correlated.
-# Do not forget, we are not choosing features yet, we are just looking to have an idea about them.
-
-# %% _cell_guid="47880bbb-5dbe-4836-938c-0816a03e8fb4" _execution_state="idle" _uuid="859ec665af4be178c3e36b1c2799f44c5ccef901"
-sns.jointplot(x.loc[:, 'concavity_worst'],
-              x.loc[:, 'concave points_worst'],
-              kind="reg",
-              color="#ce1414")
-
-# %% [markdown] _cell_guid="008330ba-393e-4d31-b086-b90332a613c5" _execution_state="idle" _uuid="43de55ee2437101c0450d989216a426d6f1f30c7"
-# What about three or more feauture comparision ? For this purpose we can use pair grid plot. Also it seems very cool :)
-# And we discover one more thing **radius_worst**, **perimeter_worst** and **area_worst** are correlated as it can be seen pair grid plot. We definetely use these discoveries for feature selection.
-
-# %% _cell_guid="3bda33fe-daf9-4f74-acbc-d9d3c8fc83d9" _execution_state="idle" _uuid="381ecb55ced22383c96320ced2299f5da37ce4b6"
-sns.set(style="white")
-df = x.loc[:, ['radius_worst', 'perimeter_worst', 'area_worst']]
-g = sns.PairGrid(df, diag_sharey=False)
-g.map_lower(sns.kdeplot, cmap="Blues_d")
-g.map_upper(plt.scatter)
-g.map_diag(sns.kdeplot, lw=3)
-
-# %% [markdown] _cell_guid="c9ef0921-19bc-4d40-aa72-0bc2333bd4b4" _execution_state="idle" _uuid="f8355f83ac16e414e3b88e67b77d33ef31c3574d"
-# Up to this point, we make some comments and discoveries on data already. If you like what we did, I am sure swarm plot will open the pub's door :)
-
-# %% [markdown] _cell_guid="03abd05a-d67a-4bfc-b951-6394de8c6fc9" _execution_state="idle" _uuid="c3807ef7f6e17b33ae383349bdee7ebfced2a847"
-# In swarm plot, I will do three part like violin plot not to make plot very complex appearance
-
-# %% _cell_guid="ef378d49-8aed-4b9e-96e8-e7d2458fdd89" _execution_state="idle" _uuid="85a2413b70c1b3d69f26a2c122c22d55f930e774"
-sns.set(style="whitegrid", palette="muted")
-data_dia = y
-data = x
-data_n_2 = (data - data.mean()) / (data.std())  # standardization
-data = pd.concat([y, data_n_2.iloc[:, 0:10]], axis=1)
-data = pd.melt(data,
-               id_vars="diagnosis",
-               var_name="features",
-               value_name='value')
-plt.figure(figsize=(10, 10))
-tic = time.time()
-sns.swarmplot(x="features", y="value", hue="diagnosis", data=data)
-
-plt.xticks(rotation=90)
-
-# %% _cell_guid="428c75b6-b5d0-47e3-a568-17b5d1896c0c" _execution_state="idle" _uuid="75dfd5e9e50adceb1d42dd000ce779e79b069cce"
-data = pd.concat([y, data_n_2.iloc[:, 10:20]], axis=1)
-data = pd.melt(data,
-               id_vars="diagnosis",
-               var_name="features",
-               value_name='value')
-plt.figure(figsize=(10, 10))
-sns.swarmplot(x="features", y="value", hue="diagnosis", data=data)
-plt.xticks(rotation=90)
-
-# %% _cell_guid="ee64bbc8-0431-482a-b08f-cdca43e41390" _execution_state="idle" _uuid="209e9e9120d6e889696d2d1190e663b5c1885a82"
-data = pd.concat([y, data_n_2.iloc[:, 20:31]], axis=1)
-data = pd.melt(data,
-               id_vars="diagnosis",
-               var_name="features",
-               value_name='value')
-plt.figure(figsize=(10, 10))
-sns.swarmplot(x="features", y="value", hue="diagnosis", data=data)
-toc = time.time()
-plt.xticks(rotation=90)
-print("swarm plot time: ", toc - tic, " s")
-
-# %% [markdown] _cell_guid="5c9efa5e-4938-477e-ab9a-4b084cc0b870" _execution_state="idle" _uuid="d23e2f9b040a92feb0d7ceb8e01e74c758f5dbc3"
-# They looks cool right. And you can see variance more clear. Let me ask you a question, **in these three plots which feature looks like more clear in terms of classification.** In my opinion **area_worst** in last swarm plot looks like malignant and benign are seprated not totaly but mostly. Hovewer, **smoothness_se** in swarm plot 2 looks like malignant and benign are mixed so it is hard to classfy while using this feature.
-
-# %% [markdown] _cell_guid="c4c68f34-e876-4e5a-a4a7-09c07381425a" _execution_state="idle" _uuid="b46f98eb7ca8d36dc7bf1516895599524bab694d"
-# **What if we want to observe all correlation between features?** Yes, you are right. The answer is heatmap that is old but powerful plot method.
-
-# %% _cell_guid="9e1e7d8a-bbf2-4aab-90e7-78d4c4ccf416" _execution_state="idle" _uuid="0eeb70ddffc8ac332ee076f2f6b2833a6ffddd2d"
-#correlation map
-f, ax = plt.subplots(figsize=(18, 18))
-sns.heatmap(x.corr(), annot=True, linewidths=.5, fmt='.1f', ax=ax)
-
-# %% [markdown] _cell_guid="8ee2e02b-9fc6-42f7-8a83-f6dd77df6c11" _execution_state="idle" _uuid="b4e3afecb204a8262330d22e4539554b2af7975a"
-# Well, finaly we are in the pub and lets choose our drinks at feature selection part while using heatmap(correlation matrix).
-
-# %% [markdown] _cell_guid="6786734a-40a9-46b6-a13a-97ee9c569636" _execution_state="idle" _uuid="84b145dd0c13a3a0d4dd4f9b9fd1bd782e11fcf8"
-# <a id='5'></a>
-# ## Feature Selection and Random Forest Classification
-# Today our purpuse is to try new cocktails. For example, we are finaly in the pub and we want to drink different tastes. Therefore, we need to compare ingredients of drinks. If one of them includes lemon, after drinking it we need to eliminate other drinks which includes lemon so as to experience very different tastes.
-
-# %% [markdown] _cell_guid="c7b2df4e-270e-4c94-8789-177f5e90ac46" _execution_state="idle" _uuid="a042df90ef7138d6f101463e93936119176bdc0d"
-# In this part we will select feature with different methods that are feature selection with correlation, univariate feature selection, recursive feature elimination (RFE), recursive feature elimination with cross validation (RFECV) and tree based feature selection. We will use random forest classification in order to train our model and predict.
-
-# %% [markdown] _cell_guid="94d217e3-b2b3-4016-b72e-f8d521d17af7" _execution_state="idle" _uuid="39003c7b75f265bf0826f407433e65923c4dd017"
-# <a id='6'></a>
-# ### 1) Feature selection with correlation and random forest classification
-
-# %% [markdown] _cell_guid="785bd27a-30d9-4e08-a864-cde7e5630aad" _execution_state="idle" _uuid="1e6ef08c98cb4bf0dedf275e4c08fae743bb3801"
-# As it can be seen in map heat figure **radius_mean, perimeter_mean and area_mean** are correlated with each other so we will use only **area_mean**. If you ask how i choose **area_mean** as a feature to use, well actually there is no correct answer, I just look at swarm plots and **area_mean** looks like clear for me but we cannot make exact separation among other correlated features without trying. So lets find other correlated features and look accuracy with random forest classifier.
-
-# %% [markdown] _cell_guid="eea2971b-b703-4e1b-b048-128501506f33" _execution_state="idle" _uuid="acde9c0b406d72122473f8292d641a9fcb8a8682"
-# **Compactness_mean, concavity_mean and concave points_mean** are correlated with each other.Therefore I only choose **concavity_mean**. Apart from these, **radius_se, perimeter_se and area_se** are correlated and I only use **area_se**.  **radius_worst, perimeter_worst and area_worst** are correlated so I use **area_worst**.  **Compactness_worst, concavity_worst and concave points_worst** so I use **concavity_worst**.  **Compactness_se, concavity_se and concave points_se** so I use **concavity_se**. **texture_mean and texture_worst are correlated** and I use **texture_mean**. **area_worst and area_mean** are correlated, I use **area_mean**.
-#
-#
-#
-
-# %% _cell_guid="ef8d06df-bfcc-4e9a-a3ba-5016ec0c5bd5" _execution_state="idle" _uuid="117f3e858e806f3f26a68dadf3fc89d471010156"
-drop_list1 = [
-    'perimeter_mean', 'radius_mean', 'compactness_mean', 'concave points_mean',
-    'radius_se', 'perimeter_se', 'radius_worst', 'perimeter_worst',
-    'compactness_worst', 'concave points_worst', 'compactness_se',
-    'concave points_se', 'texture_worst', 'area_worst'
-]
-x_1 = x.drop(drop_list1, axis=1)  # do not modify x, we will use it later
-x_1.head()
-
-# %% [markdown] _cell_guid="6de99062-7a5a-4b70-879c-54d6c8a4a7e2" _execution_state="idle" _uuid="1ab3852ed7fbeba8718e6722e8a40521033bdf29"
-# After drop correlated features, as it can be seen in below correlation matrix, there are no more correlated features. Actually, I know and you see there is correlation value 0.9 but lets see together what happen if we do not drop it.
-
-# %% _cell_guid="733f0784-4a3f-410c-a220-f98591825f2e" _execution_state="idle" _uuid="eec5424039036e1af43ba0795b76393805308f97"
-#correlation map
-f, ax = plt.subplots(figsize=(14, 14))
-sns.heatmap(x_1.corr(), annot=True, linewidths=.5, fmt='.1f', ax=ax)
-
-# %% [markdown] _cell_guid="f6551fe9-f3a8-4738-ace3-b0427be4aeb4" _execution_state="idle" _uuid="0eaf4ae8e33c2d6352a862953c1fe5eecf46ed27"
-# Well, we choose our features but **did we choose correctly ?** Lets use random forest and find accuracy according to chosen features.
-
-# %% _cell_guid="111af932-96f8-4105-8deb-ba1172edd203" _execution_state="idle" _uuid="c7a6af60a44959f81593d788934a49c9259d8b43"
-from sklearn.model_selection import train_test_split
-from sklearn.ensemble import RandomForestClassifier
-from sklearn.metrics import f1_score, confusion_matrix
-from sklearn.metrics import accuracy_score
-
-# split data train 70 % and test 30 %
-x_train, x_test, y_train, y_test = train_test_split(x_1,
-                                                    y,
-                                                    test_size=0.3,
-                                                    random_state=42)
-
-#random forest classifier with n_estimators=10 (default)
-clf_rf = RandomForestClassifier(random_state=43)
-clr_rf = clf_rf.fit(x_train, y_train)
-
-ac = accuracy_score(y_test, clf_rf.predict(x_test))
-print('Accuracy is: ', ac)
-cm = confusion_matrix(y_test, clf_rf.predict(x_test))
-sns.heatmap(cm, annot=True, fmt="d")
-
-# %% [markdown] _cell_guid="1503384d-ca2b-4b52-82b5-f1131b014269" _execution_state="idle" _uuid="21cda299619940f7b22acc9a804ee56bff71d3e7"
-# Accuracy is almost 95% and as it can be seen in confusion matrix, we make few wrong prediction.
-# Now lets see other feature selection methods to find better results.
-
-# %% [markdown] _cell_guid="3eed9ac3-e601-4e16-85bc-26a1a6fff850" _execution_state="idle" _uuid="decd86422aee506b061c905e8573abb3612734e4"
-# <a id='7'></a>
-# ### 2) Univariate feature selection and random forest classification
-# In univariate feature selection, we will use SelectKBest that removes all but the k highest scoring features.
-# <http://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.SelectKBest.html#sklearn.feature_selection.SelectKBest>
-
-# %% [markdown] _cell_guid="f053659d-9dfe-4858-a220-ef327df3bc36" _execution_state="idle" _uuid="f681583a7b20e4fb86e557b910021a263573cf18"
-# In this method we need to choose how many features we will use. For example, will k (number of features) be 5 or 10 or 15? The answer is only trying or intuitively. I do not try all combinations but I only choose k = 5 and find best 5 features.
-
-# %% _cell_guid="4f43c8bd-48f7-4ed9-aa2d-6aa8a29c0c58" _execution_state="idle" _uuid="8159f9efb106f1219dc4e8c2a340399b88f224d8" jupyter={"outputs_hidden": true}
-from sklearn.feature_selection import SelectKBest
-from sklearn.feature_selection import chi2
-# find best scored 5 features
-select_feature = SelectKBest(chi2, k=5).fit(x_train, y_train)
-
-# %% _cell_guid="c9684618-06fe-4b0a-835f-ceea46da397c" _execution_state="idle" _uuid="d9dcd1495cbf33c190a0d1211df4bac5e79bc4e5"
-print('Score list:', select_feature.scores_)
-print('Feature list:', x_train.columns)
-
-# %% [markdown] _cell_guid="b0c426ef-3072-4f6e-bf24-b5d927a98316" _execution_state="idle" _uuid="0f46ea1a9b1282a377549406d1c5e093380954b6"
-# Best 5 feature to classify is that **area_mean, area_se, texture_mean, concavity_worst and concavity_mean**. So lets se what happens if we use only these best scored 5 feature.
-
-# %% _cell_guid="efc70e04-bc9c-4f93-bcd3-b1d7160d0d5c" _execution_state="idle" _uuid="9a2bd21537f7c600f3c9baaf833c001084d6ba00"
-x_train_2 = select_feature.transform(x_train)
-x_test_2 = select_feature.transform(x_test)
-#random forest classifier with n_estimators=10 (default)
-clf_rf_2 = RandomForestClassifier()
-clr_rf_2 = clf_rf_2.fit(x_train_2, y_train)
-ac_2 = accuracy_score(y_test, clf_rf_2.predict(x_test_2))
-print('Accuracy is: ', ac_2)
-cm_2 = confusion_matrix(y_test, clf_rf_2.predict(x_test_2))
-sns.heatmap(cm_2, annot=True, fmt="d")
-
-# %% [markdown] _cell_guid="d8888dc1-b50b-46b4-b202-4e33c2630406" _execution_state="idle" _uuid="575005da62c41d12bbb3999b3e26148e12930ce3"
-# Accuracy is almost 96% and as it can be seen in confusion matrix, we make few wrong prediction. What we did up to now is that we choose features according to correlation matrix and according to selectkBest method. Although we use 5 features in selectkBest method accuracies look similar.
-# Now lets see other feature selection methods to find better results.
-
-# %% [markdown] _cell_guid="702ad2b3-5b12-4d15-93b1-e7d62dfd1040" _execution_state="idle" _uuid="7a3c3050dd9d694e52962c7c712b1ea16aab6fdf"
-# <a id='8'></a>
-# ### 3) Recursive feature elimination (RFE) with random forest
-# <http://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.RFE.html>
-# Basically, it uses one of the classification methods (random forest in our example), assign weights to each of features. Whose absolute weights are the smallest are pruned from the current set features. That procedure is recursively repeated on the pruned set until the desired number of features
-
-# %% [markdown] _cell_guid="8a34a801-c568-4598-8a07-85fc20ad0386" _execution_state="idle" _uuid="3ea45c46bb231c767160fe13ad3b21a70f0d0375"
-# Like previous method, we will use 5 features. However, which 5 features will we use ? We will choose them with RFE method.
-
-# %% _cell_guid="8df88bb5-8003-4696-9efe-63ebf8d609a5" _execution_state="idle" _uuid="c384a5240d1c1e9e2a6750e5d218dadaf24d2035"
-from sklearn.feature_selection import RFE
-# Create the RFE object and rank each pixel
-clf_rf_3 = RandomForestClassifier()
-rfe = RFE(estimator=clf_rf_3, n_features_to_select=5, step=1)
-rfe = rfe.fit(x_train, y_train)
-
-# %% _cell_guid="51d63d0b-4e00-4dc1-816c-809287b60806" _execution_state="idle" _uuid="29ba35a98954d0ae686ce46295179d1f1a27b74c"
-print('Chosen best 5 feature by rfe:', x_train.columns[rfe.support_])
-
-# %% [markdown] _cell_guid="92aa6013-3e16-4005-ab1b-b7ce53e78bd3" _execution_state="idle" _uuid="ce670f778a661e8ddc3b7b21a43ccb48a551581a"
-# Chosen 5 best features by rfe is **texture_mean, area_mean, concavity_mean, area_se, concavity_worst**. They are exactly similar with previous (selectkBest) method. Therefore we do not need to calculate accuracy again. Shortly, we can say that we make good feature selection with rfe and selectkBest methods. However as you can see there is a problem, okey I except we find best 5 feature with two different method and these features are same but why it is **5**. Maybe if we use best 2 or best 15 feature we will have better accuracy. Therefore lets see how many feature we need to use with rfecv method.
-
-# %% [markdown] _cell_guid="22a4f840-2a37-4047-9804-129e7f68f74a" _execution_state="idle" _uuid="42a8c3f2ef0e5978b620eea737e6e234dc79cfe8"
-# <a id='9'></a>
-# ### 4) Recursive feature elimination with cross validation and random forest classification
-# <http://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.RFECV.html>
-# Now we will not only **find best features** but we also find **how many features do we need** for best accuracy.
-
-# %% _cell_guid="7a5d4d69-7734-4465-89cc-f46b4af4c548" _execution_state="idle" _uuid="0d7803966979745a8bdbdbc44a1927558485640a"
-from sklearn.feature_selection import RFECV
-
-# The "accuracy" scoring is proportional to the number of correct classifications
-clf_rf_4 = RandomForestClassifier()
-rfecv = RFECV(estimator=clf_rf_4, step=1, cv=5,
-              scoring='accuracy')  #5-fold cross-validation
-rfecv = rfecv.fit(x_train, y_train)
-
-print('Optimal number of features :', rfecv.n_features_)
-print('Best features :', x_train.columns[rfecv.support_])
-
-# %% [markdown] _cell_guid="be0f7ce5-55c0-48c9-a125-616750f13943" _execution_state="idle" _uuid="cef5972f5e9fa830e92bae00ca5dd7d2b0ac8c58"
-# Finally, we find best 11 features that are **texture_mean, area_mean, concavity_mean, texture_se, area_se, concavity_se, symmetry_se, smoothness_worst, concavity_worst, symmetry_worst and fractal_dimension_worst** for best classification. Lets look at best accuracy with plot.
-#
-
-# %% _cell_guid="5b69144b-72e4-4ac3-b8a8-c9ebbf8ffa3b" _execution_state="idle" _uuid="f362bfa341032f2bb1bacc1c50675a1916f5c536"
-# Plot number of features VS. cross-validation scores
-import matplotlib.pyplot as plt
-
-plt.figure()
-plt.xlabel("Number of features selected")
-plt.ylabel("Cross validation score of number of selected features")
-plt.plot(range(1, len(rfecv.grid_scores_) + 1), rfecv.grid_scores_)
-plt.show()
-
-# %% [markdown] _cell_guid="580f19b6-1182-43d5-932c-21e2fb5bb2d9" _execution_state="idle" _uuid="f071ec67bd63d5c458c2cb6303fe6f54458db57b"
-# Lets look at what we did up to this point. Lets accept that guys this data is very easy to classification. However, our first purpose is actually not finding good accuracy. Our purpose is learning how to make **feature selection and understanding data.** Then last make our last feature selection method.
-
-# %% [markdown] _cell_guid="2637e8bc-d986-41c0-acef-ce76afc4c350" _execution_state="idle" _uuid="8bc3105398fc618e19deec4de957950cfb45c054"
-# <a id='10'></a>
-# ### 5) Tree based feature selection and random forest classification
-# <http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html>
-# In random forest classification method there is a **feature_importances_** attributes that is the feature importances (the higher, the more important the feature). **!!! To use feature_importance method, in training data there should not be correlated features. Random forest choose randomly at each iteration, therefore sequence of feature importance list can change.**
-#
-
-# %% _cell_guid="df8abc8d-3279-4c31-a6b6-e4f272ca0b47" _execution_state="idle" _uuid="31d4b248f723930ff7120ffaff2c260f07e3f0fc"
-clf_rf_5 = RandomForestClassifier()
-clr_rf_5 = clf_rf_5.fit(x_train, y_train)
-importances = clr_rf_5.feature_importances_
-std = np.std([tree.feature_importances_ for tree in clf_rf.estimators_],
-             axis=0)
-indices = np.argsort(importances)[::-1]
-
-# Print the feature ranking
-print("Feature ranking:")
-
-for f in range(x_train.shape[1]):
-    print("%d. feature %d (%f)" % (f + 1, indices[f], importances[indices[f]]))
-
-# Plot the feature importances of the forest
-
-plt.figure(1, figsize=(14, 13))
-plt.title("Feature importances")
-plt.bar(range(x_train.shape[1]),
-        importances[indices],
-        color="g",
-        yerr=std[indices],
-        align="center")
-plt.xticks(range(x_train.shape[1]), x_train.columns[indices], rotation=90)
-plt.xlim([-1, x_train.shape[1]])
-plt.show()
-
-# %% [markdown] _cell_guid="00008678-9012-4e50-aaab-010b3353ac98" _execution_state="idle" _uuid="760b045b33388f6fb7b53acdf931e8204eea80cd"
-# As you can seen in plot above, after 5 best features importance of features decrease. Therefore we can focus these 5 features. As I sad before, I give importance to understand features and find best of them.
-
-# %% [markdown] _cell_guid="21ef3e97-1eba-4f30-9714-cc45a3c1a594" _execution_state="idle" _uuid="c3fd1de4be5be26252a4105501a217a026d116b1"
-# <a id='11'></a>
-# ## Feature Extraction with PCA
-# <http://scikit-learn.org/stable/modules/generated/sklearn.decomposition.PCA.html>
-# We will use principle component analysis (PCA) for feature extraction. Before PCA, we need to normalize data for better performance of PCA.
-#
-
-# %% _cell_guid="aa440ca8-8282-4cc3-9683-d5ebdd992140" _execution_state="idle" _uuid="cf72d82fea5d8330db8ec324fb18abc6e969bac6"
-# split data train 70 % and test 30 %
-x_train, x_test, y_train, y_test = train_test_split(x,
-                                                    y,
-                                                    test_size=0.3,
-                                                    random_state=42)
-#normalization
-x_train_N = (x_train - x_train.mean()) / (x_train.max() - x_train.min())
-x_test_N = (x_test - x_test.mean()) / (x_test.max() - x_test.min())
-
-from sklearn.decomposition import PCA
-
-pca = PCA()
-pca.fit(x_train_N)
-
-plt.figure(1, figsize=(14, 13))
-plt.clf()
-plt.axes([.2, .2, .7, .7])
-plt.plot(pca.explained_variance_ratio_, linewidth=2)
-plt.axis('tight')
-plt.xlabel('n_components')
-plt.ylabel('explained_variance_ratio_')
-
-# %% [markdown] _cell_guid="264e1dba-1f40-4a28-bd4b-5261a7df185b" _execution_state="idle" _uuid="a50cb8de4ec6e2e8727a2b3b021ee42c4ce21b29"
-# * According to variance ration, 3 component can be chosen.
-# * If you have any doubt about PCA, you can check my intuitive way of PCA tutorial.
-
-# %% [markdown] _cell_guid="224c9c75-256d-4650-af3e-c4c39b565661" _execution_state="idle" _uuid="5e0f4ba2b385fe9312ced2455eed4fb87d39a0b8"
-# <a id='12'></a>
-# # Conclusion
-# Shortly, I tried to show importance of feature selection and data visualization.
-# Default data includes 33 feature but after feature selection we drop this number from 33 to 5 with accuracy 95%. In this kernel we just tried basic things, I am sure with these data visualization and feature selection methods, you can easily ecxeed the % 95 accuracy. Maybe you can use other classification methods.
-# ### I hope you enjoy in this kernel
-# ## If you have any question or advise, I will be apreciate to listen them ...
diff --git a/_kaggle/intro-to-time-series-forecasting/nb.py b/_kaggle/intro-to-time-series-forecasting/nb.py
deleted file mode 100644
index 30d7ad6..0000000
--- a/_kaggle/intro-to-time-series-forecasting/nb.py
+++ /dev/null
@@ -1,1650 +0,0 @@
-# -*- coding: utf-8 -*-
-# ---
-# jupyter:
-#   jupytext:
-#     text_representation:
-#       extension: .py
-#       format_name: percent
-#       format_version: '1.3'
-#       jupytext_version: 1.13.1
-#   kernelspec:
-#     display_name: Python 3
-#     language: python
-#     name: python3
-# ---
-
-# %% [markdown] papermill={"duration": 0.049112, "end_time": "2021-01-17T15:24:40.020719", "exception": false, "start_time": "2021-01-17T15:24:39.971607", "status": "completed"} tags=[]
-# # Introduction
-# In this notebook, we will learn how to work with and predict time series. Time series are a collection of **time-dependent data** points. That means that each data point is assigned to a specific timestamp. Ideally, these data points are in chronological order and in contant time intervals (e.g. every minute or everyday). The time series forecasting problem **analyzes patterns in the past data to make predictions about the future**. The most popular example is probably stock price prediction. Other examples are sales of seasonal clothing or weather forecasts. In contrast to regression problems, time series are time-dependent and show specific characteristics, such as **trend and seasonality**.
-#
-# **Overview**
-# * [Problem Definition](#Problem-Definition)<br>
-# * [Data Collection](#Data-Collection)<br>
-# * [Data Preprocessing](#Data-Preprocessing)<br>
-#     * [Chronological Order and Equidistant Timestamps](#[Chronological-Order-and-Equidistant-Timestamps])<br>
-#     * [Handling Missing Values](#Handling-Missing-Values)<br>
-#     * [Resampling](#Resampling)<br>
-#     * [Stationarity](#Stationarity)<br>
-# * [Feature Engineering](#Feature-Engineering)<br>
-#     * [Time Features](#Time-Features)<br>
-#     * [Decomposition](#Decomposition)<br>
-#     * [Lag](#Lag)<br>
-# * [Exploratory Data Analysis](#Exploratory-Data-Analysis)<br>
-#     * [Autocorrelation Analysis](#Autocorrelation-Analysis)<br>
-# * [Cross Validation](#Cross-Validation)<br>
-# * [Models](#Models)<br>
-#     * [Models for Univariate Time Series](#Models-for-Univariate-Time-Series)<br>
-#         * [Naive Approach](#Naive-Approach)<br>
-#         * [Moving Average](#Moving-Average)<br>
-#         * [Exponential Smoothing  (IN WORK)](#MExponential-Smoothing)<br>
-#         * [ARIMA](#ARIMA)<br>
-#     * [Models for Multivariate Time Series](#Models-for-Multivariate-Time-Series)<br>
-#         * [Vector Autoregression (VAR)](#Vector-Autoregression)<br>
-
-# %% [markdown] papermill={"duration": 0.045927, "end_time": "2021-01-17T15:24:40.113569", "exception": false, "start_time": "2021-01-17T15:24:40.067642", "status": "completed"} tags=[]
-# # Problem Definition
-# For this tutorial, we will build a model to predict the depth to groundwater of an aquifer located in Petrignano, Italy. The question we want to answer is
-# > What is the future depth to groundwater of a well belonging to the aquifier in Petrigrano over the next quarter?
-#
-# > The wells field of the alluvial plain between Ospedalicchio di Bastia Umbra and Petrignano is fed by three underground aquifers separated by low permeability septa. The aquifer can be considered a water table groundwater and is also fed by the Chiascio river. The groundwater levels are influenced by the following parameters: rainfall, depth to groundwater, temperatures and drainage volumes, level of the Chiascio river.
-#
-# > Indeed, both rainfall and temperature affect features like level, flow, depth to groundwater and hydrometry some time after it fell down.
-#
-# # Data Collection
-# In a typical workflow for time series, this would be the time for data collection. In this example, we will skip the data collection step and use data from the [Acea Smart Water Analytics challenge](https://www.kaggle.com/c/acea-water-prediction/). Therefore, this section will be a dataset overview.
-#
-# Although the dataset contains multiple waterbodies, we will only be looking at the Aquifer_Petrignano.csv file.
-#
-# Time series data usually comes in **tabular** format (e.g. csv files).
-
-# %% _cell_guid="b1076dfc-b9ad-4769-8c92-a6c4dae69d19" _kg_hide-input=true _uuid="8f2839f25d086af736a60e9eeb907d3b93b6e0e5" papermill={"duration": 1.178488, "end_time": "2021-01-17T15:24:41.339596", "exception": false, "start_time": "2021-01-17T15:24:40.161108", "status": "completed"} tags=[]
-import numpy as np  # linear algebra
-import pandas as pd  # data processing, CSV file I/O (e.g. pd.read_csv)
-
-import seaborn as sns  # Visualization
-import matplotlib.pyplot as plt  # Visualization
-
-from sklearn.metrics import mean_absolute_error, mean_squared_error
-import math
-
-import warnings  # Supress warnings
-
-warnings.filterwarnings('ignore')
-
-df = pd.read_csv("input/Aquifer_Petrignano.csv")
-
-### Simplifications for the sake of the tutorial ###
-# Drop data before 2009 for the purpose of this tutorial
-df = df[df.Rainfall_Bastia_Umbra.notna()].reset_index(drop=True)
-# Drop one of the target columns, so we can focus on only one target
-df = df.drop(['Depth_to_Groundwater_P24', 'Temperature_Petrignano'], axis=1)
-
-# Simplify column names
-df.columns = [
-    'Date', 'Rainfall', 'Depth_to_Groundwater', 'Temperature',
-    'Drainage_Volume', 'River_Hydrometry'
-]
-
-targets = ['Depth_to_Groundwater']
-features = [feature for feature in df.columns if feature not in targets]
-df.head()
-
-# %% [markdown] papermill={"duration": 0.04978, "end_time": "2021-01-17T15:24:41.437658", "exception": false, "start_time": "2021-01-17T15:24:41.387878", "status": "completed"} tags=[]
-# Since we are working with time series, the most essential features are the time related feature. In this example, we have the column `Date` which  uniquely identifies a day. Ideally, the data is already in chronological order and the time stamps are equidistant in time series. This is already the case in our data: The time interval is one day and the data is already in chronological order. Therefore, we do not have to do this additional data preparation step.
-#
-#
-# This column is provided in string format. Let's convert it to the `datetime64[ns]` data type.
-
-# %% papermill={"duration": 0.232976, "end_time": "2021-01-17T15:24:41.719795", "exception": false, "start_time": "2021-01-17T15:24:41.486819", "status": "completed"} tags=[]
-from datetime import datetime, date
-
-df['Date'] = pd.to_datetime(df.Date, format='%d/%m/%Y')
-df.head().style.set_properties(subset=['Date'],
-                               **{'background-color': 'dodgerblue'})
-
-# %% [markdown] papermill={"duration": 0.049347, "end_time": "2021-01-17T15:24:41.818199", "exception": false, "start_time": "2021-01-17T15:24:41.768852", "status": "completed"} tags=[]
-# Features:
-# * **Rainfall** indicates the quantity of rain falling (mm)
-# * **Temperature** indicates the temperature (°C)
-# * **Volume** indicates the volume of water taken from the drinking water treatment plant (m$^3$)
-# * **Hydrometry** indicates the groundwater level (m)
-#
-# Target:
-# * **Depth to Groundwater** indicates the groundwater level (m from the ground floor)
-#
-
-# %% _kg_hide-input=true _kg_hide-output=false papermill={"duration": 2.442321, "end_time": "2021-01-17T15:24:44.309917", "exception": false, "start_time": "2021-01-17T15:24:41.867596", "status": "completed"} tags=[]
-f, ax = plt.subplots(nrows=5, ncols=1, figsize=(15, 25))
-
-sns.lineplot(x=df.Date,
-             y=df.Rainfall.fillna(np.inf),
-             ax=ax[0],
-             color='dodgerblue')
-ax[0].set_title('Feature: Rainfall', fontsize=14)
-ax[0].set_ylabel(ylabel='Rainfall', fontsize=14)
-
-sns.lineplot(x=df.Date,
-             y=df.Temperature.fillna(np.inf),
-             ax=ax[1],
-             color='dodgerblue',
-             label='Bastia Umbra')
-ax[1].set_title('Feature: Temperature', fontsize=14)
-ax[1].set_ylabel(ylabel='Temperature', fontsize=14)
-
-sns.lineplot(x=df.Date,
-             y=df.Drainage_Volume.fillna(np.inf),
-             ax=ax[2],
-             color='dodgerblue')
-ax[2].set_title('Feature: Volume', fontsize=14)
-ax[2].set_ylabel(ylabel='Volume', fontsize=14)
-
-sns.lineplot(x=df.Date,
-             y=df.River_Hydrometry.fillna(np.inf),
-             ax=ax[3],
-             color='dodgerblue')
-ax[3].set_title('Feature: Hydrometry', fontsize=14)
-ax[3].set_ylabel(ylabel='Hydrometry', fontsize=14)
-
-sns.lineplot(x=df.Date,
-             y=df.Depth_to_Groundwater.fillna(np.inf),
-             ax=ax[4],
-             color='dodgerblue')
-ax[4].set_title('Target: Depth to Groundwater', fontsize=14)
-ax[4].set_ylabel(ylabel='Depth to Groundwater', fontsize=14)
-
-for i in range(5):
-    ax[i].set_xlim([date(2009, 1, 1), date(2020, 6, 30)])
-
-plt.show()
-
-# %% [markdown] papermill={"duration": 0.053201, "end_time": "2021-01-17T15:24:44.417274", "exception": false, "start_time": "2021-01-17T15:24:44.364073", "status": "completed"} tags=[]
-# # Data Preprocessing
-#
-# ## Chronological Order and Equidistant Timestamps
-# The data should be in **chronological order** and the **timestamps should be equidistant** in time series. The chronological order can be achieved by sorting the dataframe by the timestamps. Equidisant timestamps indicates constant time intervals. To check this, the difference between each timestamp can be taken. If this is not the case, you can decide on a constant time interval and resample the data (see [Resampling](#Resampling)).
-#
-# This is already the case in our data: The time interval is one day and the data is already in chronological order. Therefore, we do not have to do this additional data preparation step.
-
-# %% papermill={"duration": 0.087239, "end_time": "2021-01-17T15:24:44.567105", "exception": false, "start_time": "2021-01-17T15:24:44.479866", "status": "completed"} tags=[]
-# Sort values by timestamp (not necessary in this case)
-df = df.sort_values(by='Date')
-
-# Check time intervals
-df['Time_Interval'] = df.Date - df.Date.shift(1)
-
-df[['Date', 'Time_Interval']].head()
-
-# %% _kg_hide-input=true papermill={"duration": 0.069336, "end_time": "2021-01-17T15:24:44.701883", "exception": false, "start_time": "2021-01-17T15:24:44.632547", "status": "completed"} tags=[]
-print(f"{df['Time_Interval'].value_counts()}")
-df = df.drop('Time_Interval', axis=1)
-
-# %% [markdown] papermill={"duration": 0.055138, "end_time": "2021-01-17T15:24:44.814942", "exception": false, "start_time": "2021-01-17T15:24:44.759804", "status": "completed"} tags=[]
-# ## Handling Missing Values
-#
-# We can see that `Depth_to_Groundwater` has missing values.
-#
-# Furthermore, plotting the time series reveals that there seem to be some **implausible zero values** for `Drainage_Volume`, and `River_Hydrometry`. We will have to clean them by replacing them by `nan` values and filling them afterwards.
-
-# %% _kg_hide-input=true _kg_hide-output=false papermill={"duration": 1.668274, "end_time": "2021-01-17T15:24:46.538039", "exception": false, "start_time": "2021-01-17T15:24:44.869765", "status": "completed"} tags=[]
-f, ax = plt.subplots(nrows=2, ncols=1, figsize=(15, 15))
-old = df.River_Hydrometry.copy()
-df['River_Hydrometry'] = np.where((df.River_Hydrometry == 0), np.nan,
-                                  df.River_Hydrometry)
-
-sns.lineplot(x=df.Date,
-             y=old.fillna(np.inf),
-             ax=ax[0],
-             color='darkorange',
-             label='original')
-sns.lineplot(x=df.Date,
-             y=df.River_Hydrometry.fillna(np.inf),
-             ax=ax[0],
-             color='dodgerblue',
-             label='modified')
-ax[0].set_title('Feature: Hydrometry', fontsize=14)
-ax[0].set_ylabel(ylabel='Hydrometry', fontsize=14)
-
-old = df.Drainage_Volume.copy()
-df['Drainage_Volume'] = np.where((df.Drainage_Volume == 0), np.nan,
-                                 df.Drainage_Volume)
-
-sns.lineplot(x=df.Date,
-             y=old.fillna(np.inf),
-             ax=ax[1],
-             color='darkorange',
-             label='original')
-sns.lineplot(x=df.Date,
-             y=df.Drainage_Volume.fillna(np.inf),
-             ax=ax[1],
-             color='dodgerblue',
-             label='modified')
-ax[1].set_title('Feature: Volume', fontsize=14)
-ax[1].set_ylabel(ylabel='Volume', fontsize=14)
-
-for i in range(2):
-    ax[i].set_xlim([date(2009, 1, 1), date(2020, 6, 30)])
-
-plt.show()
-
-# %% [markdown] papermill={"duration": 0.058049, "end_time": "2021-01-17T15:24:46.655445", "exception": false, "start_time": "2021-01-17T15:24:46.597396", "status": "completed"} tags=[]
-# Now we have to think about what to do with these missing values.
-
-# %% _cell_guid="79c7e3d0-c299-4dcb-8224-4455121ee9b0" _kg_hide-input=true _uuid="d629ff2d2480ee46fbb7e2d37f6b5fab8052498a" papermill={"duration": 1.40932, "end_time": "2021-01-17T15:24:48.123363", "exception": false, "start_time": "2021-01-17T15:24:46.714043", "status": "completed"} tags=[]
-f, ax = plt.subplots(nrows=1, ncols=1, figsize=(16, 5))
-sns.heatmap(df.T.isna(), cmap='Blues')
-ax.set_title('Fields with Missing Values', fontsize=16)
-#for tick in ax.xaxis.get_major_ticks():
-#    tick.label.set_fontsize(14)
-for tick in ax.yaxis.get_major_ticks():
-    tick.label.set_fontsize(14)
-plt.show()
-
-# %% [markdown] papermill={"duration": 0.059385, "end_time": "2021-01-17T15:24:48.243078", "exception": false, "start_time": "2021-01-17T15:24:48.183693", "status": "completed"} tags=[]
-# * **Option 1: Fill NaN with Outlier or Zero**
-#
-#     In this specific example filling the missing value with an outlier value such as -999 is not a good idea. However, many notebooks in this challenge have been using -999.
-#
-# * **Option 2: Fill NaN with Mean Value**
-#
-#     Also in this example, we can see that filling NaNs with the mean value is also not sufficient.
-#
-# * **Option 3: Fill NaN with Last Value with `.ffill()`**
-#
-#     Filling NaNs with the last value is already a little bit better in this case.
-#
-# * **Option 4: Fill NaN with Linearly Interpolated Value with `.interpolate()`**
-#
-#     Filling NaNs with the interpolated values is the best option in this small examlple but it requires knowledge of the neighouring values.
-#
-
-# %% _kg_hide-input=true papermill={"duration": 3.1715, "end_time": "2021-01-17T15:24:51.474321", "exception": false, "start_time": "2021-01-17T15:24:48.302821", "status": "completed"} tags=[]
-f, ax = plt.subplots(nrows=4, ncols=1, figsize=(15, 12))
-
-sns.lineplot(x=df.Date,
-             y=df.Drainage_Volume.fillna(0),
-             ax=ax[0],
-             color='darkorange',
-             label='modified')
-sns.lineplot(x=df.Date,
-             y=df.Drainage_Volume.fillna(np.inf),
-             ax=ax[0],
-             color='dodgerblue',
-             label='original')
-ax[0].set_title('Fill NaN with 0', fontsize=14)
-ax[0].set_ylabel(ylabel='Volume C10 Petrignano', fontsize=14)
-
-mean_val = df.Drainage_Volume.mean()
-sns.lineplot(x=df.Date,
-             y=df.Drainage_Volume.fillna(mean_val),
-             ax=ax[1],
-             color='darkorange',
-             label='modified')
-sns.lineplot(x=df.Date,
-             y=df.Drainage_Volume.fillna(np.inf),
-             ax=ax[1],
-             color='dodgerblue',
-             label='original')
-ax[1].set_title(f'Fill NaN with Mean Value ({mean_val:.0f})', fontsize=14)
-ax[1].set_ylabel(ylabel='Volume C10 Petrignano', fontsize=14)
-
-sns.lineplot(x=df.Date,
-             y=df.Drainage_Volume.ffill(),
-             ax=ax[2],
-             color='darkorange',
-             label='modified')
-sns.lineplot(x=df.Date,
-             y=df.Drainage_Volume.fillna(np.inf),
-             ax=ax[2],
-             color='dodgerblue',
-             label='original')
-ax[2].set_title(f'FFill', fontsize=14)
-ax[2].set_ylabel(ylabel='Volume C10 Petrignano', fontsize=14)
-
-sns.lineplot(x=df.Date,
-             y=df.Drainage_Volume.interpolate(),
-             ax=ax[3],
-             color='darkorange',
-             label='modified')
-sns.lineplot(x=df.Date,
-             y=df.Drainage_Volume.fillna(np.inf),
-             ax=ax[3],
-             color='dodgerblue',
-             label='original')
-ax[3].set_title(f'Interpolate', fontsize=14)
-ax[3].set_ylabel(ylabel='Volume C10 Petrignano', fontsize=14)
-
-for i in range(4):
-    ax[i].set_xlim([date(2019, 5, 1), date(2019, 10, 1)])
-plt.tight_layout()
-plt.show()
-
-# %% papermill={"duration": 0.076282, "end_time": "2021-01-17T15:24:51.615549", "exception": false, "start_time": "2021-01-17T15:24:51.539267", "status": "completed"} tags=[]
-df['Drainage_Volume'] = df['Drainage_Volume'].interpolate()
-df['River_Hydrometry'] = df['River_Hydrometry'].interpolate()
-df['Depth_to_Groundwater'] = df['Depth_to_Groundwater'].interpolate()
-
-# %% [markdown] papermill={"duration": 0.063411, "end_time": "2021-01-17T15:24:51.744017", "exception": false, "start_time": "2021-01-17T15:24:51.680606", "status": "completed"} tags=[]
-# ## Resampling
-#
-# Resampling can provide additional information on the data. There are two types of resampling:
-# * **Upsampling** is when the frequency of samples is increased (e.g. days to hours)
-# * **Downsampling** is when the frequency of samples is decreased (e.g. days to weeks)
-#
-# In this example, we will do some downsampling with the `.resample()` function.
-
-# %% _kg_hide-input=true papermill={"duration": 11.3081, "end_time": "2021-01-17T15:25:03.116975", "exception": false, "start_time": "2021-01-17T15:24:51.808875", "status": "completed"} tags=[]
-fig, ax = plt.subplots(ncols=2, nrows=4, sharex=True, figsize=(16, 12))
-
-ax[0, 0].bar(df.Date, df.Rainfall, width=5, color='dodgerblue')
-ax[0, 0].set_title('Daily Rainfall (Acc.)', fontsize=14)
-
-resampled_df = df[['Date', 'Rainfall'
-                   ]].resample('7D', on='Date').sum().reset_index(drop=False)
-ax[1, 0].bar(resampled_df.Date,
-             resampled_df.Rainfall,
-             width=10,
-             color='dodgerblue')
-ax[1, 0].set_title('Weekly Rainfall (Acc.)', fontsize=14)
-
-resampled_df = df[['Date', 'Rainfall'
-                   ]].resample('M', on='Date').sum().reset_index(drop=False)
-ax[2, 0].bar(resampled_df.Date,
-             resampled_df.Rainfall,
-             width=15,
-             color='dodgerblue')
-ax[2, 0].set_title('Monthly Rainfall (Acc.)', fontsize=14)
-
-resampled_df = df[['Date', 'Rainfall'
-                   ]].resample('12M', on='Date').sum().reset_index(drop=False)
-ax[3, 0].bar(resampled_df.Date,
-             resampled_df.Rainfall,
-             width=20,
-             color='dodgerblue')
-ax[3, 0].set_title('Annual Rainfall (Acc.)', fontsize=14)
-
-for i in range(4):
-    ax[i, 0].set_xlim([date(2009, 1, 1), date(2020, 6, 30)])
-
-sns.lineplot(df.Date, df.Temperature, color='dodgerblue', ax=ax[0, 1])
-ax[0, 1].set_title('Daily Temperature (Acc.)', fontsize=14)
-
-resampled_df = df[['Date', 'Temperature'
-                   ]].resample('7D', on='Date').mean().reset_index(drop=False)
-sns.lineplot(resampled_df.Date,
-             resampled_df.Temperature,
-             color='dodgerblue',
-             ax=ax[1, 1])
-ax[1, 1].set_title('Weekly Temperature (Acc.)', fontsize=14)
-
-resampled_df = df[['Date', 'Temperature'
-                   ]].resample('M', on='Date').mean().reset_index(drop=False)
-sns.lineplot(resampled_df.Date,
-             resampled_df.Temperature,
-             color='dodgerblue',
-             ax=ax[2, 1])
-ax[2, 1].set_title('Monthly Temperature (Acc.)', fontsize=14)
-
-resampled_df = df[['Date', 'Temperature'
-                   ]].resample('365D',
-                               on='Date').mean().reset_index(drop=False)
-sns.lineplot(resampled_df.Date,
-             resampled_df.Temperature,
-             color='dodgerblue',
-             ax=ax[3, 1])
-ax[3, 1].set_title('Annual Temperature (Acc.)', fontsize=14)
-
-for i in range(4):
-    ax[i, 1].set_xlim([date(2009, 1, 1), date(2020, 6, 30)])
-    ax[i, 1].set_ylim([-5, 35])
-plt.show()
-
-# %% [markdown] papermill={"duration": 0.066262, "end_time": "2021-01-17T15:25:03.251035", "exception": false, "start_time": "2021-01-17T15:25:03.184773", "status": "completed"} tags=[]
-# In this example, resampling would not be necessary. On the other hand, there is no necessity to look at the daily data. Considering weekly data seems to be sufficient as well. Therefore, we will **downsample the data to a weekly basis**.
-
-# %% papermill={"duration": 0.090569, "end_time": "2021-01-17T15:25:03.408281", "exception": false, "start_time": "2021-01-17T15:25:03.317712", "status": "completed"} tags=[]
-df_downsampled = df[[
-    'Date', 'Depth_to_Groundwater', 'Temperature', 'Drainage_Volume',
-    'River_Hydrometry'
-]].resample('7D', on='Date').mean().reset_index(drop=False)
-
-df_downsampled['Rainfall'] = df[['Date', 'Rainfall']].resample(
-    '7D', on='Date').sum().reset_index(drop=False)[['Rainfall']]
-
-df = df_downsampled
-
-# %% [markdown] papermill={"duration": 0.06652, "end_time": "2021-01-17T15:25:03.542412", "exception": false, "start_time": "2021-01-17T15:25:03.475892", "status": "completed"} tags=[]
-# ## Stationarity
-#
-# Some time-series models, such as such as [ARIMA](#ARIMA), assume that the underlying data is stationary.
-# Stationarity describes that the time-series has
-# * constant mean and mean is not time-dependent
-# * constant variance and variance is not time-dependent
-# * constant covariance and covariance is not time-dependent
-#
-# > If a time series has a specific (stationary) behavior over a given time interval, then it can be assumed that the time series will behave the same at a later time.
-#
-# Time series **with trend and/or seasonality are not stationary**. Trend indicates that the mean is not constant over time and seasonality indicates that the variance is not constant over time.
-
-# %% _kg_hide-input=true papermill={"duration": 0.865994, "end_time": "2021-01-17T15:25:04.474960", "exception": false, "start_time": "2021-01-17T15:25:03.608966", "status": "completed"} tags=[]
-t = np.linspace(0, 19, 20)
-
-fig, ax = plt.subplots(ncols=4, nrows=1, figsize=(20, 4))
-stationary = [
-    5,
-    4,
-    5,
-    6,
-    5,
-    4,
-    5,
-    6,
-    5,
-    4,
-    5,
-    6,
-    5,
-    4,
-    5,
-    6,
-    5,
-    4,
-    5,
-    6,
-]
-sns.lineplot(x=t, y=stationary, ax=ax[0], color='forestgreen')
-sns.lineplot(x=t, y=5, ax=ax[0], color='grey')
-sns.lineplot(x=t, y=6, ax=ax[0], color='grey')
-sns.lineplot(x=t, y=4, ax=ax[0], color='grey')
-ax[0].lines[2].set_linestyle("--")
-ax[0].lines[3].set_linestyle("--")
-ax[0].set_title(
-    f'Stationary \nconstant mean \nconstant variance \nconstant covariance',
-    fontsize=14)
-
-nonstationary1 = [9, 0, 1, 10, 8, 1, 2, 9, 7, 2, 3, 8, 6, 3, 4, 7, 5, 4, 5, 6]
-sns.lineplot(x=t, y=nonstationary1, ax=ax[1], color='indianred')
-sns.lineplot(x=t, y=5, ax=ax[1], color='grey')
-sns.lineplot(x=t, y=t * 0.25 - 0.5, ax=ax[1], color='grey')
-sns.lineplot(x=t, y=t * (-0.25) + 11, ax=ax[1], color='grey')
-ax[1].lines[2].set_linestyle("--")
-ax[1].lines[3].set_linestyle("--")
-ax[1].set_title(
-    f'Non Stationary \nconstant mean \n non-constant variance\nnconstant covariance',
-    fontsize=14)
-
-nonstationary2 = [
-    0,
-    2,
-    1,
-    3,
-    2,
-    4,
-    3,
-    5,
-    4,
-    6,
-    5,
-    7,
-    6,
-    8,
-    7,
-    9,
-    8,
-    10,
-    9,
-    11,
-]
-sns.lineplot(x=t, y=nonstationary2, ax=ax[2], color='indianred')
-sns.lineplot(x=t, y=t * 0.5 + 0.7, ax=ax[2], color='grey')
-sns.lineplot(x=t, y=t * 0.5, ax=ax[2], color='grey')
-sns.lineplot(x=t, y=t * 0.5 + 1.5, ax=ax[2], color='grey')
-ax[2].lines[2].set_linestyle("--")
-ax[2].lines[3].set_linestyle("--")
-ax[2].set_title(
-    f'Non Stationary \n non-constant mean\nconstant variance\nnconstant covariance',
-    fontsize=14)
-
-nonstationary3 = [
-    5,
-    4.5,
-    4,
-    4.5,
-    5,
-    5.5,
-    6,
-    5.5,
-    5,
-    4.5,
-    4,
-    5,
-    6,
-    5,
-    4,
-    6,
-    4,
-    6,
-    4,
-    6,
-]
-sns.lineplot(x=t, y=nonstationary3, ax=ax[3], color='indianred')
-sns.lineplot(x=t, y=5, ax=ax[3], color='grey')
-sns.lineplot(x=t, y=6, ax=ax[3], color='grey')
-sns.lineplot(x=t, y=4, ax=ax[3], color='grey')
-ax[3].lines[2].set_linestyle("--")
-ax[3].lines[3].set_linestyle("--")
-ax[3].set_title(
-    f'Stationary \nconstant mean \nconstant variance \nnon-constant covariance',
-    fontsize=14)
-
-for i in range(4):
-    ax[i].set_ylim([-1, 12])
-    ax[i].set_xlabel('Time', fontsize=14)
-
-# %% [markdown] papermill={"duration": 0.069159, "end_time": "2021-01-17T15:25:04.613281", "exception": false, "start_time": "2021-01-17T15:25:04.544122", "status": "completed"} tags=[]
-# The check for stationarity can be done via three different approaches:
-# 1. **visually**: plot time series and check for trends or seasonality
-# 2. **basic statistics**: split time series and compare the mean and variance of each partition
-# 3. **statistical test**: Augmented Dickey Fuller test
-#
-# Let's do the **visual check** first. We can see that all features except `Temperature` have non-constant mean and non-constant variance. Therefore, **none of these seem to be stationary**. However, `Temperature` shows strong seasonality (hot in summer, cold in winter) and therefore it is not stationary either.
-
-# %% _kg_hide-input=true papermill={"duration": 2.012837, "end_time": "2021-01-17T15:25:06.695661", "exception": false, "start_time": "2021-01-17T15:25:04.682824", "status": "completed"} tags=[]
-rolling_window = 52
-f, ax = plt.subplots(nrows=3, ncols=2, figsize=(15, 12))
-
-sns.lineplot(x=df.Date, y=df.Rainfall, ax=ax[0, 0], color='indianred')
-sns.lineplot(x=df.Date,
-             y=df.Rainfall.rolling(rolling_window).mean(),
-             ax=ax[0, 0],
-             color='black',
-             label='rolling mean')
-sns.lineplot(x=df.Date,
-             y=df.Rainfall.rolling(rolling_window).std(),
-             ax=ax[0, 0],
-             color='blue',
-             label='rolling std')
-ax[0, 0].set_title(
-    'Rainfall: Non-stationary \nnon-constant mean & non-constant variance',
-    fontsize=14)
-ax[0, 0].set_ylabel(ylabel='Rainfall', fontsize=14)
-
-sns.lineplot(x=df.Date, y=df.Temperature, ax=ax[1, 0], color='indianred')
-sns.lineplot(x=df.Date,
-             y=df.Temperature.rolling(rolling_window).mean(),
-             ax=ax[1, 0],
-             color='black',
-             label='rolling mean')
-sns.lineplot(x=df.Date,
-             y=df.Temperature.rolling(rolling_window).std(),
-             ax=ax[1, 0],
-             color='blue',
-             label='rolling std')
-ax[1, 0].set_title(
-    'Temperature: Non-stationary \nvariance is time-dependent (seasonality)',
-    fontsize=14)
-ax[1, 0].set_ylabel(ylabel='Temperature', fontsize=14)
-
-sns.lineplot(x=df.Date, y=df.River_Hydrometry, ax=ax[0, 1], color='indianred')
-sns.lineplot(x=df.Date,
-             y=df.River_Hydrometry.rolling(rolling_window).mean(),
-             ax=ax[0, 1],
-             color='black',
-             label='rolling mean')
-sns.lineplot(x=df.Date,
-             y=df.River_Hydrometry.rolling(rolling_window).std(),
-             ax=ax[0, 1],
-             color='blue',
-             label='rolling std')
-ax[0, 1].set_title(
-    'Hydrometry: Non-stationary \nnon-constant mean & non-constant variance',
-    fontsize=14)
-ax[0, 1].set_ylabel(ylabel='Hydrometry', fontsize=14)
-
-sns.lineplot(x=df.Date, y=df.Drainage_Volume, ax=ax[1, 1], color='indianred')
-sns.lineplot(x=df.Date,
-             y=df.Drainage_Volume.rolling(rolling_window).mean(),
-             ax=ax[1, 1],
-             color='black',
-             label='rolling mean')
-sns.lineplot(x=df.Date,
-             y=df.Drainage_Volume.rolling(rolling_window).std(),
-             ax=ax[1, 1],
-             color='blue',
-             label='rolling std')
-ax[1, 1].set_title(
-    'Volume: Non-stationary \nnon-constant mean & non-constant variance',
-    fontsize=14)
-ax[1, 1].set_ylabel(ylabel='Volume', fontsize=14)
-
-sns.lineplot(x=df.Date,
-             y=df.Depth_to_Groundwater,
-             ax=ax[2, 0],
-             color='indianred')
-sns.lineplot(x=df.Date,
-             y=df.Depth_to_Groundwater.rolling(rolling_window).mean(),
-             ax=ax[2, 0],
-             color='black',
-             label='rolling mean')
-sns.lineplot(x=df.Date,
-             y=df.Depth_to_Groundwater.rolling(rolling_window).std(),
-             ax=ax[2, 0],
-             color='blue',
-             label='rolling std')
-ax[2, 0].set_title(
-    'Depth to Groundwater: Non-stationary \nnon-constant mean & non-constant variance',
-    fontsize=14)
-ax[2, 0].set_ylabel(ylabel='Depth to Groundwater', fontsize=14)
-
-for i in range(3):
-    ax[i, 0].set_xlim([date(2009, 1, 1), date(2020, 6, 30)])
-    ax[i, 1].set_xlim([date(2009, 1, 1), date(2020, 6, 30)])
-
-f.delaxes(ax[2, 1])
-plt.tight_layout()
-plt.show()
-
-# %% [markdown] papermill={"duration": 0.076945, "end_time": "2021-01-17T15:25:06.850471", "exception": false, "start_time": "2021-01-17T15:25:06.773526", "status": "completed"} tags=[]
-# Next, we will **check the underlying statistics**. For this we will **split the time series into two sections** and check the mean and the variance. You could do more partitions if you wanted.
-#
-# With this method, `Temperature` and `River_Hydrometry` show **somewhat similar (constant) mean and variance** and could be seen as stationary. However, with this method, we are not able to see the seasonality in the `Temperature` feature.
-
-# %% papermill={"duration": 0.095371, "end_time": "2021-01-17T15:25:07.023184", "exception": false, "start_time": "2021-01-17T15:25:06.927813", "status": "completed"} tags=[]
-num_partitions = 2
-partition_length = int(len(df) / num_partitions)
-
-partition1_mean = df.head(partition_length).mean()
-partition1_var = df.head(partition_length).var()
-partition2_mean = df.tail(partition_length).mean()
-partition2_var = df.tail(partition_length).var()
-
-# %% _kg_hide-input=true papermill={"duration": 0.119633, "end_time": "2021-01-17T15:25:07.217921", "exception": false, "start_time": "2021-01-17T15:25:07.098288", "status": "completed"} tags=[]
-stationarity_test = pd.concat(
-    [partition1_mean, partition2_mean, partition1_var, partition2_var], axis=1)
-stationarity_test.columns = [
-    'Partition 1 Mean', 'Partition 2 Mean', 'Partition 1 Variance',
-    'Partition 2 Variance'
-]
-
-
-def highlight_greater(x):
-    temp = x.copy()
-    temp = temp.round(0).astype(int)
-    m1 = (temp['Partition 1 Mean'] == temp['Partition 2 Mean'])
-    m2 = (temp['Partition 1 Variance'] == temp['Partition 2 Variance'])
-    m3 = (temp['Partition 1 Mean'] < temp['Partition 2 Mean'] + 3) & (
-        temp['Partition 1 Mean'] > temp['Partition 2 Mean'] - 3)
-    m4 = (temp['Partition 1 Variance'] < temp['Partition 2 Variance'] + 3) & (
-        temp['Partition 1 Variance'] > temp['Partition 2 Variance'] - 3)
-
-    df1 = pd.DataFrame('background-color: ', index=x.index, columns=x.columns)
-    #rewrite values by boolean masks
-    df1['Partition 1 Mean'] = np.where(~m1,
-                                       'background-color: {}'.format('salmon'),
-                                       df1['Partition 1 Mean'])
-    df1['Partition 2 Mean'] = np.where(~m1,
-                                       'background-color: {}'.format('salmon'),
-                                       df1['Partition 2 Mean'])
-    df1['Partition 1 Mean'] = np.where(m3,
-                                       'background-color: {}'.format('gold'),
-                                       df1['Partition 1 Mean'])
-    df1['Partition 2 Mean'] = np.where(m3,
-                                       'background-color: {}'.format('gold'),
-                                       df1['Partition 2 Mean'])
-    df1['Partition 1 Mean'] = np.where(
-        m1, 'background-color: {}'.format('mediumseagreen'),
-        df1['Partition 1 Mean'])
-    df1['Partition 2 Mean'] = np.where(
-        m1, 'background-color: {}'.format('mediumseagreen'),
-        df1['Partition 2 Mean'])
-
-    df1['Partition 1 Variance'] = np.where(
-        ~m2, 'background-color: {}'.format('salmon'),
-        df1['Partition 1 Variance'])
-    df1['Partition 2 Variance'] = np.where(
-        ~m2, 'background-color: {}'.format('salmon'),
-        df1['Partition 2 Variance'])
-    df1['Partition 1 Variance'] = np.where(
-        m4, 'background-color: {}'.format('gold'), df1['Partition 1 Variance'])
-    df1['Partition 2 Variance'] = np.where(
-        m4, 'background-color: {}'.format('gold'), df1['Partition 2 Variance'])
-    df1['Partition 1 Variance'] = np.where(
-        m2, 'background-color: {}'.format('mediumseagreen'),
-        df1['Partition 1 Variance'])
-    df1['Partition 2 Variance'] = np.where(
-        m2, 'background-color: {}'.format('mediumseagreen'),
-        df1['Partition 2 Variance'])
-
-    return df1
-
-
-stationarity_test.style.apply(highlight_greater, axis=None).format("{:20,.0f}")
-
-# %% [markdown] papermill={"duration": 0.07589, "end_time": "2021-01-17T15:25:07.369891", "exception": false, "start_time": "2021-01-17T15:25:07.294001", "status": "completed"} tags=[]
-# Let's evaluate the histograms. Since we are looking at the mean and variance, we are expecting that the data conforms to a Gaussian distribution (bell shaped distribution) in case of stationarity.
-
-# %% _kg_hide-input=true papermill={"duration": 1.738263, "end_time": "2021-01-17T15:25:09.184244", "exception": false, "start_time": "2021-01-17T15:25:07.445981", "status": "completed"} tags=[]
-f, ax = plt.subplots(nrows=3, ncols=2, figsize=(15, 9))
-
-sns.distplot(df.Rainfall.fillna(np.inf), ax=ax[0, 0], color='indianred')
-ax[0, 0].set_title(
-    'Rainfall: Non-stationary \nnon-constant mean & non-constant variance',
-    fontsize=14)
-ax[0, 0].set_ylabel(ylabel='Rainfall', fontsize=14)
-
-sns.distplot(df.Temperature.fillna(np.inf), ax=ax[1, 0], color='indianred')
-ax[1, 0].set_title(
-    'Temperature: Non-stationary \nvariance is time-dependent (seasonality)',
-    fontsize=14)
-ax[1, 0].set_ylabel(ylabel='Temperature', fontsize=14)
-
-sns.distplot(df.River_Hydrometry.fillna(np.inf),
-             ax=ax[0, 1],
-             color='indianred')
-ax[0, 1].set_title(
-    'Hydrometry: Non-stationary \nnon-constant mean & non-constant variance',
-    fontsize=14)
-ax[0, 1].set_ylabel(ylabel='Hydrometry', fontsize=14)
-
-sns.distplot(df.Drainage_Volume.fillna(np.inf), ax=ax[1, 1], color='indianred')
-ax[1, 1].set_title(
-    'Volume: Non-stationary \nnon-constant mean & non-constant variance',
-    fontsize=14)
-ax[1, 1].set_ylabel(ylabel='Volume', fontsize=14)
-
-sns.distplot(df.Depth_to_Groundwater.fillna(np.inf),
-             ax=ax[2, 0],
-             color='indianred')
-ax[2, 0].set_title(
-    'Depth to Groundwater: Non-stationary \nnon-constant mean & non-constant variance',
-    fontsize=14)
-ax[2, 0].set_ylabel(ylabel='Depth to Groundwater', fontsize=14)
-
-f.delaxes(ax[2, 1])
-plt.tight_layout()
-plt.show()
-
-# %% [markdown] papermill={"duration": 0.079289, "end_time": "2021-01-17T15:25:09.342433", "exception": false, "start_time": "2021-01-17T15:25:09.263144", "status": "completed"} tags=[]
-# **Augmented Dickey-Fuller (ADF) test**  is a type of statistical test called a unit root test.  Unit roots are a cause for non-stationarity.
-#
-# * **Null Hypothesis (H0)**: Time series has a unit root. (Time series is **not stationary**).
-#
-# * **Alternate Hypothesis (H1)**: Time series has no unit root (Time series is **stationary**).
-#
-# If the **null hypothesis can be rejected**, we can conclude that the **time series is stationary**.
-#
-# There are two ways to rejects the null hypothesis:
-#
-# On the one hand, the null hypothesis can be rejected if the p-value is below a set significance level. The defaults significance level is 5%
-#
-# * <font color='red'>**p-value > significance level (default: 0.05)**</font>: Fail to reject the null hypothesis (H0), the data has a unit root and is <font color='red'>non-stationary</font>.
-# * <font color='green'>**p-value <= significance level (default: 0.05)**</font>: Reject the null hypothesis (H0), the data does not have a unit root and is <font color='green'>stationary</font>.
-#
-# On the other hand, the null hypothesis can be rejects if the test statistic is less than the critical value.
-# * <font color='red'>**ADF statistic > critical value**</font>: Fail to reject the null hypothesis (H0), the data has a unit root and is <font color='red'>non-stationary</font>.
-# * <font color='green'>**ADF statistic < critical value**</font>: Reject the null hypothesis (H0), the data does not have a unit root and is <font color='green'>stationary</font>.
-
-# %% papermill={"duration": 0.365045, "end_time": "2021-01-17T15:25:09.787168", "exception": false, "start_time": "2021-01-17T15:25:09.422123", "status": "completed"} tags=[]
-from statsmodels.tsa.stattools import adfuller
-
-result = adfuller(df.Depth_to_Groundwater.values)
-adf_stat = result[0]
-p_val = result[1]
-crit_val_1 = result[4]['1%']
-crit_val_5 = result[4]['5%']
-crit_val_10 = result[4]['10%']
-
-# %% _kg_hide-input=true papermill={"duration": 1.223719, "end_time": "2021-01-17T15:25:11.091679", "exception": false, "start_time": "2021-01-17T15:25:09.867960", "status": "completed"} tags=[]
-
-f, ax = plt.subplots(nrows=3, ncols=2, figsize=(15, 9))
-
-
-def visualize_adfuller_results(series, title, ax, df):
-    result = adfuller(series)
-    significance_level = 0.05
-    adf_stat = result[0]
-    p_val = result[1]
-    crit_val_1 = result[4]['1%']
-    crit_val_5 = result[4]['5%']
-    crit_val_10 = result[4]['10%']
-
-    if (p_val < significance_level) & ((adf_stat < crit_val_1)):
-        linecolor = 'forestgreen'
-    elif (p_val < significance_level) & (adf_stat < crit_val_5):
-        linecolor = 'gold'
-    elif (p_val < significance_level) & (adf_stat < crit_val_10):
-        linecolor = 'orange'
-    else:
-        linecolor = 'indianred'
-    sns.lineplot(x=df.Date, y=series, ax=ax, color=linecolor)
-    ax.set_title(
-        f'ADF Statistic {adf_stat:0.3f}, p-value: {p_val:0.3f}\nCritical Values 1%: {crit_val_1:0.3f}, 5%: {crit_val_5:0.3f}, 10%: {crit_val_10:0.3f}',
-        fontsize=14)
-    ax.set_ylabel(ylabel=title, fontsize=14)
-
-
-visualize_adfuller_results(df.Rainfall.values, 'Rainfall', ax[0, 0], df)
-visualize_adfuller_results(df.Temperature.values, 'Temperature', ax[1, 0], df)
-visualize_adfuller_results(df.River_Hydrometry.values, 'River_Hydrometry',
-                           ax[0, 1], df)
-visualize_adfuller_results(df.Drainage_Volume.values, 'Drainage_Volume',
-                           ax[1, 1], df)
-visualize_adfuller_results(df.Depth_to_Groundwater.values,
-                           'Depth_to_Groundwater', ax[2, 0], df)
-
-f.delaxes(ax[2, 1])
-plt.tight_layout()
-plt.show()
-
-# %% [markdown] papermill={"duration": 0.085084, "end_time": "2021-01-17T15:25:11.261673", "exception": false, "start_time": "2021-01-17T15:25:11.176589", "status": "completed"} tags=[]
-# **TODO: How to interpret the contradictory results from the different checks...**
-
-# %% [markdown] papermill={"duration": 0.084347, "end_time": "2021-01-17T15:25:11.430525", "exception": false, "start_time": "2021-01-17T15:25:11.346178", "status": "completed"} tags=[]
-# If the data is not stationary but we want to use a model that requires with characteristic, the data has to be transformed. However, if the data is not stationary to begin with, we should rethink the choice of model.
-#
-# The two most common methods to achieve stationarity are:
-# * **Transformation**: e.g. log or square root to stabilize non-constant variance
-# * **Differencing**: subtracts the current value from the previous
-#
-
-# %% papermill={"duration": 0.093741, "end_time": "2021-01-17T15:25:11.608525", "exception": false, "start_time": "2021-01-17T15:25:11.514784", "status": "completed"} tags=[]
-# Log Transform of absolute values
-# (Log transoform of negative values will return NaN)
-df['Depth_to_Groundwater_log'] = np.log(abs(df.Depth_to_Groundwater))
-
-# %% _kg_hide-input=true papermill={"duration": 0.910914, "end_time": "2021-01-17T15:25:12.602248", "exception": false, "start_time": "2021-01-17T15:25:11.691334", "status": "completed"} tags=[]
-f, ax = plt.subplots(nrows=2, ncols=2, figsize=(15, 6))
-visualize_adfuller_results(abs(df.Depth_to_Groundwater),
-                           'Absolute \n Depth to Groundwater', ax[0, 0], df)
-
-sns.distplot(df.Depth_to_Groundwater_log, ax=ax[0, 1])
-visualize_adfuller_results(df.Depth_to_Groundwater_log,
-                           'Transformed \n Depth to Groundwater', ax[1, 0], df)
-
-sns.distplot(df.Depth_to_Groundwater_log, ax=ax[1, 1])
-
-plt.tight_layout()
-plt.show()
-
-# %% [markdown] papermill={"duration": 0.085014, "end_time": "2021-01-17T15:25:12.772829", "exception": false, "start_time": "2021-01-17T15:25:12.687815", "status": "completed"} tags=[]
-# Differencing can be done in different orders:
-# * First order differencing: linear trends with $z_i = y_i - y_{i-1}$
-# * Second-order differencing: quadratic trends with $z_i = (y_i - y_{i-1}) - (y_{i-1} - y_{i-2})$
-# * and so on...
-
-# %% papermill={"duration": 0.097265, "end_time": "2021-01-17T15:25:12.955144", "exception": false, "start_time": "2021-01-17T15:25:12.857879", "status": "completed"} tags=[]
-# First Order Differencing
-ts_diff = np.diff(df.Depth_to_Groundwater)
-df['Depth_to_Groundwater_diff_1'] = np.append([0], ts_diff)
-
-# Second Order Differencing
-ts_diff = np.diff(df.Depth_to_Groundwater_diff_1)
-df['Depth_to_Groundwater_diff_2'] = np.append([0], ts_diff)
-
-# %% _kg_hide-input=true papermill={"duration": 0.565564, "end_time": "2021-01-17T15:25:13.607429", "exception": false, "start_time": "2021-01-17T15:25:13.041865", "status": "completed"} tags=[]
-f, ax = plt.subplots(nrows=2, ncols=1, figsize=(15, 6))
-
-visualize_adfuller_results(df.Depth_to_Groundwater_diff_1,
-                           'Differenced (1. Order) \n Depth to Groundwater',
-                           ax[0], df)
-visualize_adfuller_results(df.Depth_to_Groundwater_diff_2,
-                           'Differenced (2. Order) \n Depth to Groundwater',
-                           ax[1], df)
-plt.tight_layout()
-plt.show()
-
-# %% [markdown] papermill={"duration": 0.088768, "end_time": "2021-01-17T15:25:13.786342", "exception": false, "start_time": "2021-01-17T15:25:13.697574", "status": "completed"} tags=[]
-# The differencing can be reverted if the the first value before differencing is known. In this case, we can accumulate all values with the function `.cumsum()` and add the first value of the original time series.
-
-# %% papermill={"duration": 0.099347, "end_time": "2021-01-17T15:25:13.974278", "exception": false, "start_time": "2021-01-17T15:25:13.874931", "status": "completed"} tags=[]
-df.Depth_to_Groundwater.equals(df.Depth_to_Groundwater_diff_1.cumsum() +
-                               df.Depth_to_Groundwater.iloc[0])
-
-# %% [markdown] papermill={"duration": 0.089467, "end_time": "2021-01-17T15:25:14.155862", "exception": false, "start_time": "2021-01-17T15:25:14.066395", "status": "completed"} tags=[]
-# # Feature Engineering
-#
-# ## Time Features
-
-# %% papermill={"duration": 0.123481, "end_time": "2021-01-17T15:25:14.370092", "exception": false, "start_time": "2021-01-17T15:25:14.246611", "status": "completed"} tags=[]
-df['year'] = pd.DatetimeIndex(df['Date']).year
-df['month'] = pd.DatetimeIndex(df['Date']).month
-df['day'] = pd.DatetimeIndex(df['Date']).day
-df['day_of_year'] = pd.DatetimeIndex(df['Date']).dayofyear
-df['week_of_year'] = pd.DatetimeIndex(df['Date']).weekofyear
-df['quarter'] = pd.DatetimeIndex(df['Date']).quarter
-df['season'] = df.month % 12 // 3 + 1
-
-df[[
-    'Date', 'year', 'month', 'day', 'day_of_year', 'week_of_year', 'quarter',
-    'season'
-]].head()
-
-# %% [markdown] papermill={"duration": 0.091852, "end_time": "2021-01-17T15:25:14.557349", "exception": false, "start_time": "2021-01-17T15:25:14.465497", "status": "completed"} tags=[]
-# ## Encoding Cyclical Features
-# The new time features are cyclical. For example,the feature `month` cycles between 1 and 12 for every year.
-# While the difference between each month increments by 1 during the year, between two years the `month` feature jumps from 12 (December) to 1 (January). This results in a -11 difference, which can confuse a lot of models.
-
-# %% _kg_hide-input=true papermill={"duration": 0.301777, "end_time": "2021-01-17T15:25:14.949940", "exception": false, "start_time": "2021-01-17T15:25:14.648163", "status": "completed"} tags=[]
-f, ax = plt.subplots(nrows=1, ncols=1, figsize=(20, 3))
-
-sns.lineplot(x=df.Date, y=df.month, color='dodgerblue')
-ax.set_xlim([date(2009, 1, 1), date(2020, 6, 30)])
-plt.show()
-
-# %% [markdown] papermill={"duration": 0.092283, "end_time": "2021-01-17T15:25:15.134903", "exception": false, "start_time": "2021-01-17T15:25:15.042620", "status": "completed"} tags=[]
-# Ideally, we want the underlying data to represent the same difference between two consecutive months, even between December and January. A common remedy for this issue is to encode cyclical features into two dimensions with sine and cosine transformation.
-
-# %% _kg_hide-input=true _kg_hide-output=true papermill={"duration": 0.104048, "end_time": "2021-01-17T15:25:15.331591", "exception": false, "start_time": "2021-01-17T15:25:15.227543", "status": "completed"} tags=[]
-month_in_year = 12
-df['month_sin'] = np.sin(2 * np.pi * df.month / month_in_year)
-df['month_cos'] = np.cos(2 * np.pi * df.month / month_in_year)
-
-# %% _kg_hide-input=true papermill={"duration": 0.295424, "end_time": "2021-01-17T15:25:15.718934", "exception": false, "start_time": "2021-01-17T15:25:15.423510", "status": "completed"} tags=[]
-days_in_month = 30
-df['day_sin'] = np.sin(2 * np.pi * df.day / days_in_month)
-df['day_cos'] = np.cos(2 * np.pi * df.day / days_in_month)
-
-days_in_year = 365
-df['day_of_year_sin'] = np.sin(2 * np.pi * df.day_of_year / days_in_year)
-df['day_of_year_cos'] = np.cos(2 * np.pi * df.day_of_year / days_in_year)
-
-weeks_in_year = 52.1429
-df['week_of_year_sin'] = np.sin(2 * np.pi * df.week_of_year / weeks_in_year)
-df['week_of_year_cos'] = np.cos(2 * np.pi * df.week_of_year / weeks_in_year)
-
-quarters_in_year = 4
-df['quarter_sin'] = np.sin(2 * np.pi * df.quarter / quarters_in_year)
-df['quarter_cos'] = np.cos(2 * np.pi * df.quarter / quarters_in_year)
-
-seasons_in_year = 4
-df['season_sin'] = np.sin(2 * np.pi * df.season / seasons_in_year)
-df['season_cos'] = np.cos(2 * np.pi * df.season / seasons_in_year)
-
-f, ax = plt.subplots(nrows=1, ncols=1, figsize=(6, 6))
-
-sns.scatterplot(x=df.month_sin, y=df.month_cos, color='dodgerblue')
-plt.show()
-
-# %% [markdown] papermill={"duration": 0.09423, "end_time": "2021-01-17T15:25:15.906123", "exception": false, "start_time": "2021-01-17T15:25:15.811893", "status": "completed"} tags=[]
-# ## Decomposition
-#
-# The **characteristics of a time series** are
-# * Trend and Level
-# * Seasonality
-# * Random / Noise
-#
-# We can use the function `seasonal_decompose()` from the [statsmodels](https://www.statsmodels.org) library.
-#
-# * Additive: $y(t) = Level + Trend + Seasonality + Noise$
-# * Multiplicative: $y(t) = Level * Trend * Seasonality * Noise$
-
-# %% papermill={"duration": 0.152167, "end_time": "2021-01-17T15:25:16.152057", "exception": false, "start_time": "2021-01-17T15:25:15.999890", "status": "completed"} tags=[]
-from statsmodels.tsa.seasonal import seasonal_decompose
-
-decompose_cols = [
-    'Rainfall', 'Temperature', 'Drainage_Volume', 'River_Hydrometry',
-    'Depth_to_Groundwater'
-]
-
-for col in decompose_cols:
-    decomp = seasonal_decompose(df[col],
-                                freq=52,
-                                model='additive',
-                                extrapolate_trend='freq')
-    df[f"{col}_trend"] = decomp.trend
-    df[f"{col}_seasonal"] = decomp.seasonal
-
-# %% _kg_hide-input=true papermill={"duration": 2.38781, "end_time": "2021-01-17T15:25:18.634264", "exception": false, "start_time": "2021-01-17T15:25:16.246454", "status": "completed"} tags=[]
-fig, ax = plt.subplots(ncols=2, nrows=4, sharex=True, figsize=(16, 8))
-res = seasonal_decompose(df.Temperature,
-                         freq=52,
-                         model='additive',
-                         extrapolate_trend='freq')
-
-ax[0, 0].set_title('Decomposition of Temperature', fontsize=16)
-res.observed.plot(ax=ax[0, 0], legend=False, color='dodgerblue')
-ax[0, 0].set_ylabel('Observed', fontsize=14)
-ax[0, 0].set_ylim([-5, 35])
-
-res.trend.plot(ax=ax[1, 0], legend=False, color='dodgerblue')
-ax[1, 0].set_ylabel('Trend', fontsize=14)
-ax[1, 0].set_ylim([-5, 35])
-
-res.seasonal.plot(ax=ax[2, 0], legend=False, color='dodgerblue')
-ax[2, 0].set_ylabel('Seasonal', fontsize=14)
-ax[2, 0].set_ylim([-15, 15])
-
-res.resid.plot(ax=ax[3, 0], legend=False, color='dodgerblue')
-ax[3, 0].set_ylabel('Residual', fontsize=14)
-ax[3, 0].set_ylim([-15, 15])
-
-ax[0, 1].set_title('Decomposition of Depth_to_Groundwater', fontsize=16)
-res = seasonal_decompose(df.Depth_to_Groundwater,
-                         freq=52,
-                         model='additive',
-                         extrapolate_trend='freq')
-
-res.observed.plot(ax=ax[0, 1], legend=False, color='dodgerblue')
-ax[0, 1].set_ylabel('Observed', fontsize=14)
-
-res.trend.plot(ax=ax[1, 1], legend=False, color='dodgerblue')
-ax[1, 1].set_ylabel('Trend', fontsize=14)
-
-res.seasonal.plot(ax=ax[2, 1], legend=False, color='dodgerblue')
-ax[2, 1].set_ylabel('Seasonal', fontsize=14)
-
-res.resid.plot(ax=ax[3, 1], legend=False, color='dodgerblue')
-ax[3, 1].set_ylabel('Residual', fontsize=14)
-
-plt.show()
-
-# %% _kg_hide-input=true papermill={"duration": 0.131376, "end_time": "2021-01-17T15:25:18.862798", "exception": false, "start_time": "2021-01-17T15:25:18.731422", "status": "completed"} tags=[]
-df[['Rainfall', 'Rainfall_trend', 'Rainfall_seasonal',
-          'Temperature', 'Temperature_trend', 'Temperature_seasonal',
-          'Drainage_Volume', 'Drainage_Volume_trend', 'Drainage_Volume_seasonal',
-          'River_Hydrometry', 'River_Hydrometry_trend', 'River_Hydrometry_seasonal',
-          'Depth_to_Groundwater', 'Depth_to_Groundwater_trend', 'Depth_to_Groundwater_seasonal']].head()\
-.style.set_properties(subset=['Rainfall_trend', 'Rainfall_seasonal',
-                              'Temperature_trend', 'Temperature_seasonal',
-                              'Drainage_Volume_trend', 'Drainage_Volume_seasonal',
-                              'River_Hydrometry_trend', 'River_Hydrometry_seasonal',
-                              'Depth_to_Groundwater_trend', 'Depth_to_Groundwater_seasonal'
-                             ], **{'background-color': 'dodgerblue'})
-
-# %% [markdown] papermill={"duration": 0.096207, "end_time": "2021-01-17T15:25:19.055201", "exception": false, "start_time": "2021-01-17T15:25:18.958994", "status": "completed"} tags=[]
-# ## Lag
-# `.shift()`
-#
-#
-
-# %% papermill={"duration": 0.112268, "end_time": "2021-01-17T15:25:19.263927", "exception": false, "start_time": "2021-01-17T15:25:19.151659", "status": "completed"} tags=[]
-weeks_in_month = 4
-
-df['Temperature_seasonal_shift_r_2M'] = df.Temperature_seasonal.shift(
-    -2 * weeks_in_month)
-df['Temperature_seasonal_shift_r_1M'] = df.Temperature_seasonal.shift(
-    -1 * weeks_in_month)
-df['Temperature_seasonal_shift_1M'] = df.Temperature_seasonal.shift(
-    1 * weeks_in_month)
-df['Temperature_seasonal_shift_2M'] = df.Temperature_seasonal.shift(
-    2 * weeks_in_month)
-df['Temperature_seasonal_shift_3M'] = df.Temperature_seasonal.shift(
-    3 * weeks_in_month)
-
-# %% _kg_hide-input=true papermill={"duration": 0.745036, "end_time": "2021-01-17T15:25:20.106199", "exception": false, "start_time": "2021-01-17T15:25:19.361163", "status": "completed"} tags=[]
-df['Drainage_Volume_seasonal_shift_r_2M'] = df.Drainage_Volume_seasonal.shift(
-    -2 * weeks_in_month)
-df['Drainage_Volume_seasonal_shift_r_1M'] = df.Drainage_Volume_seasonal.shift(
-    -1 * weeks_in_month)
-df['Drainage_Volume_seasonal_shift_1M'] = df.Drainage_Volume_seasonal.shift(
-    1 * weeks_in_month)
-df['Drainage_Volume_seasonal_shift_2M'] = df.Drainage_Volume_seasonal.shift(
-    2 * weeks_in_month)
-df['Drainage_Volume_seasonal_shift_3M'] = df.Drainage_Volume_seasonal.shift(
-    3 * weeks_in_month)
-
-df['River_Hydrometry_seasonal_shift_r_2M'] = df.River_Hydrometry_seasonal.shift(
-    -2 * weeks_in_month)
-df['River_Hydrometry_seasonal_shift_r_1M'] = df.River_Hydrometry_seasonal.shift(
-    -1 * weeks_in_month)
-df['River_Hydrometry_seasonal_shift_1M'] = df.River_Hydrometry_seasonal.shift(
-    1 * weeks_in_month)
-df['River_Hydrometry_seasonal_shift_2M'] = df.River_Hydrometry_seasonal.shift(
-    2 * weeks_in_month)
-df['River_Hydrometry_seasonal_shift_3M'] = df.River_Hydrometry_seasonal.shift(
-    3 * weeks_in_month)
-
-fig, ax = plt.subplots(ncols=1, nrows=1, figsize=(16, 4))
-sns.lineplot(x=df.Date,
-             y=df.Temperature_seasonal_shift_r_2M,
-             label='shifted by -2 month',
-             ax=ax,
-             color='lightblue')
-sns.lineplot(x=df.Date,
-             y=df.Temperature_seasonal_shift_r_1M,
-             label='shifted by -1 month',
-             ax=ax,
-             color='skyblue')
-
-sns.lineplot(x=df.Date,
-             y=df.Temperature_seasonal,
-             label='original',
-             ax=ax,
-             color='darkorange')
-
-sns.lineplot(x=df.Date,
-             y=df.Temperature_seasonal_shift_1M,
-             label='shifted by 1 month',
-             ax=ax,
-             color='dodgerblue')
-sns.lineplot(x=df.Date,
-             y=df.Temperature_seasonal_shift_2M,
-             label='shifted by 2 month',
-             ax=ax,
-             color='blue')
-sns.lineplot(x=df.Date,
-             y=df.Temperature_seasonal_shift_3M,
-             label='shifted by 3 month',
-             ax=ax,
-             color='navy')
-
-ax.set_title('Shifted Time Series', fontsize=16)
-
-ax.set_xlim([date(2017, 6, 30), date(2020, 6, 30)])
-ax.set_ylabel(ylabel='Temperature Bastia Umbra', fontsize=14)
-
-plt.show()
-
-# %% [markdown] papermill={"duration": 0.101968, "end_time": "2021-01-17T15:25:20.309732", "exception": false, "start_time": "2021-01-17T15:25:20.207764", "status": "completed"} tags=[]
-# # Exploratory Data Analysis
-#
-# Let's begin by plotting the seasonal components of each feature and comparing the minima and maxima. By doing this, we can already gain some insights:
-# * The depth to groundwater reaches its maximum around May/June and its minimum around November/December
-# * The temperature reaches its maxmium around August and its minimum around January
-# * The volume reaches its maximum around June and its minimum around August/September. It takes longer to reach its maximum than to reach its minimum.
-# * The hydrometry reaches its maximum around March and its minimum around September
-#
-# * The volume and hydrometry reach their minimum roughly around the same time
-# * The volume and hydrometry reach their minimum when the temperature reaches its maximum
-# * Temperature lags begind depth to groundwater by around 2 to 3 months
-
-# %% _kg_hide-input=true papermill={"duration": 1.424459, "end_time": "2021-01-17T15:25:21.835676", "exception": false, "start_time": "2021-01-17T15:25:20.411217", "status": "completed"} tags=[]
-f, ax = plt.subplots(nrows=5, ncols=1, figsize=(15, 12))
-f.suptitle('Seasonal Components of Features', fontsize=16)
-sns.lineplot(x=df.Date,
-             y=df.Depth_to_Groundwater_seasonal,
-             ax=ax[0],
-             color='dodgerblue',
-             label='P25')
-ax[0].set_ylabel(ylabel='Depth to Groundwater', fontsize=14)
-
-sns.lineplot(x=df.Date,
-             y=df.Temperature_seasonal,
-             ax=ax[1],
-             color='dodgerblue',
-             label='Bastia Umbra')
-ax[1].set_ylabel(ylabel='Temperature', fontsize=14)
-
-sns.lineplot(x=df.Date,
-             y=df.Drainage_Volume_seasonal,
-             ax=ax[2],
-             color='dodgerblue')
-ax[2].set_ylabel(ylabel='Volume', fontsize=14)
-
-sns.lineplot(x=df.Date,
-             y=df.River_Hydrometry_seasonal,
-             ax=ax[3],
-             color='dodgerblue')
-ax[3].set_ylabel(ylabel='Hydrometry', fontsize=14)
-
-sns.lineplot(x=df.Date, y=df.Rainfall_seasonal, ax=ax[4], color='dodgerblue')
-ax[4].set_ylabel(ylabel='Rainfall', fontsize=14)
-
-for i in range(5):
-    ax[i].set_xlim([date(2017, 9, 30), date(2020, 6, 30)])
-plt.tight_layout()
-plt.show()
-
-# %% [markdown] papermill={"duration": 0.104954, "end_time": "2021-01-17T15:25:22.047263", "exception": false, "start_time": "2021-01-17T15:25:21.942309", "status": "completed"} tags=[]
-# We can see that the correlation to the target variables increases if we use the time shifted features in comparison to the original features.
-
-# %% _kg_hide-input=true _kg_hide-output=false papermill={"duration": 0.783059, "end_time": "2021-01-17T15:25:22.934789", "exception": false, "start_time": "2021-01-17T15:25:22.151730", "status": "completed"} tags=[]
-f, ax = plt.subplots(nrows=1, ncols=2, figsize=(16, 8))
-
-original_cols = [
-    'Depth_to_Groundwater_seasonal', 'Temperature_seasonal',
-    'Drainage_Volume_seasonal', 'River_Hydrometry_seasonal'
-]
-
-corrmat = df[original_cols].corr()
-
-sns.heatmap(corrmat, annot=True, vmin=-1, vmax=1, cmap='coolwarm_r', ax=ax[0])
-ax[0].set_title('Correlation Matrix of Original Features', fontsize=16)
-
-shifted_cols = [
-    'Depth_to_Groundwater_seasonal', 'Temperature_seasonal_shift_r_2M',
-    'Drainage_Volume_seasonal_shift_1M', 'River_Hydrometry_seasonal_shift_3M'
-]
-corrmat = df[shifted_cols].corr()
-
-sns.heatmap(corrmat, annot=True, vmin=-1, vmax=1, cmap='coolwarm_r', ax=ax[1])
-ax[1].set_title('Correlation Matrix of Shifted Features', fontsize=16)
-
-plt.tight_layout()
-plt.show()
-
-# %% [markdown] papermill={"duration": 0.105977, "end_time": "2021-01-17T15:25:23.148401", "exception": false, "start_time": "2021-01-17T15:25:23.042424", "status": "completed"} tags=[]
-# ## Autocorrelation Analysis
-#
-# This EDA step is especially important when using [ARIMA](#ARIMA). The autocorrelation analysis helps to identify the AR and MA parameters for the [ARIMA](#ARIMA) model.
-#
-# Autocorrelation Function (ACF) and Partial Autocorrelation Function (PACF)
-#
-# * **Autocorrelation  Function (ACF)**: Correlation between time series with a lagged version of itself. The autocorrelation function starts a lag 0, which is the correlation of the time series with itself and therefore results in a correlation of 1. -> <font color='blue'>MA parameter is q significant lags</font>
-# * **Partial Autocorrelation Function (PACF)**: Additional correlation explained by each successive lagged term -> <font color='purple'>AR parameter is p significant lags</font>
-#
-# Autocorrelation pluts help in detecting seasonality.
-#
-# A good starting point for the AR parameter of the model may be TODO.
-#
-#
-# As we can infer from the graph above, the autocorrelation continues to decrease as the lag increases, confirming that there is no linear association between observations separated by larger lags.
-#
-# For the AR process, we expect that the ACF plot will gradually decrease and simultaneously the PACF should have a sharp drop after p significant lags. To define a MA process, we expect the opposite from the ACF and PACF plots, meaning that: the ACF should show a sharp drop after a certain q number of lags while PACF should show a geometric or gradual decreasing trend.
-
-# %% papermill={"duration": 0.284427, "end_time": "2021-01-17T15:25:23.540183", "exception": false, "start_time": "2021-01-17T15:25:23.255756", "status": "completed"} tags=[]
-from pandas.plotting import autocorrelation_plot
-
-autocorrelation_plot(df.Depth_to_Groundwater_diff_1)
-plt.show()
-
-# %% [markdown] papermill={"duration": 0.107098, "end_time": "2021-01-17T15:25:23.755173", "exception": false, "start_time": "2021-01-17T15:25:23.648075", "status": "completed"} tags=[]
-# We can see some sinusoidal shape in both ACF and PACF functions. This suggests that both AR and MA processes are present.
-
-# %% papermill={"duration": 0.486149, "end_time": "2021-01-17T15:25:24.348758", "exception": false, "start_time": "2021-01-17T15:25:23.862609", "status": "completed"} tags=[]
-from statsmodels.graphics.tsaplots import plot_acf
-from statsmodels.graphics.tsaplots import plot_pacf
-
-f, ax = plt.subplots(nrows=2, ncols=1, figsize=(16, 8))
-
-plot_acf(df.Depth_to_Groundwater_diff_1, lags=100, ax=ax[0])
-plot_pacf(df.Depth_to_Groundwater_diff_1, lags=100, ax=ax[1])
-plt.show()
-
-# %% _kg_hide-input=true _kg_hide-output=true papermill={"duration": 0.118076, "end_time": "2021-01-17T15:25:24.575605", "exception": false, "start_time": "2021-01-17T15:25:24.457529", "status": "completed"} tags=[]
-"""
-## Spectral Analysis
-to analyse cyclic behavior
-Frequency domain analysis
-
-## Trend estimation and decomposition
-used for seasonal adjustment
-"""
-
-# %% [markdown] papermill={"duration": 0.108611, "end_time": "2021-01-17T15:25:24.793172", "exception": false, "start_time": "2021-01-17T15:25:24.684561", "status": "completed"} tags=[]
-# # Cross Validation
-#
-# For cross validation, you can use the [Time Series Split](https://scikit-learn.org/stable/modules/cross_validation.html#time-series-split) library.
-# In [Time Series Forecasting: Building Intuition](https://www.kaggle.com/iamleonie/time-series-forecasting-building-intuition), I go into depth about different types of time series problems and their cross validation methods.
-
-# %% papermill={"duration": 0.132608, "end_time": "2021-01-17T15:25:25.034998", "exception": false, "start_time": "2021-01-17T15:25:24.902390", "status": "completed"} tags=[]
-from sklearn.model_selection import TimeSeriesSplit
-
-N_SPLITS = 3
-
-X = df.Date
-y = df.Depth_to_Groundwater
-
-folds = TimeSeriesSplit(n_splits=N_SPLITS)
-
-# %% _kg_hide-input=true papermill={"duration": 1.60402, "end_time": "2021-01-17T15:25:26.749090", "exception": false, "start_time": "2021-01-17T15:25:25.145070", "status": "completed"} tags=[]
-f, ax = plt.subplots(nrows=N_SPLITS, ncols=2, figsize=(16, 9))
-
-for i, (train_index, valid_index) in enumerate(folds.split(X)):
-    X_train, X_valid = X[train_index], X[valid_index]
-    y_train, y_valid = y[train_index], y[valid_index]
-
-    sns.lineplot(x=X_train,
-                 y=y_train,
-                 ax=ax[i, 0],
-                 color='dodgerblue',
-                 label='train')
-    sns.lineplot(
-        x=X_train[len(X_train) - len(X_valid):(len(X_train) - len(X_valid) +
-                                               len(X_valid))],
-        y=y_train[len(X_train) - len(X_valid):(len(X_train) - len(X_valid) +
-                                               len(X_valid))],
-        ax=ax[i, 1],
-        color='dodgerblue',
-        label='train')
-
-    for j in range(2):
-        sns.lineplot(x=X_valid,
-                     y=y_valid,
-                     ax=ax[i, j],
-                     color='darkorange',
-                     label='validation')
-    ax[i, 0].set_title(
-        f"Rolling Window with Adjusting Training Size (Split {i+1})",
-        fontsize=16)
-    ax[i, 1].set_title(
-        f"Rolling Window with Constant Training Size (Split {i+1})",
-        fontsize=16)
-
-for i in range(N_SPLITS):
-    ax[i, 0].set_xlim([date(2009, 1, 1), date(2020, 6, 30)])
-    ax[i, 1].set_xlim([date(2009, 1, 1), date(2020, 6, 30)])
-plt.tight_layout()
-plt.show()
-
-# %% [markdown] papermill={"duration": 0.117539, "end_time": "2021-01-17T15:25:26.982761", "exception": false, "start_time": "2021-01-17T15:25:26.865222", "status": "completed"} tags=[]
-# # Models
-#
-# Time series can be either **univariate or multivariate**:
-# * **Univariate** time series only has a single time-dependent variable.
-# * **Multivariate** time series have a multiple time-dependent variable.
-#
-# Our example originally is a multivariate time series because its has multiple features that are all time-dependent. However, by only looking at the target variable `Depth to Groundwater` we can convert it to a univariate time series.
-#
-# We will focus on a **quarterly forecast**. We will use the **Q2 2020 as test data** and the remaining data will be **split by quarter for cross validation**.
-#
-# We will evaluate the Mean Absolute Error (MAE) and the Root Mean Square Error (RMSE) of the models. For metrics are better the smaller they are.
-#
-#
-
-# %% [markdown] papermill={"duration": 0.114342, "end_time": "2021-01-17T15:25:27.211919", "exception": false, "start_time": "2021-01-17T15:25:27.097577", "status": "completed"} tags=[]
-# ## Models for Univariate Time Series
-#
-# * Stochastic Models
-#     * [Naive Approach](#Naive-Approach)<br>
-#     * [Moving Average](#Moving-Average)<br>
-#     * [Exponential Smoothing](#MExponential-Smoothing)<br>
-#     * [ARIMA](#ARIMA)<br>
-#     * [Prophet](#Prophet)<br>
-# * Deep Learning
-#     * [LSTM](#LSTM)<br>
-#     * [GRU](#GRU)<br>
-
-# %% _kg_hide-input=true _kg_hide-output=true papermill={"duration": 0.136393, "end_time": "2021-01-17T15:25:27.462982", "exception": false, "start_time": "2021-01-17T15:25:27.326589", "status": "completed"} tags=[]
-df['quarter_idx'] = (df.quarter != df.quarter.shift(1)).cumsum()
-
-target = 'Depth_to_Groundwater'
-features = [feature for feature in df.columns if feature != target]
-
-N_SPLITS = 46
-
-X = df[df.quarter_idx < N_SPLITS][features]
-y = df[df.quarter_idx < N_SPLITS][target]
-
-X_test = df[df.quarter_idx == N_SPLITS][features].reset_index(drop=True)
-y_test = df[df.quarter_idx == N_SPLITS][target].reset_index(drop=True)
-
-# %% _kg_hide-input=true papermill={"duration": 0.60257, "end_time": "2021-01-17T15:25:28.180699", "exception": false, "start_time": "2021-01-17T15:25:27.578129", "status": "completed"} tags=[]
-folds = np.linspace(0, N_SPLITS - 3, num=N_SPLITS - 2)
-
-f, ax = plt.subplots(nrows=2, ncols=1, figsize=(16, 8))
-
-sns.lineplot(x=X.Date, y=y, ax=ax[0], color='dodgerblue', label='train')
-sns.lineplot(x=X_test.Date,
-             y=y_test,
-             ax=ax[0],
-             color='darkorange',
-             label='test')
-
-sns.lineplot(x=df.Date, y=df.quarter_idx, ax=ax[1], color='dodgerblue')
-ax[0].set_xlim([date(2009, 1, 1), date(2020, 6, 30)])
-ax[1].set_xlim([date(2009, 1, 1), date(2020, 6, 30)])
-ax[1].set_ylim([0, N_SPLITS + 1])
-#ax[0].set_ylim([-28, -23])
-
-plt.show()
-
-
-# %% _kg_hide-input=true _kg_hide-output=true papermill={"duration": 0.134872, "end_time": "2021-01-17T15:25:28.432001", "exception": false, "start_time": "2021-01-17T15:25:28.297129", "status": "completed"} tags=[]
-def plot_approach_evaluation(y_pred, score_mae, score_rsme, approach_name,
-                             y_valid, X, folds, y, y_valid_pred, y_test,
-                             X_test):
-    f, ax = plt.subplots(nrows=1, ncols=2, figsize=(16, 8))
-    f.suptitle(approach_name, fontsize=16)
-    sns.lineplot(x=X.Date,
-                 y=y,
-                 ax=ax[0],
-                 color='dodgerblue',
-                 label='Training',
-                 linewidth=2)
-    sns.lineplot(x=X_test.Date,
-                 y=y_test,
-                 ax=ax[0],
-                 color='gold',
-                 label='Ground Truth',
-                 linewidth=2)  #navajowhite
-    sns.lineplot(x=X_test.Date,
-                 y=y_pred,
-                 ax=ax[0],
-                 color='darkorange',
-                 label='Predicted',
-                 linewidth=2)
-    ax[0].set_xlim([date(2018, 6, 30), date(2020, 6, 30)])
-    ax[0].set_ylim([-27, -23])
-    ax[0].set_title(
-        f'Prediction \n MAE: {mean_absolute_error(y_test, y_pred):.2f}, RSME: {math.sqrt(mean_squared_error(y_valid, y_valid_pred)):.2f}',
-        fontsize=14)
-    ax[0].set_xlabel(xlabel='Date', fontsize=14)
-    ax[0].set_ylabel(ylabel='Depth to Groundwater P25', fontsize=14)
-
-    sns.lineplot(x=folds, y=score_mae, color='gold', label='MAE',
-                 ax=ax[1])  #marker='o',
-    sns.lineplot(x=folds,
-                 y=score_rsme,
-                 color='indianred',
-                 label='RSME',
-                 ax=ax[1])
-    ax[1].set_title('Loss', fontsize=14)
-    ax[1].set_xlabel(xlabel='Fold', fontsize=14)
-    ax[1].set_ylabel(ylabel='Loss', fontsize=14)
-    ax[1].set_ylim([0, 4])
-    plt.show()
-
-
-# %% [markdown] papermill={"duration": 0.124053, "end_time": "2021-01-17T15:25:28.708573", "exception": false, "start_time": "2021-01-17T15:25:28.584520", "status": "completed"} tags=[]
-# ### Naive Approach
-#
-# $\hat y_{t+1} = y_t$
-
-# %% _kg_hide-input=true _kg_hide-output=false papermill={"duration": 0.747855, "end_time": "2021-01-17T15:25:29.570757", "exception": false, "start_time": "2021-01-17T15:25:28.822902", "status": "completed"} tags=[]
-score_mae = []
-score_rsme = []
-for fold, valid_quarter_id in enumerate(range(2, N_SPLITS)):
-    # Get indices for this fold
-    train_index = df[df.quarter_idx < valid_quarter_id].index
-    valid_index = df[df.quarter_idx == valid_quarter_id].index
-
-    # Prepare training and validation data for this fold
-    #X_train, X_valid = X.iloc[train_index], X.iloc[valid_index]
-    y_train, y_valid = y.iloc[train_index], y.iloc[valid_index]
-
-    # Initialize y_valid_pred
-    y_valid_pred = pd.Series(np.ones(len(y_valid)))
-
-    # Prediction: Naive approach
-    y_valid_pred = y_valid_pred * y_train.iloc[-1]
-
-    # Calcuate metrics
-    score_mae.append(mean_absolute_error(y_valid, y_valid_pred))
-    score_rsme.append(math.sqrt(mean_squared_error(y_valid, y_valid_pred)))
-
-y_pred = pd.Series(np.ones(len(X_test))) * y.iloc[-1]
-
-plot_approach_evaluation(y_pred, score_mae, score_rsme, 'Naive Approach',
-                         y_valid, X, folds, y, y_valid_pred, y_test, X_test)
-
-# %% [markdown] papermill={"duration": 0.120142, "end_time": "2021-01-17T15:25:29.808436", "exception": false, "start_time": "2021-01-17T15:25:29.688294", "status": "completed"} tags=[]
-# ### Moving Average
-
-# %% _kg_hide-input=true papermill={"duration": 1.235442, "end_time": "2021-01-17T15:25:31.161562", "exception": false, "start_time": "2021-01-17T15:25:29.926120", "status": "completed"} tags=[]
-score_mae = []
-score_rsme = []
-for fold, valid_quarter_id in enumerate(range(2, N_SPLITS)):
-    # Get indices for this fold
-    train_index = df[df.quarter_idx < valid_quarter_id].index
-    valid_index = df[df.quarter_idx == valid_quarter_id].index
-
-    # Prepare training and validation data for this fold
-    #X_train, X_valid = X.iloc[train_index], X.iloc[valid_index]
-    y_train, y_valid = y.iloc[train_index], y.iloc[valid_index]
-
-    # Initialize y_valid_pred
-    y_valid_pred = pd.Series(np.ones(len(y_valid)))
-
-    # Prediction: Naive approach
-    for i in range(len(y_valid_pred)):
-        y_valid_pred.iloc[i] = y_train.append(
-            y_valid_pred.iloc[:(i)]).reset_index(
-                drop=True).rolling(4).mean().iloc[-1]
-
-    # Calcuate metrics
-    score_mae.append(mean_absolute_error(y_valid, y_valid_pred))
-    score_rsme.append(math.sqrt(mean_squared_error(y_valid, y_valid_pred)))
-
-y_pred = pd.Series(np.zeros(len(X_test)))
-
-for i in range(len(y_pred)):
-    y_pred.iloc[i] = y.append(
-        y_pred.iloc[:(i)]).reset_index(drop=True).rolling(4).mean().iloc[-1]
-
-plot_approach_evaluation(y_pred, score_mae, score_rsme,
-                         'Moving Average (Window = 4 Weeks)', y_valid, X,
-                         folds, y, y_valid_pred, y_test, X_test)
-
-# %% [markdown] papermill={"duration": 0.118398, "end_time": "2021-01-17T15:25:31.400077", "exception": false, "start_time": "2021-01-17T15:25:31.281679", "status": "completed"} tags=[]
-# Neither the Naive Approach nor the Moving Average Approach are yielding good results for our example. Usually, these approaches serve as a benchmark rather than the method of choice.
-
-# %% [markdown] papermill={"duration": 0.121728, "end_time": "2021-01-17T15:25:31.643253", "exception": false, "start_time": "2021-01-17T15:25:31.521525", "status": "completed"} tags=[]
-# ### ARIMA
-# The Auto-Regressive Integrated Moving Average (ARIMA) model describes the **autocorrelations** in the data. The model assumes that the time-series is **stationary**. It consists of three main parts:
-# * <font color='purple'>Auto-Regressive (AR) filter (long term)</font>:
-#
-#     $\color{purple}{y_t = c + \alpha_1 y_{t-1} + \dots \alpha_{\color{purple}p}y_{t-\color{purple}p} + \epsilon_t = c + \sum_{i=1}^p{\alpha_i}y_{t-i} + \epsilon_t}$  -> p
-# * <font color='orange'> Integration filter (stochastic trend)</font>
-#
-#     -> d
-# * <font color='blue'>Moving Average (MA) filter (short term)</font>:
-#
-#     $\color{blue}{y_t = c + \epsilon_t + \beta_1 \epsilon_{t-1} + \dots + \beta_{q} \epsilon_{t-q} = c + \epsilon_t + \sum_{i=1}^q{\beta_i}\epsilon_{t-i}} $  -> q
-#
-#
-# **ARIMA**: $y_t = c + \color{purple}{\alpha_1 y_{t-1} + \dots + \alpha_{\color{purple}p}y_{t-\color{purple}p}}
-# + \color{blue}{\epsilon_t + \beta_1 \epsilon_{t-1} + \dots + \beta_{q} \epsilon_{t-q}}$
-#
-#
-# ARIMA(
-# <font color='purple'>p</font>,
-# <font color='orange'>d</font>,
-# <font color='blue'>q</font>)
-#
-# * <font color='purple'>p</font>: Lag order (to determine see  PACF in [Autocorrelation Analysis](#Autocorrelation-Analysis))
-# * <font color='orange'>d</font>: Degree of differencing. (to determine see  Differencing in [Stationarity](#Stationarity))
-# * <font color='blue'>q</font>: Order of moving average (to determine see  ACF in [Autocorrelation Analysis](#Autocorrelation-Analysis))
-#
-# In our example, we can use <font color='orange'>d=0</font> if we use the feature `Depth_to_Groundwater_diff_1`, which is `Depth_to_Groundwater` differenced by the first degree. Otherwise, if we were to use the non-stationary feature `Depth_to_Groundwater` as it is, we should set <font color='orange'>d=1</font>.
-#
-# (work in progress...)
-
-# %% _kg_hide-input=true _kg_hide-output=false papermill={"duration": 4.61287, "end_time": "2021-01-17T15:25:36.376551", "exception": false, "start_time": "2021-01-17T15:25:31.763681", "status": "completed"} tags=[]
-from statsmodels.tsa.arima.model import ARIMA
-
-score_mae = []
-score_rsme = []
-
-for fold, valid_quarter_id in enumerate(range(2, N_SPLITS)):
-    # Get indices for this fold
-    train_index = df[df.quarter_idx < valid_quarter_id].index
-    valid_index = df[df.quarter_idx == valid_quarter_id].index
-
-    # Prepare training and validation data for this fold
-    y_train, y_valid = y.iloc[train_index], y.iloc[valid_index]
-
-    # Fit model with Vector Auto Regression (VAR)
-    model = ARIMA(y_train, order=(1, 1, 1))
-    model_fit = model.fit()
-
-    # Prediction with Vector Auto Regression (VAR)
-    y_valid_pred = model_fit.predict(valid_index[0], valid_index[-1])
-
-    # Calcuate metrics
-    score_mae.append(mean_absolute_error(y_valid, y_valid_pred))
-    score_rsme.append(math.sqrt(mean_squared_error(y_valid, y_valid_pred)))
-
-# Fit model with Vector Auto Regression (VAR)
-model = ARIMA(y, order=(1, 1, 1))
-model_fit = model.fit()
-
-# Prediction with Vector Auto Regression (VAR)
-y_pred = model_fit.predict(y.index[-1] + 1,
-                           y.index[-1] + len(y_test)).reset_index(drop=True)
-plot_approach_evaluation(y_pred, score_mae, score_rsme, 'ARIMA', y_valid, X,
-                         folds, y, y_valid_pred, y_test, X_test)
-
-# %% [markdown] papermill={"duration": 0.122224, "end_time": "2021-01-17T15:25:36.622529", "exception": false, "start_time": "2021-01-17T15:25:36.500305", "status": "completed"} tags=[]
-# ## Models for Multivariate Time Series
-#
-# ### Vector Auto Regression (VAR)
-
-# %% _kg_hide-input=true papermill={"duration": 1.136181, "end_time": "2021-01-17T15:25:37.879666", "exception": false, "start_time": "2021-01-17T15:25:36.743485", "status": "completed"} tags=[]
-from statsmodels.tsa.api import VAR
-
-score_mae = []
-score_rsme = []
-
-features = ['Temperature', 'Drainage_Volume', 'River_Hydrometry', 'Rainfall']
-for fold, valid_quarter_id in enumerate(range(2, N_SPLITS)):
-    # Get indices for this fold
-    train_index = df[df.quarter_idx < valid_quarter_id].index
-    valid_index = df[df.quarter_idx == valid_quarter_id].index
-
-    # Prepare training and validation data for this fold
-    X_train, X_valid = X.iloc[train_index][features], X.iloc[valid_index][
-        features]
-    y_train, y_valid = y.iloc[train_index], y.iloc[valid_index]
-
-    # Fit model with Vector Auto Regression (VAR)
-    model = VAR(pd.concat([y_train, X_train], axis=1))
-    model_fit = model.fit()
-
-    # Prediction with Vector Auto Regression (VAR)
-    y_valid_pred = model_fit.forecast(model_fit.y, steps=len(X_valid))
-    y_valid_pred = pd.Series(y_valid_pred[:, 0])
-
-    # Calcuate metrics
-    score_mae.append(mean_absolute_error(y_valid, y_valid_pred))
-    score_rsme.append(math.sqrt(mean_squared_error(y_valid, y_valid_pred)))
-
-# Fit model with Vector Auto Regression (VAR)
-model = VAR(pd.concat([y, X[features]], axis=1))
-model_fit = model.fit()
-
-# Prediction with Vector Auto Regression (VAR)
-y_pred = model_fit.forecast(model_fit.y, steps=len(X_valid))
-y_pred = pd.Series(y_pred[:, 0])
-
-plot_approach_evaluation(y_pred, score_mae, score_rsme,
-                         'Vector Auto Regression (VAR)', y_valid, X, folds, y,
-                         y_valid_pred, y_test, X_test)
-
-# %% [markdown] papermill={"duration": 0.12475, "end_time": "2021-01-17T15:25:38.132380", "exception": false, "start_time": "2021-01-17T15:25:38.007630", "status": "completed"} tags=[]
-# # Additional Ressources
-#
-# My other notebook on time series:
-# * [Time Series Forecasting: Building Intuition](https://www.kaggle.com/iamleonie/time-series-forecasting-building-intuition)
-#
-# Here are some additional ressources that helped me learn about time series
-# * [Getting started with Time Series using Pandas ](https://www.kaggle.com/parulpandey/getting-started-with-time-series-using-pandas)
-# * [Time Series Analysis || An Introductory Start](https://www.kaggle.com/janiobachmann/time-series-analysis-an-introductory-start)
-# * [Everything you can do with a time series](https://www.kaggle.com/thebrownviking20/everything-you-can-do-with-a-time-series)
-# * [Deep Learning for Time Series | Dimitry Larko | Kaggle Days](https://www.youtube.com/watch?v=svNwWSgz2NM)
-# * [Encoding Cyclical Features for Deep Learning](https://www.kaggle.com/avanwyk/encoding-cyclical-features-for-deep-learning)
-# * [Tamara Louie: Applying Statistical Modeling & Machine Learning to Perform Time-Series Forecasting](https://www.youtube.com/watch?v=JntA9XaTebs)
-# * [Forecasting: Principles and Practice](https://otexts.com/fpp2/)
-# * [Easy Guide on Time Series Forecasting](https://beingdatum.com/time-series-forecasting/)
-#
-# # Useful Libraries
-# * [statsmodels](https://www.statsmodels.org)
-# * [Pandas Time series / date functionality](https://pandas.pydata.org/pandas-docs/stable/user_guide/timeseries.html)
-# * [Time Series Split](https://scikit-learn.org/stable/modules/cross_validation.html#time-series-split)
-
-# %% _kg_hide-input=true _kg_hide-output=true papermill={"duration": 0.133916, "end_time": "2021-01-17T15:25:38.391010", "exception": false, "start_time": "2021-01-17T15:25:38.257094", "status": "completed"} tags=[]
-"""
-### Exponential Smoothing
-based on a description of the **trend and seasonality** in the data
-
-### Prophet 
-
-### LSTM
-
-### GRU
-
-(work in progress)
-
-Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU)
-
-Recurrent Neural Network (RNN)
-"""
diff --git a/_kaggle/intro-to-time-series-forecasting/requirements.txt b/_kaggle/intro-to-time-series-forecasting/requirements.txt
deleted file mode 100644
index a9b54f6..0000000
--- a/_kaggle/intro-to-time-series-forecasting/requirements.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-statsmodels==0.12
\ No newline at end of file
diff --git a/_kaggle/linear-regression-house-price-prediction/nb.py b/_kaggle/linear-regression-house-price-prediction/nb.py
deleted file mode 100644
index cb591bb..0000000
--- a/_kaggle/linear-regression-house-price-prediction/nb.py
+++ /dev/null
@@ -1,661 +0,0 @@
-# -*- coding: utf-8 -*-
-# ---
-# jupyter:
-#   jupytext:
-#     text_representation:
-#       extension: .py
-#       format_name: percent
-#       format_version: '1.3'
-#       jupytext_version: 1.13.1
-#   kernelspec:
-#     display_name: Python 3
-#     language: python
-#     name: python3
-# ---
-
-# %% [markdown]
-# # 📈 Linear Regression with Python
-#
-# > Linear Regression is the simplest algorithm in machine learning, it can be trained in different ways. In this notebook we will cover the following linear algorithms:
-#
-# > 1. Linear Regression
-# > 2. Robust Regression
-# > 3. Ridge Regression
-# > 4. LASSO Regression
-# > 5. Elastic Net
-# > 6. Polynomial Regression
-# > 7. Stochastic Gradient Descent
-# > 8. Artificial Neaural Networks
-#
-# # 💾 Data
-#
-# > We are going to use the `USA_Housing` dataset. Since house price is a continues variable, this is a regression problem. The data contains the following columns:
-#
-# > * '`Avg. Area Income`': Avg. Income of residents of the city house is located in.
-# > * '`Avg. Area House Age`': Avg Age of Houses in same city
-# > * '`Avg. Area Number of Rooms`': Avg Number of Rooms for Houses in same city
-# > * '`Avg. Area Number of Bedrooms`': Avg Number of Bedrooms for Houses in same city
-# > * '`Area Population`': Population of city hou  se is located in
-# > * '`Price`': Price that the house sold at
-# > * '`Address`': Address for the house
-#
-
-# %% _kg_hide-input=true _kg_hide-output=true
-
-# %% [markdown]
-# # 📤 Import Libraries
-
-# %%
-import pandas as pd
-import numpy as np
-import matplotlib.pyplot as plt
-import seaborn as sns
-import hvplot.pandas
-
-# sns.set_style("whitegrid")
-# plt.style.use("fivethirtyeight")
-
-# %% [markdown]
-# ## 💾 Check out the Data
-
-# %%
-USAhousing = pd.read_csv('input/USA_Housing.csv')
-USAhousing.head()
-
-# %%
-USAhousing.info()
-
-# %%
-USAhousing.describe()
-
-# %%
-USAhousing.columns
-
-# %% [markdown]
-# # 📊 Exploratory Data Analysis (EDA)
-#
-# Let's create some simple plots to check out the data!
-
-# %%
-sns.pairplot(USAhousing)
-
-# %%
-USAhousing.hvplot.hist(by='Price', subplots=False, width=1000)
-
-# %%
-USAhousing.hvplot.hist("Price")
-
-# %%
-USAhousing.hvplot.scatter(x='Avg. Area House Age', y='Price')
-
-# %%
-USAhousing.hvplot.scatter(x='Avg. Area Income', y='Price')
-
-# %%
-USAhousing.columns
-
-# %%
-sns.heatmap(USAhousing.corr(), annot=True)
-
-# %% [markdown]
-# # 📈 Training a Linear Regression Model
-#
-# > Let's now begin to train out regression model! We will need to first split up our data into an X array that contains the features to train on, and a y array with the target variable, in this case the Price column. We will toss out the Address column because it only has text info that the linear regression model can't use.
-#
-# ## X and y arrays
-
-# %%
-X = USAhousing[[
-    'Avg. Area Income', 'Avg. Area House Age', 'Avg. Area Number of Rooms',
-    'Avg. Area Number of Bedrooms', 'Area Population'
-]]
-y = USAhousing['Price']
-
-# %% [markdown]
-# ## 🧱 Train Test Split
-#
-# Now let's split the data into a training set and a testing set. We will train out model on the training set and then use the test set to evaluate the model.
-
-# %%
-from sklearn.model_selection import train_test_split
-
-X_train, X_test, y_train, y_test = train_test_split(X,
-                                                    y,
-                                                    test_size=0.3,
-                                                    random_state=42)
-
-# %%
-from sklearn import metrics
-from sklearn.model_selection import cross_val_score
-
-
-def cross_val(model, X, y):
-    pred = cross_val_score(model, X, y, cv=10)
-    return pred.mean()
-
-
-def print_evaluate(true, predicted):
-    mae = metrics.mean_absolute_error(true, predicted)
-    mse = metrics.mean_squared_error(true, predicted)
-    rmse = np.sqrt(metrics.mean_squared_error(true, predicted))
-    r2_square = metrics.r2_score(true, predicted)
-    print('MAE:', mae)
-    print('MSE:', mse)
-    print('RMSE:', rmse)
-    print('R2 Square', r2_square)
-    print('__________________________________')
-
-
-def evaluate(true, predicted):
-    mae = metrics.mean_absolute_error(true, predicted)
-    mse = metrics.mean_squared_error(true, predicted)
-    rmse = np.sqrt(metrics.mean_squared_error(true, predicted))
-    r2_square = metrics.r2_score(true, predicted)
-    return mae, mse, rmse, r2_square
-
-
-# %% [markdown]
-# # 📦 Preparing Data For Linear Regression
-# > Linear regression is been studied at great length, and there is a lot of literature on how your data must be structured to make best use of the model.
-#
-# > As such, there is a lot of sophistication when talking about these requirements and expectations which can be intimidating. In practice, you can uses these rules more as rules of thumb when using Ordinary Least Squares Regression, the most common implementation of linear regression.
-#
-# > Try different preparations of your data using these heuristics and see what works best for your problem.
-# - **Linear Assumption.** Linear regression assumes that the relationship between your input and output is linear. It does not support anything else. This may be obvious, but it is good to remember when you have a lot of attributes. You may need to transform data to make the relationship linear (e.g. log transform for an exponential relationship).
-# - **Remove Noise.** Linear regression assumes that your input and output variables are not noisy. Consider using data cleaning operations that let you better expose and clarify the signal in your data. This is most important for the output variable and you want to remove outliers in the output variable (y) if possible.
-# - **Remove Collinearity.** Linear regression will over-fit your data when you have highly correlated input variables. Consider calculating pairwise correlations for your input data and removing the most correlated.
-# - **Gaussian Distributions.** Linear regression will make more reliable predictions if your input and output variables have a Gaussian distribution. You may get some benefit using transforms (e.g. log or BoxCox) on you variables to make their distribution more Gaussian looking.
-# - **Rescale Inputs:** Linear regression will often make more reliable predictions if you rescale input variables using standardization or normalization.
-
-# %%
-from sklearn.preprocessing import StandardScaler
-from sklearn.pipeline import Pipeline
-
-pipeline = Pipeline([('std_scalar', StandardScaler())])
-
-X_train = pipeline.fit_transform(X_train)
-X_test = pipeline.transform(X_test)
-
-# %% [markdown]
-# # ✔️ Linear Regression
-
-# %%
-from sklearn.linear_model import LinearRegression
-
-lin_reg = LinearRegression(normalize=True)
-lin_reg.fit(X_train, y_train)
-
-# %% [markdown]
-# ## ✔️ Model Evaluation
-#
-# Let's evaluate the model by checking out it's coefficients and how we can interpret them.
-
-# %%
-# print the intercept
-print(lin_reg.intercept_)
-
-# %%
-coeff_df = pd.DataFrame(lin_reg.coef_, X.columns, columns=['Coefficient'])
-coeff_df
-
-# %% [markdown]
-# > Interpreting the coefficients:
-# - Holding all other features fixed, a 1 unit increase in **Avg. Area Income** is associated with an **increase of \$21.52**.
-# - Holding all other features fixed, a 1 unit increase in **Avg. Area House Age** is associated with an **increase of \$164883.28**.
-# - Holding all other features fixed, a 1 unit increase in **Avg. Area Number of Rooms** is associated with an **increase of \$122368.67**.
-# - Holding all other features fixed, a 1 unit increase in **Avg. Area Number of Bedrooms** is associated with an **increase of \$2233.80**.
-# - Holding all other features fixed, a 1 unit increase in **Area Population** is associated with an **increase of \$15.15**.
-#
-# Does this make sense? Probably not because I made up this data.
-
-# %% [markdown]
-# ## ✔️ Predictions from our Model
-#
-# Let's grab predictions off our test set and see how well it did!
-
-# %%
-pred = lin_reg.predict(X_test)
-
-# %%
-pd.DataFrame({
-    'True Values': y_test,
-    'Predicted Values': pred
-}).hvplot.scatter(x='True Values', y='Predicted Values')
-
-# %% [markdown]
-# **Residual Histogram**
-
-# %%
-pd.DataFrame({'Error Values': (y_test - pred)}).hvplot.kde()
-
-# %% [markdown]
-# ## ✔️ Regression Evaluation Metrics
-#
-#
-# Here are three common evaluation metrics for regression problems:
-#
-# > - **Mean Absolute Error** (MAE) is the mean of the absolute value of the errors:
-# $$\frac 1n\sum_{i=1}^n|y_i-\hat{y}_i|$$
-#
-# > - **Mean Squared Error** (MSE) is the mean of the squared errors:
-# $$\frac 1n\sum_{i=1}^n(y_i-\hat{y}_i)^2$$
-#
-# > - **Root Mean Squared Error** (RMSE) is the square root of the mean of the squared errors:
-# $$\sqrt{\frac 1n\sum_{i=1}^n(y_i-\hat{y}_i)^2}$$
-#
-# > 📌 Comparing these metrics:
-# - **MAE** is the easiest to understand, because it's the average error.
-# - **MSE** is more popular than MAE, because MSE "punishes" larger errors, which tends to be useful in the real world.
-# - **RMSE** is even more popular than MSE, because RMSE is interpretable in the "y" units.
-#
-# > All of these are **loss functions**, because we want to minimize them.
-
-# %%
-test_pred = lin_reg.predict(X_test)
-train_pred = lin_reg.predict(X_train)
-
-print('Test set evaluation:\n_____________________________________')
-print_evaluate(y_test, test_pred)
-print('Train set evaluation:\n_____________________________________')
-print_evaluate(y_train, train_pred)
-
-# %%
-results_df = pd.DataFrame(
-    data=[[
-        "Linear Regression", *evaluate(y_test, test_pred),
-        cross_val(LinearRegression(), X=X, y=y)
-    ]],
-    columns=['Model', 'MAE', 'MSE', 'RMSE', 'R2 Square', "Cross Validation"])
-results_df
-
-# %% [markdown]
-# # ✔️ Robust Regression
-#
-# > Robust regression is a form of regression analysis designed to overcome some limitations of traditional parametric and non-parametric methods. Robust regression methods are designed to be not overly affected by violations of assumptions by the underlying data-generating process.
-#
-# > One instance in which robust estimation should be considered is when there is a strong suspicion of `heteroscedasticity`.
-#
-# > A common situation in which robust estimation is used occurs when the data contain outliers. In the presence of outliers that do not come from the same data-generating process as the rest of the data, least squares estimation is inefficient and can be biased. Because the least squares predictions are dragged towards the outliers, and because the variance of the estimates is artificially inflated, the result is that outliers can be masked. (In many situations, including some areas of geostatistics and medical statistics, it is precisely the outliers that are of interest.)
-
-# %% [markdown]
-# ## Random Sample Consensus - RANSAC
-#
-# > Random sample consensus (`RANSAC`) is an iterative method to estimate parameters of a mathematical model from a set of observed data that contains outliers, when outliers are to be accorded no influence on the values of the estimates. Therefore, it also can be interpreted as an outlier detection method.
-#
-# > A basic assumption is that the data consists of "inliers", i.e., data whose distribution can be explained by some set of model parameters, though may be subject to noise, and "outliers" which are data that do not fit the model. The outliers can come, for example, from extreme values of the noise or from erroneous measurements or incorrect hypotheses about the interpretation of data. RANSAC also assumes that, given a (usually small) set of inliers, there exists a procedure which can estimate the parameters of a model that optimally explains or fits this data.
-
-# %%
-from sklearn.linear_model import RANSACRegressor
-
-model = RANSACRegressor(base_estimator=LinearRegression(), max_trials=100)
-model.fit(X_train, y_train)
-
-test_pred = model.predict(X_test)
-train_pred = model.predict(X_train)
-
-print('Test set evaluation:\n_____________________________________')
-print_evaluate(y_test, test_pred)
-print('====================================')
-print('Train set evaluation:\n_____________________________________')
-print_evaluate(y_train, train_pred)
-
-# %%
-results_df_2 = pd.DataFrame(
-    data=[[
-        "Robust Regression", *evaluate(y_test, test_pred),
-        cross_val(RANSACRegressor(), X=X, y=y)
-    ]],
-    columns=['Model', 'MAE', 'MSE', 'RMSE', 'R2 Square', "Cross Validation"])
-results_df = results_df.append(results_df_2, ignore_index=True)
-results_df
-
-# %% [markdown]
-# # ✔️ Ridge Regression
-#
-# > Source: [scikit-learn](http://scikit-learn.org/stable/modules/linear_model.html#ridge-regression)
-#
-# > Ridge regression addresses some of the problems of **Ordinary Least Squares** by imposing a penalty on the size of coefficients. The ridge coefficients minimize a penalized residual sum of squares,
-#
-# $$\min_{w}\big|\big|Xw-y\big|\big|^2_2+\alpha\big|\big|w\big|\big|^2_2$$
-#
-# > $\alpha>=0$ is a complexity parameter that controls the amount of shrinkage: the larger the value of $\alpha$, the greater the amount of shrinkage and thus the coefficients become more robust to collinearity.
-#
-# > Ridge regression is an L2 penalized model. Add the squared sum of the weights to the least-squares cost function.
-# ***
-
-# %%
-from sklearn.linear_model import Ridge
-
-model = Ridge(alpha=100, solver='cholesky', tol=0.0001, random_state=42)
-model.fit(X_train, y_train)
-pred = model.predict(X_test)
-
-test_pred = model.predict(X_test)
-train_pred = model.predict(X_train)
-
-print('Test set evaluation:\n_____________________________________')
-print_evaluate(y_test, test_pred)
-print('====================================')
-print('Train set evaluation:\n_____________________________________')
-print_evaluate(y_train, train_pred)
-
-# %%
-results_df_2 = pd.DataFrame(
-    data=[[
-        "Ridge Regression", *evaluate(y_test, test_pred),
-        cross_val(Ridge(), X=X, y=y)
-    ]],
-    columns=['Model', 'MAE', 'MSE', 'RMSE', 'R2 Square', "Cross Validation"])
-results_df = results_df.append(results_df_2, ignore_index=True)
-results_df
-
-# %% [markdown]
-# # ✔️ LASSO Regression
-#
-# > A linear model that estimates sparse coefficients.
-#
-# > Mathematically, it consists of a linear model trained with $\ell_1$ prior as regularizer. The objective function to minimize is:
-#
-# $$\min_{w}\frac{1}{2n_{samples}} \big|\big|Xw - y\big|\big|_2^2 + \alpha \big|\big|w\big|\big|_1$$
-#
-# > The lasso estimate thus solves the minimization of the least-squares penalty with $\alpha \big|\big|w\big|\big|_1$ added, where $\alpha$ is a constant and $\big|\big|w\big|\big|_1$ is the $\ell_1-norm$ of the parameter vector.
-# ***
-
-# %%
-from sklearn.linear_model import Lasso
-
-model = Lasso(
-    alpha=0.1,
-    precompute=True,
-    #               warm_start=True,
-    positive=True,
-    selection='random',
-    random_state=42)
-model.fit(X_train, y_train)
-
-test_pred = model.predict(X_test)
-train_pred = model.predict(X_train)
-
-print('Test set evaluation:\n_____________________________________')
-print_evaluate(y_test, test_pred)
-print('====================================')
-print('Train set evaluation:\n_____________________________________')
-print_evaluate(y_train, train_pred)
-
-# %%
-results_df_2 = pd.DataFrame(
-    data=[[
-        "Lasso Regression", *evaluate(y_test, test_pred),
-        cross_val(Lasso(), X=X, y=y)
-    ]],
-    columns=['Model', 'MAE', 'MSE', 'RMSE', 'R2 Square', "Cross Validation"])
-results_df = results_df.append(results_df_2, ignore_index=True)
-results_df
-
-# %% [markdown]
-# # ✔️ Elastic Net
-#
-# > A linear regression model trained with L1 and L2 prior as regularizer.
-#
-# > This combination allows for learning a sparse model where few of the weights are non-zero like Lasso, while still maintaining the regularization properties of Ridge.
-#
-# > Elastic-net is useful when there are multiple features which are correlated with one another. Lasso is likely to pick one of these at random, while elastic-net is likely to pick both.
-#
-# > A practical advantage of trading-off between Lasso and Ridge is it allows Elastic-Net to inherit some of Ridge’s stability under rotation.
-#
-# > The objective function to minimize is in this case
-#
-# $$\min_{w}{\frac{1}{2n_{samples}} \big|\big|X w - y\big|\big|_2 ^ 2 + \alpha \rho \big|\big|w\big|\big|_1 +
-# \frac{\alpha(1-\rho)}{2} \big|\big|w\big|\big|_2 ^ 2}$$
-# ***
-
-# %%
-from sklearn.linear_model import ElasticNet
-
-model = ElasticNet(alpha=0.1,
-                   l1_ratio=0.9,
-                   selection='random',
-                   random_state=42)
-model.fit(X_train, y_train)
-
-test_pred = model.predict(X_test)
-train_pred = model.predict(X_train)
-
-print('Test set evaluation:\n_____________________________________')
-print_evaluate(y_test, test_pred)
-print('====================================')
-print('Train set evaluation:\n_____________________________________')
-print_evaluate(y_train, train_pred)
-
-# %%
-results_df_2 = pd.DataFrame(
-    data=[[
-        "Elastic Net Regression", *evaluate(y_test, test_pred),
-        cross_val(ElasticNet(), X=X, y=y)
-    ]],
-    columns=['Model', 'MAE', 'MSE', 'RMSE', 'R2 Square', "Cross Validation"])
-results_df = results_df.append(results_df_2, ignore_index=True)
-results_df
-
-# %% [markdown]
-# # ✔️ Polynomial Regression
-# > Source: [scikit-learn](http://scikit-learn.org/stable/modules/linear_model.html#polynomial-regression-extending-linear-models-with-basis-functions)
-#
-# ***
-#
-# > One common pattern within machine learning is to use linear models trained on nonlinear functions of the data. This approach maintains the generally fast performance of linear methods, while allowing them to fit a much wider range of data.
-#
-# > For example, a simple linear regression can be extended by constructing polynomial features from the coefficients. In the standard linear regression case, you might have a model that looks like this for two-dimensional data:
-#
-# $$\hat{y}(w, x) = w_0 + w_1 x_1 + w_2 x_2$$
-#
-# > If we want to fit a paraboloid to the data instead of a plane, we can combine the features in second-order polynomials, so that the model looks like this:
-#
-# $$\hat{y}(w, x) = w_0 + w_1 x_1 + w_2 x_2 + w_3 x_1 x_2 + w_4 x_1^2 + w_5 x_2^2$$
-#
-# > The (sometimes surprising) observation is that this is still a linear model: to see this, imagine creating a new variable
-#
-# $$z = [x_1, x_2, x_1 x_2, x_1^2, x_2^2]$$
-#
-# > With this re-labeling of the data, our problem can be written
-#
-# $$\hat{y}(w, x) = w_0 + w_1 z_1 + w_2 z_2 + w_3 z_3 + w_4 z_4 + w_5 z_5$$
-#
-# > We see that the resulting polynomial regression is in the same class of linear models we’d considered above (i.e. the model is linear in w) and can be solved by the same techniques. By considering linear fits within a higher-dimensional space built with these basis functions, the model has the flexibility to fit a much broader range of data.
-# ***
-
-# %%
-from sklearn.preprocessing import PolynomialFeatures
-
-poly_reg = PolynomialFeatures(degree=2)
-
-X_train_2_d = poly_reg.fit_transform(X_train)
-X_test_2_d = poly_reg.transform(X_test)
-
-lin_reg = LinearRegression(normalize=True)
-lin_reg.fit(X_train_2_d, y_train)
-
-test_pred = lin_reg.predict(X_test_2_d)
-train_pred = lin_reg.predict(X_train_2_d)
-
-print('Test set evaluation:\n_____________________________________')
-print_evaluate(y_test, test_pred)
-print('====================================')
-print('Train set evaluation:\n_____________________________________')
-print_evaluate(y_train, train_pred)
-
-# %%
-results_df_2 = pd.DataFrame(
-    data=[["Polynomail Regression", *evaluate(y_test, test_pred), 0]],
-    columns=['Model', 'MAE', 'MSE', 'RMSE', 'R2 Square', 'Cross Validation'])
-results_df = results_df.append(results_df_2, ignore_index=True)
-results_df
-
-# %% [markdown]
-# # ✔️ Stochastic Gradient Descent
-#
-# > Gradient Descent is a very generic optimization algorithm capable of finding optimal solutions to a wide range of problems. The general idea of Gradient Sescent is to tweak parameters iteratively in order to minimize a cost function. Gradient Descent measures the local gradient of the error function with regards to the parameters vector, and it goes in the direction of descending gradient. Once the gradient is zero, you have reached a minimum.
-
-# %%
-from sklearn.linear_model import SGDRegressor
-
-sgd_reg = SGDRegressor(n_iter_no_change=250,
-                       penalty=None,
-                       eta0=0.0001,
-                       max_iter=100000)
-sgd_reg.fit(X_train, y_train)
-
-test_pred = sgd_reg.predict(X_test)
-train_pred = sgd_reg.predict(X_train)
-
-print('Test set evaluation:\n_____________________________________')
-print_evaluate(y_test, test_pred)
-print('====================================')
-print('Train set evaluation:\n_____________________________________')
-print_evaluate(y_train, train_pred)
-
-# %%
-results_df_2 = pd.DataFrame(
-    data=[["Stochastic Gradient Descent", *evaluate(y_test, test_pred), 0]],
-    columns=['Model', 'MAE', 'MSE', 'RMSE', 'R2 Square', 'Cross Validation'])
-results_df = results_df.append(results_df_2, ignore_index=True)
-results_df
-
-# %% [markdown]
-# # ✔️ Artficial Neural Network
-
-# %% _kg_hide-output=true
-from tensorflow.keras.models import Sequential
-from tensorflow.keras.layers import Input, Dense, Activation, Dropout
-from tensorflow.keras.optimizers import Adam
-
-X_train = np.array(X_train)
-X_test = np.array(X_test)
-y_train = np.array(y_train)
-y_test = np.array(y_test)
-
-model = Sequential()
-
-model.add(Dense(X_train.shape[1], activation='relu'))
-model.add(Dense(32, activation='relu'))
-# model.add(Dropout(0.2))
-
-model.add(Dense(64, activation='relu'))
-# model.add(Dropout(0.2))
-
-model.add(Dense(128, activation='relu'))
-# model.add(Dropout(0.2))
-
-model.add(Dense(512, activation='relu'))
-model.add(Dropout(0.1))
-model.add(Dense(1))
-
-model.compile(optimizer=Adam(0.00001), loss='mse')
-
-r = model.fit(X_train,
-              y_train,
-              validation_data=(X_test, y_test),
-              batch_size=1,
-              epochs=1)
-
-# %%
-pd.DataFrame({
-    'True Values': y_test,
-    'Predicted Values': pred
-}).hvplot.scatter(x='True Values', y='Predicted Values')
-
-# %%
-pd.DataFrame(r.history)
-
-# %%
-pd.DataFrame(r.history).hvplot.line(y=['loss', 'val_loss'])
-
-# %%
-test_pred = model.predict(X_test)
-train_pred = model.predict(X_train)
-
-print('Test set evaluation:\n_____________________________________')
-print_evaluate(y_test, test_pred)
-
-print('Train set evaluation:\n_____________________________________')
-print_evaluate(y_train, train_pred)
-
-# %%
-results_df_2 = pd.DataFrame(
-    data=[["Artficial Neural Network", *evaluate(y_test, test_pred), 0]],
-    columns=['Model', 'MAE', 'MSE', 'RMSE', 'R2 Square', 'Cross Validation'])
-results_df = results_df.append(results_df_2, ignore_index=True)
-results_df
-
-# %% [markdown]
-# # ✔️ Random Forest Regressor
-
-# %%
-from sklearn.ensemble import RandomForestRegressor
-
-rf_reg = RandomForestRegressor(n_estimators=1000)
-rf_reg.fit(X_train, y_train)
-
-test_pred = rf_reg.predict(X_test)
-train_pred = rf_reg.predict(X_train)
-
-print('Test set evaluation:\n_____________________________________')
-print_evaluate(y_test, test_pred)
-
-print('Train set evaluation:\n_____________________________________')
-print_evaluate(y_train, train_pred)
-
-# %%
-results_df_2 = pd.DataFrame(
-    data=[["Random Forest Regressor", *evaluate(y_test, test_pred), 0]],
-    columns=['Model', 'MAE', 'MSE', 'RMSE', 'R2 Square', 'Cross Validation'])
-results_df = results_df.append(results_df_2, ignore_index=True)
-results_df
-
-# %% [markdown]
-# # ✔️ Support Vector Machine
-
-# %%
-from sklearn.svm import SVR
-
-svm_reg = SVR(kernel='rbf', C=1000000, epsilon=0.001)
-svm_reg.fit(X_train, y_train)
-
-test_pred = svm_reg.predict(X_test)
-train_pred = svm_reg.predict(X_train)
-
-print('Test set evaluation:\n_____________________________________')
-print_evaluate(y_test, test_pred)
-
-print('Train set evaluation:\n_____________________________________')
-print_evaluate(y_train, train_pred)
-
-# %%
-results_df_2 = pd.DataFrame(
-    data=[["SVM Regressor", *evaluate(y_test, test_pred), 0]],
-    columns=['Model', 'MAE', 'MSE', 'RMSE', 'R2 Square', 'Cross Validation'])
-results_df = results_df.append(results_df_2, ignore_index=True)
-results_df
-
-# %% [markdown]
-# # 📊 Models Comparison
-
-# %%
-results_df.set_index('Model', inplace=True)
-results_df['R2 Square'].plot(kind='barh', figsize=(12, 8))
-
-# %% [markdown]
-# # 📝 Summary
-# In this notebook you discovered the linear regression algorithm for machine learning.
-#
-# You covered a lot of ground including:
-# > - The common linear regression models (Ridge, Lasso, ElasticNet, ...).
-# > - The representation used by the model.
-# > - Learning algorithms used to estimate the coefficients in the model.
-# > - Rules of thumb to consider when preparing data for use with linear regression.
-# > - How to evaluate a linear regression model.
-#
-#
-# # 🔗 References:
-# - [Scikit-learn library](https://scikit-learn.org/stable/supervised_learning.html#supervised-learning)
-# - [Linear Regression for Machine Learning by Jason Brownlee PhD](https://machinelearningmastery.com/linear-regression-for-machine-learning/)
diff --git a/_kaggle/linear-regression-house-price-prediction/requirements.txt b/_kaggle/linear-regression-house-price-prediction/requirements.txt
deleted file mode 100644
index 812ae60..0000000
--- a/_kaggle/linear-regression-house-price-prediction/requirements.txt
+++ /dev/null
@@ -1,2 +0,0 @@
-hvplot
-tensorflow
\ No newline at end of file
diff --git a/_kaggle/look-at-this-note-feature-engineering-is-easy/nb.py b/_kaggle/look-at-this-note-feature-engineering-is-easy/nb.py
deleted file mode 100644
index 044eea6..0000000
--- a/_kaggle/look-at-this-note-feature-engineering-is-easy/nb.py
+++ /dev/null
@@ -1,534 +0,0 @@
-# -*- coding: utf-8 -*-
-# ---
-# jupyter:
-#   jupytext:
-#     text_representation:
-#       extension: .py
-#       format_name: percent
-#       format_version: '1.3'
-#       jupytext_version: 1.13.1
-#   kernelspec:
-#     display_name: Python 3
-#     language: python
-#     name: python3
-# ---
-
-# %% [markdown]
-# #### Hi all.  🙋
-#
-# #### Nice to meet you!
-#
-# #### We all know that feature engineering is the key to dynamically growing a model's performance in machine learning.
-#
-# #### You will try a lot of thought and various methods when doing feature engineering! I'm going to suggest a lot of ways to reduce the trouble and process.
-#
-# #### The methods I will introduce are both known and unfamiliar methods. I hope you will refer to them when conducting competitions on Kaggle in the future! 💯
-#
-#
-
-# %% _uuid="8f2839f25d086af736a60e9eeb907d3b93b6e0e5" _cell_guid="b1076dfc-b9ad-4769-8c92-a6c4dae69d19"
-import numpy as np  # linear algebra
-import pandas as pd  # data processing, CSV file I/O (e.g. pd.read_csv)
-from sklearn.model_selection import train_test_split
-import seaborn as sns
-import matplotlib.pyplot as plt
-
-# %%
-train = pd.read_csv('input/train.csv')
-train = train.drop(['Id'], axis=1)
-pd.set_option('display.max_columns', None)
-train.head()
-
-# %%
-# data segmentation
-X = train.drop('SalePrice', axis=1)
-y = train['SalePrice']
-
-train_x, test_x, train_y, test_y = train_test_split(
-    X, y, test_size=0.2, shuffle=True, random_state=0)  # train, valid 8:2 분할
-
-# %%
-# We need to duplicate the original state of our training data and test data.
-train_x_saved = train_x.copy()
-test_x_saved = test_x.copy()
-
-
-# Functions that return training data and test data
-def load_data(train_x_saved, test_x_saved):
-    train_x, test_x = train_x_saved.copy(), test_x_saved.copy()
-    return train_x, test_x
-
-
-# %%
-# Store the numeric variable to be converted into a list
-num_cols = [
-    'MSSubClass', 'LotFrontage', 'LotArea', 'OverallQual', 'OverallCond',
-    'YearBuilt', 'YearRemodAdd', 'MasVnrArea', 'BsmtFinSF1', '1stFlrSF',
-    '2ndFlrSF', 'GrLivArea', 'GarageYrBlt', 'GarageArea', 'WoodDeckSF'
-]
-
-# %% [markdown]
-# # Linear Transform
-
-# %% [markdown]
-# <div style="background-color:rgba(0, 255, 255, 0.6);border-radius:5px;display:fill;">
-#     <h1><center style ="margin-left : 20px;">Standardization</center></h1>
-# </div>
-
-# %% [markdown]
-# #### This is the most basic transformation method.
-# #### It is a method that makes the mean 0 and the standard deviation 1 through a linear transformation!
-
-# %% [markdown]
-# ![캡처.PNG](attachment:1f1a31d7-34c7-4485-8929-617724947a4a.PNG)
-
-# %%
-# Load Data
-train_x, test_x = load_data(train_x_saved=train_x_saved,
-                            test_x_saved=test_x_saved)
-
-# %%
-from sklearn.preprocessing import StandardScaler
-
-# %%
-scaler = StandardScaler()
-scaler.fit(train_x[num_cols])
-
-# %%
-# Permuting each column after normalization
-train_x[num_cols] = scaler.transform(train_x[num_cols])
-test_x[num_cols] = scaler.transform(test_x[num_cols])
-
-# %% [markdown]
-# <div style="background-color:red;border-radius:5px;display:fill;">
-#     <h1><center style ="margin-left : 20px;">BAD Standardization</center></h1>
-# </div>
-
-# %% [markdown]
-# #### In this method, training data and test data are transformed according to the mean and standard deviation of different criteria.
-#
-# #### If the distribution of each data does not differ significantly from each other, it is not a problem. However, this method should not be used. 💥
-
-# %%
-train_x, test_x = load_data(train_x_saved=train_x_saved,
-                            test_x_saved=test_x_saved)
-
-# %%
-from sklearn.preprocessing import StandardScaler
-
-# %%
-# Normalize training data and test data respectively (bad example)
-scaler_train = StandardScaler()
-scaler_train.fit(train_x[num_cols])
-train_x[num_cols] = scaler_train.transform(train_x[num_cols])
-
-scaler_test = StandardScaler()
-scaler_test.fit(test_x[num_cols])
-test_x[num_cols] = scaler_test.transform(test_x[num_cols])
-
-# %% [markdown]
-# <div style="background-color:rgba(0, 255, 255, 0.6);border-radius:5px;display:fill;">
-#     <h1><center style ="margin-left : 20px;">Min-Max Scaling</center></h1>
-# </div>
-
-# %% [markdown]
-# #### This is a Min-Max Scaling method that converts the range taken by the variable value into a specific interval (between 0 and 1).
-
-# %% [markdown]
-# ![min max.PNG](attachment:672ff330-b426-4986-8be3-ffcd502b763c.PNG)
-
-# %%
-train_x, test_x = load_data(train_x_saved=train_x_saved,
-                            test_x_saved=test_x_saved)
-
-# %%
-from sklearn.preprocessing import MinMaxScaler
-
-scaler = MinMaxScaler()
-scaler.fit(train_x[num_cols])
-
-# %%
-train_x[num_cols] = scaler.transform(train_x[num_cols])
-test_x[num_cols] = scaler.transform(test_x[num_cols])
-
-# %%
-train_x[num_cols].describe().T.style.bar(subset=['mean'], color='#205ff2')\
-                            .background_gradient(subset=['min'], cmap='Reds')\
-                            .background_gradient(subset=['max'], cmap='coolwarm')
-
-## The minimum value is 0 and the maximum value is 1.
-
-# %% [markdown]
-# # Non-linear Transformation
-
-# %% [markdown]
-# <div style="background-color:rgba(0, 255, 255, 0.6);border-radius:5px;display:fill;">
-#     <h1><center style ="margin-left : 20px;">Log</center></h1>
-# </div>
-
-# %% [markdown]
-# #### It is recommended that the distribution of variables is not skewed to one side.
-#
-# #### For example, a variable representing a specific amount or number of times tends
-#
-# #### to have a distribution that is biased in one direction,
-#
-# #### so log transformation is sometimes performed. And when the value is 0,
-#
-# #### log(x+1) transformation is often used because it cannot take the log as it is.
-
-# %%
-train_x, test_x = load_data(train_x_saved=train_x_saved,
-                            test_x_saved=test_x_saved)
-
-# %%
-x = train_x[num_cols]
-
-# %%
-# take log
-x1 = np.log(x)
-x1
-
-# %%
-# Add 1 and then take the logarithm
-x2 = np.log1p(x)
-x2
-
-# %%
-# After taking the logarithm of the absolute value, add the original sign
-x3 = np.sign(x) * np.log(np.abs(x))
-x3
-
-# %% [markdown]
-# <div style="background-color:rgba(0, 255, 255, 0.6);border-radius:5px;display:fill;">
-#     <h1><center style ="margin-left : 20px;">Box-Cox Transform</center></h1>
-# </div>
-
-# %% [markdown]
-# #### In addition to the BOX-COX Transform, which is a generalized log transformation,
-#
-# #### there is also the Yeo-Johnson Transform that can be applied to variables with negative values.
-# #### These transformations approximate a normal distribution after log transformation.
-
-# %% [markdown]
-# ![박스 칵스.PNG](attachment:9b085297-8728-451a-a81f-d145ab3db9f6.PNG)
-
-# %%
-train_x, test_x = load_data(train_x_saved=train_x_saved,
-                            test_x_saved=test_x_saved)
-
-# %%
-# Storing variables that take only positive integer values ​​as conversion targets in a list
-# Also, if missing values ​​are included, be careful because (~(train_x[c] <= 0.0)).all() should be used.
-pos_cols = [
-    c for c in num_cols
-    if (train_x[c] > 0.0).all() and (test_x[c] > 0.0).all()
-]
-
-## List of features with positive values
-pos_cols
-
-# %%
-from sklearn.preprocessing import PowerTransformer
-
-# %%
-pt = PowerTransformer(method='box-cox')
-pt.fit(train_x[pos_cols])
-
-# %%
-# 변환 후의 데이터로 각 열을 치환
-train_x[pos_cols] = pt.transform(train_x[pos_cols])
-test_x[pos_cols] = pt.transform(test_x[pos_cols])
-
-# %% [markdown]
-# #### LotArea column before after comparison
-
-# %%
-x = train.LotArea.values
-sns.kdeplot(x)
-plt.title("before Box-Cox-transform")
-plt.show()
-
-# %%
-x = train_x.LotArea.values
-sns.kdeplot(x)
-plt.title("after Box-Cox-transform")
-plt.show()
-
-## The existing data also has a form of a normal distribution,
-## so there is little difference between it and after the Box-Cox transformation.
-
-# %% [markdown]
-# #### GrLivArea column before after comparison
-
-# %%
-x = train.GrLivArea.values
-sns.kdeplot(x)
-plt.title("before Box-Cox-transform")
-plt.show()
-
-# %%
-x = train_x.GrLivArea.values
-sns.kdeplot(x)
-plt.title("after Box-Cox-transform")
-plt.show()
-
-## The existing data also has a form of a normal distribution,
-## so there is little difference between it and after the Box-Cox transformation.
-
-# %% [markdown]
-# <div style="background-color:rgba(0, 255, 255, 0.6);border-radius:5px;display:fill;">
-#     <h1><center style ="margin-left : 20px;">Yeo-Johnson Transform</center></h1>
-# </div>
-
-# %% [markdown]
-# #### Yeo-Johnson transform can also take negative values.
-
-# %% [markdown]
-# ![여존슨.PNG](attachment:667c009f-ca8a-4188-84b3-9539717b9634.PNG)
-
-# %%
-train_x, test_x = load_data(train_x_saved=train_x_saved,
-                            test_x_saved=test_x_saved)
-
-# %%
-from sklearn.preprocessing import PowerTransformer
-
-# %%
-pt = PowerTransformer(method='yeo-johnson')
-pt.fit(train_x[num_cols])
-
-# %%
-# 변환 후의 데이터로 각 열을 치환
-train_x[num_cols] = pt.transform(train_x[num_cols])
-test_x[num_cols] = pt.transform(test_x[num_cols])
-
-# %%
-train_x[num_cols]
-
-# %%
-import plotly.graph_objects as go
-
-fig = go.Figure()
-fig.add_trace(
-    go.Histogram(
-        x=train.MSSubClass,
-        xbins=dict(  # bins used for histogram
-            start=-100, end=200),
-        marker_color='blue',
-        opacity=1))
-
-fig.update_layout(
-    title_text='MSSubClass yeo-johnson Before',
-    xaxis_title_text='MSSubClass',
-    yaxis_title_text='Value',
-    bargap=0.05,  # gap between bars of adjacent location coordinates
-    xaxis={'showgrid': False},
-    yaxis={'showgrid': False},
-    template='plotly_dark')
-
-fig.show()
-
-# %%
-import plotly.graph_objects as go
-
-fig = go.Figure()
-fig.add_trace(
-    go.Histogram(
-        x=train_x.MSSubClass,
-        xbins=dict(  # bins used for histogram
-            start=0, end=200),
-        marker_color='blue',
-        opacity=1))
-
-fig.update_layout(
-    title_text='MSSubClass yeo-johnson After',
-    xaxis_title_text='MSSubClass',
-    yaxis_title_text='Value',
-    bargap=0.05,  # gap between bars of adjacent location coordinates
-    xaxis={'showgrid': False},
-    yaxis={'showgrid': False},
-    template='plotly_dark')
-
-fig.show()
-
-## The spread distribution was forced to approximate the normal distribution.
-
-# %% [markdown]
-# # Setting TransForm
-
-# %% [markdown]
-# <div style="background-color:rgba(0, 255, 255, 0.6);border-radius:5px;display:fill;">
-#     <h1><center style ="margin-left : 20px;">Clipping</center></h1>
-# </div>
-
-# %% [markdown]
-# #### Numerical variables sometimes include outliers, but you can exclude outliers
-# #### outside a certain range by setting upper and lower limits and replacing values
-# #### outside the range with upper and lower limits. It is also a good idea to check the distribution first and then set the threshold.
-
-# %%
-train_x, test_x = load_data(train_x_saved=train_x_saved,
-                            test_x_saved=test_x_saved)
-
-# %%
-# Check 1%, 99% points of training data per column
-p01 = train_x[num_cols].quantile(0.01)
-p99 = train_x[num_cols].quantile(0.99)
-
-p01
-p99
-
-# %%
-# Values ​​below 1% point are clipped to 1% point, and values ​​above 99% point are clipped to 99% point.
-train_x[num_cols] = train_x[num_cols].clip(p01, p99, axis=1)
-test_x[num_cols] = test_x[num_cols].clip(p01, p99, axis=1)
-
-# %% [markdown]
-# #### LotArea column before after comparison
-
-# %%
-import plotly.graph_objects as go
-
-fig = go.Figure()
-fig.add_trace(
-    go.Histogram(
-        x=train.LotArea,
-        xbins=dict(  # bins used for histogram
-            start=0, end=50000, size=2),
-        marker_color='#e8ab60',
-        opacity=1))
-
-fig.update_layout(
-    title_text='LotArea Clipping Before',
-    xaxis_title_text='LotArea',
-    yaxis_title_text='COUNT',
-    bargap=0.05,  # gap between bars of adjacent location coordinates
-    xaxis={'showgrid': False},
-    yaxis={'showgrid': False},
-    template='plotly_dark')
-
-fig.show()
-
-# %%
-import plotly.graph_objects as go
-
-fig = go.Figure()
-fig.add_trace(
-    go.Histogram(
-        x=train_x.LotArea,
-        xbins=dict(  # bins used for histogram
-            start=0, end=50000),
-        marker_color='#e8ab60',
-        opacity=1))
-
-fig.update_layout(
-    title_text='LotArea Clipping After',
-    xaxis_title_text='LotArea',
-    yaxis_title_text='COUNT',
-    bargap=0.05,  # gap between bars of adjacent location coordinates
-    xaxis={'showgrid': False},
-    yaxis={'showgrid': False},
-    template='plotly_dark')
-
-fig.show()
-
-## Values ​​from 0 to 80 are substituted.
-
-# %% [markdown]
-# #### RestingBP column before after comparison
-
-# %%
-fig = go.Figure()
-fig.add_trace(
-    go.Histogram(
-        x=train.GrLivArea,
-        xbins=dict(  # bins used for histogram
-            start=0, end=10000, size=15),
-        marker_color='#FE6F5E',
-        opacity=1))
-
-fig.update_layout(
-    title_text='GrLivArea Clipping Before',
-    xaxis_title_text='GrLivArea',
-    yaxis_title_text='COUNT',
-    bargap=0.05,  # gap between bars of adjacent location coordinates
-    xaxis={'showgrid': False},
-    yaxis={'showgrid': False},
-    template='plotly_dark')
-
-fig.show()
-
-# %%
-fig = go.Figure()
-fig.add_trace(
-    go.Histogram(
-        x=train_x.GrLivArea,
-        xbins=dict(  # bins used for histogram
-            start=0, end=10000, size=15),
-        marker_color='#FE6F5E',
-        opacity=1))
-
-fig.update_layout(
-    title_text='GrLivArea Clipping After',
-    xaxis_title_text='GrLivArea',
-    yaxis_title_text='COUNT',
-    bargap=0.05,  # gap between bars of adjacent location coordinates
-    xaxis={'showgrid': False},
-    yaxis={'showgrid': False},
-    template='plotly_dark')
-
-fig.show()
-
-# %% [markdown]
-# #### If you look at the graph, you can clearly see that the values ​​are not spread widely but are clustered like a normal distribution.
-
-# %% [markdown]
-# <div style="background-color:rgba(0, 255, 255, 0.6);border-radius:5px;display:fill;">
-#     <h1><center style ="margin-left : 20px;">Rank Gauss</center></h1>
-# </div>
-
-# %% [markdown]
-# #### This is a method of converting numeric variables into ranks and then semi-forced normal
-# #### distributions while maintaining the order. The method used by Kaggle Grandmaster
-# #### Michael Jahrer was revealed in the 1st solution in Porto Seguro's Safe Driver Prediction competition.
-# #### In particular, it is said to have better performance than general standardization as a transformation when building a model in a neural network.
-
-# %%
-train_x, test_x = load_data(train_x_saved=train_x_saved,
-                            test_x_saved=test_x_saved)
-
-from sklearn.preprocessing import QuantileTransformer
-
-# %%
-transformer = QuantileTransformer(n_quantiles=100,
-                                  random_state=0,
-                                  output_distribution='normal')
-transformer.fit(train_x[num_cols])
-
-# %%
-train_x[num_cols] = transformer.transform(train_x[num_cols])
-test_x[num_cols] = transformer.transform(test_x[num_cols])
-
-# %%
-train_x[num_cols]
-
-# %%
-p = sns.boxplot(x=train.GarageArea, color='teal')
-p.set_title("GarageArea RankGauss Before")
-plt.show()
-
-# %%
-p = sns.boxplot(x=train_x.GarageArea, color='teal')
-p.set_title("GarageArea RankGauss After")
-plt.show()
-
-# %% [markdown]
-# #### The values ​​were semi-forced to be normally distributed. The impact of outliers is also expected to decrease.
-
-# %% [markdown]
-# # NEXT PLAN
-
-# %% [markdown]
-# #### The following tabular data conversion will deal with numeric conversion of category types.
-# #### If you are interested in my kernel, please find the next category type conversion kernel as well.
diff --git a/_kaggle/look-at-this-note-feature-engineering-is-easy/requirements.txt b/_kaggle/look-at-this-note-feature-engineering-is-easy/requirements.txt
deleted file mode 100644
index d42d0ad..0000000
--- a/_kaggle/look-at-this-note-feature-engineering-is-easy/requirements.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-plotly
\ No newline at end of file
diff --git a/_kaggle/titanic-logistic-regression-with-python/nb.py b/_kaggle/titanic-logistic-regression-with-python/nb.py
deleted file mode 100644
index 52e534f..0000000
--- a/_kaggle/titanic-logistic-regression-with-python/nb.py
+++ /dev/null
@@ -1,825 +0,0 @@
-# ---
-# jupyter:
-#   jupytext:
-#     text_representation:
-#       extension: .py
-#       format_name: percent
-#       format_version: '1.3'
-#       jupytext_version: 1.13.1
-#   kernelspec:
-#     display_name: Python 3
-#     language: python
-#     name: python3
-# ---
-
-# %% [markdown] _cell_guid="cb19f71d-51c8-417f-829e-3179d3319dcd" _uuid="c3b9226e142667d6b96e34daf7d6e42bea0ea1e2"
-# # **Introduction**
-#
-# This kernel was inspired in part by the work of [SarahG](https://www.kaggle.com/sgus1318/titanic-analysis-learning-to-swim-with-python)'s analysis that I thank very much for the quality of her analysis. This work represents a deeper analysis by playing on several parameters while using only logistic regression estimator. In a future work, I will discuss other techniques. I am open to any criticism and proposal. You do not hesitate to evaluate this analysis.
-# The following kernel contains the steps enumerated below for assessing the Titanic survival dataset:
-#
-# 1. [Import data and python packages](#t1.)
-# 2. [Assess Data Quality & Missing Values](#t2.)
-#     * 2.1. [Age - Missing Values](#t2.1.)
-#     * 2.2. [Cabin - Missing Values](#t2.2.)
-#     * 2.3. [Embarked - Missing Values](#t2.3.)
-#     * 2.4. [Final Adjustments to Data](#t2.4.)
-#         * 2.4.1 [Additional Variables](#t2.4.1.)
-# 3. [Exploratory Data Analysis](#t3.)
-# 4. [Logistic Regression and Results](#t4.)
-#     * 4.1. [Feature selection](#t4.1.)
-#         * 4.1.1. [Recursive feature elimination](#t4.1.1.)
-#         * 4.1.2. [Feature ranking with recursive feature elimination and cross-validation](#t4.1.2.)
-#     * 4.2. [Review of model evaluation procedures](#t4.2.)
-#         * 4.2.1. [Model evaluation based on simple train/test split using `train_test_split()`](#t4.2.1.)
-#         * 4.2.2. [Model evaluation based on K-fold cross-validation using `cross_val_score()`](#t4.2.2.)
-#         * 4.2.3. [Model evaluation based on K-fold cross-validation using `cross_validate()`](#t4.2.3.)
-#     * 4.3. [GridSearchCV evaluating using multiple scorers simultaneously](#t4.3.)
-#     * 4.4. [GridSearchCV evaluating using multiple scorers, RepeatedStratifiedKFold and pipeline for preprocessing simultaneously](#t4.4.)
-
-# %% [markdown] _cell_guid="33c91cae-2ff8-45a6-b8cb-671619e9c933" _uuid="0a395fd25f20834b070ef55cb8987c8c1f9b55f9"
-# <a id="t1."></a>
-# # 1. Import Data & Python Packages
-
-# %% _cell_guid="de05512e-6991-44df-9599-da92a7e459ac" _uuid="d8bdd5f0320e244e4702ed8ec1c2482b022c51cd"
-import numpy as np
-import pandas as pd
-
-from sklearn import preprocessing
-import matplotlib.pyplot as plt
-
-plt.rc("font", size=14)
-import seaborn as sns
-
-sns.set(style="white")  #white background style for seaborn plots
-sns.set(style="whitegrid", color_codes=True)
-
-import warnings
-
-warnings.simplefilter(action='ignore')
-
-# %% _cell_guid="e0a17223-f682-45fc-89a5-667af9782bbe" _uuid="7964157913fbcff581fc1929eed487708e81ac9c"
-# Read CSV train data file into DataFrame
-train_df = pd.read_csv("input/train.csv")
-
-# Read CSV test data file into DataFrame
-test_df = pd.read_csv("input/test.csv")
-
-# preview train data
-train_df.head()
-
-# %% _cell_guid="872d0de9-a873-4b60-b1ee-d557ee39d8a1" _uuid="d38222a64d4dfd1d1ee1a7ee1f58c4aa54560de3"
-print('The number of samples into the train data is {}.'.format(
-    train_df.shape[0]))
-
-# %% _cell_guid="1d969b76-ea88-4d32-a58e-f22a070258bf" _uuid="bff38fcf31baf67493513c06f0c2f6e50576ff09"
-# preview test data
-test_df.head()
-
-# %% _cell_guid="254dd074-e07e-49f2-9184-80046b10b481" _uuid="62de7ddd73fed8d88ccbe1ba79e59b8e596cbb13"
-print('The number of samples into the test data is {}.'.format(
-    test_df.shape[0]))
-
-# %% [markdown] _cell_guid="4cd08f1e-9cb9-4d8e-99d0-3a9ce91b91c3" _uuid="2b1d45128663b9466fe9ac0059a13cfd4bd43657"
-# <font color=red>  Note: there is no target variable into test data (i.e. "Survival" column is missing), so the goal is to predict this target using different machine learning algorithms such as logistic regression. </font>
-
-# %% [markdown] _cell_guid="6578c0da-7bcf-433d-9f28-a66d8dfa6fa3" _uuid="8660e63a62c2fcdb4f7633380166438caf5edae9"
-# <a id="t2."></a>
-# # 2. Data Quality & Missing Value Assessment
-
-# %% _cell_guid="29dddd33-d995-4b0f-92ea-a361b368cc42" _uuid="d4fe22ead7e187724ca6f3ba7ba0e6412ae0e874"
-# check missing values in train data
-train_df.isnull().sum()
-
-# %% [markdown] _cell_guid="7776faeb-6a8f-4460-a367-4b087d2cc089" _uuid="696b428bd3ca49421f650665267ce7ca1b358814"
-# <a id="t2.1."></a>
-# ## 2.1.    Age - Missing Values
-
-# %% _cell_guid="d4ee6559-6d0c-409d-9dca-1d105a4ccd8a" _uuid="129cf984d05d9ce97c54548145e65f9e4b9b0c37"
-# percent of missing "Age"
-print('Percent of missing "Age" records is %.2f%%' %
-      ((train_df['Age'].isnull().sum() / train_df.shape[0]) * 100))
-
-# %% [markdown] _cell_guid="951f7bb8-779c-4eac-85a2-3fdcfdcd293e" _uuid="c8fff460fb532a063f6944450809014ca831ca52"
-# ~20% of entries for passenger age are missing. Let's see what the 'Age' variable looks like in general.
-
-# %% _cell_guid="6d65fcfa-52bf-45ab-b959-64a32c1c1976" _uuid="c6fd60f15d5e803d4dffc89e782c6fbc72445a83"
-ax = train_df["Age"].hist(bins=15,
-                          density=True,
-                          stacked=True,
-                          color='teal',
-                          alpha=0.6)
-train_df["Age"].plot(kind='density', color='teal')
-ax.set(xlabel='Age')
-plt.xlim(-10, 85)
-plt.show()
-
-# %% [markdown] _cell_guid="e62d6951-d968-43ba-aabf-add90524d042" _uuid="24c201948b9c8c8076ab01271a4790d9db9096b5"
-# Since "Age" is (right) skewed, using the mean might give us biased results by filling in ages that are older than desired. To deal with this, we'll use the median to impute the missing values.
-
-# %% _cell_guid="1d70c27b-1e4d-4d5e-8a39-c134389d436c" _uuid="4f13840d4f9bf1b4331523c99274aa0627485e6c"
-# mean age
-print('The mean of "Age" is %.2f' % (train_df["Age"].mean(skipna=True)))
-# median age
-print('The median of "Age" is %.2f' % (train_df["Age"].median(skipna=True)))
-
-# %% [markdown] _cell_guid="dea7b01c-c8c1-401f-a336-36ee73de2222" _uuid="e1a08114e302ddc90266e5f065b3f0b5a200bc89"
-# <a id="t2.2."></a>
-# ## 2.2. Cabin - Missing Values
-
-# %% _cell_guid="1a1ad808-0a63-43ac-b757-71195880ed4f" _uuid="1acbce9c6bc5d586dda3e47b7506067a85524e66"
-# percent of missing "Cabin"
-print('Percent of missing "Cabin" records is %.2f%%' %
-      ((train_df['Cabin'].isnull().sum() / train_df.shape[0]) * 100))
-
-# %% [markdown] _cell_guid="eda8c434-63ff-4875-8566-2e194c0d3f66" _uuid="b6e037c7ac5ec476516031a06b042d8b9999ba44"
-# 77% of records are missing, which means that imputing information and using this variable for prediction is probably not wise.  We'll ignore this variable in our model.
-
-# %% [markdown] _cell_guid="0e696cff-ca80-4cb5-862c-ee80f4b1ab1f" _uuid="d575319b1f528c7a153d8ab680282048cb163b14"
-# <a id="t2.3."></a>
-# ## 2.3. Embarked - Missing Values
-
-# %% _cell_guid="f21c2b55-2126-439d-8b1d-e96dafc97d81" _uuid="92ab9e62fb62f2a0fb9972baf6ada444187540e6"
-# percent of missing "Embarked"
-print('Percent of missing "Embarked" records is %.2f%%' %
-      ((train_df['Embarked'].isnull().sum() / train_df.shape[0]) * 100))
-
-# %% [markdown] _cell_guid="d03a4187-c527-4f71-8260-0495f4523e9e" _uuid="dc97b80524057522f024d0ae6f1abe77cb994903"
-# There are only 2 (0.22%) missing values for "Embarked", so we can just impute with the port where most people boarded.
-
-# %% _cell_guid="22924bc4-5dfa-4df7-b0d0-de3ede9c58b7" _uuid="f2a915f45264f8a580de6cc382d96b370eb75730"
-print(
-    'Boarded passengers grouped by port of embarkation (C = Cherbourg, Q = Queenstown, S = Southampton):'
-)
-print(train_df['Embarked'].value_counts())
-sns.countplot(x='Embarked', data=train_df, palette='Set2')
-plt.show()
-
-# %% _cell_guid="def67427-3257-4dce-872e-7f5b4202d18a" _uuid="c57a9f8a54efa382bc94b695c9664330d01709ea"
-print('The most common boarding port of embarkation is %s.' %
-      train_df['Embarked'].value_counts().idxmax())
-
-# %% [markdown] _cell_guid="c4c55f99-ce99-44f9-b7a8-d4d623ae9295" _uuid="19cfaae8c484dcb1d00f69b2771e86dc249e9793"
-# By far the most passengers boarded in Southhampton, so we'll impute those 2 NaN's w/ "S".
-
-# %% [markdown] _cell_guid="684c308f-25ae-4039-9332-ddb58953a054" _uuid="3609e785d210d5a8110f7ce550e61007d066449b"
-# <a id="t2.4."></a>
-# ## 2.4. Final Adjustments to Data (Train & Test)
-
-# %% [markdown] _cell_guid="b3025cdc-fe9f-43b6-bda1-e45c1f25e77c" _uuid="06d2762ccec3f11564870fe941fc9ac45d71662f"
-# Based on my assessment of the missing values in the dataset, I'll make the following changes to the data:
-# * If "Age" is missing for a given row, I'll impute with 28 (median age).
-# * If "Embarked" is missing for a riven row, I'll impute with "S" (the most common boarding port).
-# * I'll ignore "Cabin" as a variable. There are too many missing values for imputation. Based on the information available, it appears that this value is associated with the passenger's class and fare paid.
-
-# %% _cell_guid="bc0d7121-1008-4890-9043-07eba1524e15" _uuid="feeed4b6775f88edf5de12b0ee6ee73c16eba61d"
-train_data = train_df.copy()
-train_data["Age"].fillna(train_df["Age"].median(skipna=True), inplace=True)
-train_data["Embarked"].fillna(train_df['Embarked'].value_counts().idxmax(),
-                              inplace=True)
-train_data.drop('Cabin', axis=1, inplace=True)
-
-# %% _cell_guid="0cfe1c08-71a6-493e-803d-db255af01697" _uuid="d6be29651bb903964e02d3a7bcc7033513eb76c9"
-# check missing values in adjusted train data
-train_data.isnull().sum()
-
-# %% _cell_guid="10dcfe1b-34f1-4bd8-b937-5ae8daf4a378" _uuid="3ee37b1151416aeeec8ebd7b94bb0184aabc57cd"
-# preview adjusted train data
-train_data.head()
-
-# %% _cell_guid="dda26046-b93b-49ee-a52e-35355ecb425c" _uuid="293aec20df86ef529d10ae1f051dfe921ba07b88"
-plt.figure(figsize=(15, 8))
-ax = train_df["Age"].hist(bins=15,
-                          density=True,
-                          stacked=True,
-                          color='teal',
-                          alpha=0.6)
-train_df["Age"].plot(kind='density', color='teal')
-ax = train_data["Age"].hist(bins=15,
-                            density=True,
-                            stacked=True,
-                            color='orange',
-                            alpha=0.5)
-train_data["Age"].plot(kind='density', color='orange')
-ax.legend(['Raw Age', 'Adjusted Age'])
-ax.set(xlabel='Age')
-plt.xlim(-10, 85)
-plt.show()
-
-# %% [markdown] _cell_guid="6925fcc2-977b-4369-85e1-77a9210326a7" _uuid="d8280757e6bc627821fb0540c87ccd6ca110f1e0"
-# <a id="t2.4.1."></a>
-# ## 2.4.1. Additional Variables
-
-# %% [markdown] _cell_guid="5cf98f33-fdd5-4a16-b6bf-fa36bc8b84e0" _uuid="3bfdee842f11d27ca490f466c45ef9bf3673e7ae"
-# According to the Kaggle data dictionary, both SibSp and Parch relate to traveling with family.  For simplicity's sake (and to account for possible multicollinearity), I'll combine the effect of these variables into one categorical predictor: whether or not that individual was traveling alone.
-
-# %% _cell_guid="759c3c8e-8db6-41d9-a1a2-058a15b338a6" _uuid="d1f5815ba663f7e8cc17d7efcff73653af5b1bdb"
-## Create categorical variable for traveling alone
-train_data['TravelAlone'] = np.where(
-    (train_data["SibSp"] + train_data["Parch"]) > 0, 0, 1)
-train_data.drop('SibSp', axis=1, inplace=True)
-train_data.drop('Parch', axis=1, inplace=True)
-
-# %% [markdown] _cell_guid="e4a22367-b719-4204-952f-d2e9a3b8075e" _uuid="ca53796bf788bd3b015f1a79a97e050bafa2c770"
-# I'll also create categorical variables for Passenger Class ("Pclass"), Gender ("Sex"), and Port Embarked ("Embarked").
-
-# %% _cell_guid="f95361e8-2533-4731-a7ab-a99cf686ed50" _uuid="4494fcbf9faa90151e20042f74d73395fac3cc8e"
-#create categorical variables and drop some variables
-training = pd.get_dummies(train_data, columns=["Pclass", "Embarked", "Sex"])
-training.drop('Sex_female', axis=1, inplace=True)
-training.drop('PassengerId', axis=1, inplace=True)
-training.drop('Name', axis=1, inplace=True)
-training.drop('Ticket', axis=1, inplace=True)
-
-final_train = training
-final_train.head()
-
-# %% [markdown] _cell_guid="768cf074-6ecb-47eb-9b42-8b9079ffb811" _uuid="6a8e533e77c7f1f1a68d136119f93972447a31cf"
-# ### Now, apply the same changes to the test data. <br>
-# I will apply to same imputation for "Age" in the Test data as I did for my Training data (if missing, Age = 28).  <br> I'll also remove the "Cabin" variable from the test data, as I've decided not to include it in my analysis. <br> There were no missing values in the "Embarked" port variable. <br> I'll add the dummy variables to finalize the test set.  <br> Finally, I'll impute the 1 missing value for "Fare" with the median, 14.45.
-
-# %% _cell_guid="501f9a53-881d-4440-9366-7aae67eb358b" _uuid="d80416a026d17ccac3bf793408dd5f4f1e17bf63"
-test_df.isnull().sum()
-
-# %% _cell_guid="8b9ef076-3669-4339-8d10-0d8783a92e07" _uuid="145675b90aa2befa533c640aaedd4bf8069b12d4"
-test_data = test_df.copy()
-test_data["Age"].fillna(train_df["Age"].median(skipna=True), inplace=True)
-test_data["Fare"].fillna(train_df["Fare"].median(skipna=True), inplace=True)
-test_data.drop('Cabin', axis=1, inplace=True)
-
-test_data['TravelAlone'] = np.where(
-    (test_data["SibSp"] + test_data["Parch"]) > 0, 0, 1)
-
-test_data.drop('SibSp', axis=1, inplace=True)
-test_data.drop('Parch', axis=1, inplace=True)
-
-testing = pd.get_dummies(test_data, columns=["Pclass", "Embarked", "Sex"])
-testing.drop('Sex_female', axis=1, inplace=True)
-testing.drop('PassengerId', axis=1, inplace=True)
-testing.drop('Name', axis=1, inplace=True)
-testing.drop('Ticket', axis=1, inplace=True)
-
-final_test = testing
-final_test.head()
-
-# %% [markdown] _cell_guid="1430d510-1c8d-4544-8009-3911fff7afbb" _uuid="4e26c19bf719b7086addc0e1981c00836a19f189"
-# <a id="t3."></a>
-# # 3. Exploratory Data Analysis
-
-# %% [markdown] _cell_guid="2655428b-d69d-4c0f-85ff-e31ada8e37b9" _uuid="32e9c04a3281fb1aa8c77e1406c56cd820459202"
-# <a id="t3.1."></a>
-# ## 3.1. Exploration of Age
-
-# %% _cell_guid="9f9ca9e5-50a0-4487-ba53-815dda90af1c" _uuid="790e8d7ca89d19e276b3398e299c42893a796b79"
-plt.figure(figsize=(15, 8))
-ax = sns.kdeplot(final_train["Age"][final_train.Survived == 1],
-                 color="darkturquoise",
-                 shade=True)
-sns.kdeplot(final_train["Age"][final_train.Survived == 0],
-            color="lightcoral",
-            shade=True)
-plt.legend(['Survived', 'Died'])
-plt.title(
-    'Density Plot of Age for Surviving Population and Deceased Population')
-ax.set(xlabel='Age')
-plt.xlim(-10, 85)
-plt.show()
-
-# %% [markdown] _cell_guid="8e304d72-27f3-41cf-863f-63872f4c37df" _uuid="6c5625b454f5e01dd6b6d843d801851c14c64d1e"
-# The age distribution for survivors and deceased is actually very similar.  One notable difference is that, of the survivors, a larger proportion were children.  The passengers evidently made an attempt to save children by giving them a place on the life rafts.
-
-# %% _cell_guid="d2aa9f59-c433-4258-b8db-225b63a5eab6" _uuid="9cf1794d9db2fdc314c20ca97a76e9470e81a354"
-plt.figure(figsize=(20, 8))
-avg_survival_byage = final_train[["Age",
-                                  "Survived"]].groupby(['Age'],
-                                                       as_index=False).mean()
-g = sns.barplot(x='Age',
-                y='Survived',
-                data=avg_survival_byage,
-                color="LightSeaGreen")
-plt.show()
-
-# %% [markdown] _cell_guid="0b636440-ab38-46a8-8cc9-9421683d5c0b" _uuid="67051cf653243b3103c9f8015c501d89d92bd3bc"
-# Considering the survival rate of passengers under 16, I'll also include another categorical variable in my dataset: "Minor"
-
-# %% _cell_guid="1655b49b-b33f-4236-8b31-d995ef26c6f6" _uuid="8918defa6e17b83c700ea45357ebd67a3a22f02f"
-final_train['IsMinor'] = np.where(final_train['Age'] <= 16, 1, 0)
-
-final_test['IsMinor'] = np.where(final_test['Age'] <= 16, 1, 0)
-
-# %% [markdown] _cell_guid="a643b196-91c6-4b12-9463-0f984fbfc91a" _uuid="337b3ced0c6423cf1d126f23a7e60c0181af6a47"
-# <a id="t3.2."></a>
-# ## 3.2. Exploration of Fare
-
-# %% _cell_guid="9f31ffe1-7cd8-4169-b193-ed44e56d0bd4" _uuid="4a1c521f08460f6983eca0c4e01294fb7c86e4f9"
-plt.figure(figsize=(15, 8))
-ax = sns.kdeplot(final_train["Fare"][final_train.Survived == 1],
-                 color="darkturquoise",
-                 shade=True)
-sns.kdeplot(final_train["Fare"][final_train.Survived == 0],
-            color="lightcoral",
-            shade=True)
-plt.legend(['Survived', 'Died'])
-plt.title(
-    'Density Plot of Fare for Surviving Population and Deceased Population')
-ax.set(xlabel='Fare')
-plt.xlim(-20, 200)
-plt.show()
-
-# %% [markdown] _cell_guid="346b7322-a3e4-48df-bbb1-4d8ec7716f3f" _uuid="2717310b6c443d675c7342be0c2c18b265723273"
-# As the distributions are clearly different for the fares of survivors vs. deceased, it's likely that this would be a significant predictor in our final model.  Passengers who paid lower fare appear to have been less likely to survive.  This is probably strongly correlated with Passenger Class, which we'll look at next.
-
-# %% [markdown] _cell_guid="cf585311-4029-4be4-8af2-3eea8258801a" _uuid="4524affda51265ea23fa923e2ea7f93d7bb91875"
-# <a id="t3.3."></a>
-# ## 3.3. Exploration of Passenger Class
-
-# %% _cell_guid="676548e8-6dd4-4180-800c-7b164acb3877" _uuid="08fd677214959e0b938a0f8a94b63ab548673ea5"
-sns.barplot('Pclass', 'Survived', data=train_df, color="darkturquoise")
-plt.show()
-
-# %% [markdown] _cell_guid="193233f8-b220-4cae-aa0f-f822316d5623" _uuid="8ddb19191253a6e09dfcb0beff2b3690f1052d52"
-# Unsurprisingly, being a first class passenger was safest.
-
-# %% [markdown] _cell_guid="c59f8e8f-e8c2-40fb-b9c8-12dddd6d318f" _uuid="2fc06b75321946b721852f78431435f9ba5fef39"
-# <a id="t3.4."></a>
-# ## 3.4. Exploration of Embarked Port
-
-# %% _cell_guid="6e5bec50-2f5e-433e-9130-c56956fddad3" _uuid="a9f0598701c7c5224eaa73dafa869af73beffe18"
-sns.barplot('Embarked', 'Survived', data=train_df, color="teal")
-plt.show()
-
-# %% [markdown] _cell_guid="88d78820-35a5-48fd-a234-9f3ca3fca779" _uuid="2f6a0329cf0c7b771a707ec790efc065924e1ee2"
-# Passengers who boarded in Cherbourg, France, appear to have the highest survival rate.  Passengers who boarded in Southhampton were marginally less likely to survive than those who boarded in Queenstown.  This is probably related to passenger class, or maybe even the order of room assignments (e.g. maybe earlier passengers were more likely to have rooms closer to deck). <br> It's also worth noting the size of the whiskers in these plots.  Because the number of passengers who boarded at Southhampton was highest, the confidence around the survival rate is the highest.  The whisker of the Queenstown plot includes the Southhampton average, as well as the lower bound of its whisker.  It's possible that Queenstown passengers were equally, or even more, ill-fated than their Southhampton counterparts.
-
-# %% [markdown] _cell_guid="9e6dc87e-ba59-4004-8145-79709328fe27" _uuid="92bacce85a7dec5509217b9570bc2a2fea6a8452"
-# <a id="t3.5."></a>
-# ## 3.5. Exploration of Traveling Alone vs. With Family
-
-# %% _cell_guid="67017a88-93d4-412b-9adf-8b4d1d9b9db0" _uuid="e0c3dc16292ef0bcabf0fc680d821ef654084ab4"
-sns.barplot(x='TravelAlone',
-            y='Survived',
-            data=final_train,
-            color="mediumturquoise")
-plt.show()
-
-# %% [markdown] _cell_guid="e9e68cef-5e74-46aa-8343-39afbbf00efe" _uuid="f160bd7399e024ae669d55f09caf6e7902768851"
-# Individuals traveling without family were more likely to die in the disaster than those with family aboard. Given the era, it's likely that individuals traveling alone were likely male.
-
-# %% [markdown] _cell_guid="201b4c9d-b9f0-4ae9-8580-0b4e24ee62be" _uuid="693c25c25f3590f0b027725471ddd74d56f154af"
-# <a id="t3.6."></a>
-# ## 3.6. Exploration of Gender Variable
-
-# %% _cell_guid="7b416e59-8616-4a44-93e1-a8005eff78a9" _uuid="354794315925dff1e96229cc737eaf299aaea17a"
-sns.barplot('Sex', 'Survived', data=train_df, color="aquamarine")
-plt.show()
-
-# %% [markdown] _cell_guid="490ed298-f0e4-466b-acc8-81280315e6a2" _uuid="80c02b9fe2151c443f189cbe44c9cacf7e5c44a4"
-# This is a very obvious difference.  Clearly being female greatly increased your chances of survival.
-
-# %% [markdown] _cell_guid="c833cbf5-74db-44ff-90fa-b600ff0a09d7" _uuid="39dbc095f99dcec6d25a7a4561e81bb641078622"
-# <a id="t4."></a>
-# # 4. Logistic Regression and Results
-
-# %% [markdown] _cell_guid="b70cda8a-e8d9-44a6-b9f0-2b365fdf3428" _uuid="136cf9e02ea1ab48a397f534b491fb2d9dbb5684"
-# <a id="t4.1."></a>
-# ## 4.1. Feature selection
-#
-# <a id="t4.1.1."></a>
-# ### 4.1.1. Recursive feature elimination
-#
-# Given an external estimator that assigns weights to features, recursive feature elimination (RFE) is to select features by recursively considering smaller and smaller sets of features. First, the estimator is trained on the initial set of features and the importance of each feature is obtained either through a `coef_ attribute` or through a `feature_importances_` attribute. Then, the least important features are pruned from current set of features.That procedure is recursively repeated on the pruned set until the desired number of features to select is eventually reached.
-#
-# References: <br>
-# http://scikit-learn.org/stable/modules/feature_selection.html <br>
-
-# %% _cell_guid="11a2a468-20df-40cd-a4ba-4ae7bd2fc403" _uuid="64befdf1182c2b4e845f488f5bfd0e19ce3dc17a"
-from sklearn.linear_model import LogisticRegression
-from sklearn.feature_selection import RFE
-
-cols = [
-    "Age", "Fare", "TravelAlone", "Pclass_1", "Pclass_2", "Embarked_C",
-    "Embarked_S", "Sex_male", "IsMinor"
-]
-X = final_train[cols]
-y = final_train['Survived']
-# Build a logreg and compute the feature importances
-model = LogisticRegression()
-# create the RFE model and select 8 attributes
-rfe = RFE(model, n_features_to_select=8)
-rfe = rfe.fit(X, y)
-# summarize the selection of the attributes
-print('Selected features: %s' % list(X.columns[rfe.support_]))
-
-# %% [markdown] _cell_guid="29281bd5-b954-4f3d-87e3-7416b1ec8c6b" _uuid="626da3348b48ced3564e6e05bdb0c3b4bd1402e6"
-# <a id="t4.1.2."></a>
-# ### 4.1.2. Feature ranking with recursive feature elimination and cross-validation
-#
-# RFECV performs RFE in a cross-validation loop to find the optimal number or the best number of features. Hereafter a recursive feature elimination applied on logistic regression with automatic tuning of the number of features selected with cross-validation.
-
-# %% _cell_guid="7239aa6f-7fd2-4b75-a387-f6624f1c338c" _uuid="53d79f38cfe33d75d6ff869a443b9a29c93b4cbd"
-from sklearn.feature_selection import RFECV
-# Create the RFE object and compute a cross-validated score.
-# The "accuracy" scoring is proportional to the number of correct classifications
-rfecv = RFECV(estimator=LogisticRegression(),
-              step=1,
-              cv=10,
-              scoring='accuracy')
-rfecv.fit(X, y)
-
-print("Optimal number of features: %d" % rfecv.n_features_)
-print('Selected features: %s' % list(X.columns[rfecv.support_]))
-
-# Plot number of features VS. cross-validation scores
-plt.figure(figsize=(10, 6))
-plt.xlabel("Number of features selected")
-plt.ylabel("Cross validation score (nb of correct classifications)")
-plt.plot(range(1, len(rfecv.grid_scores_) + 1), rfecv.grid_scores_)
-plt.show()
-
-# %% [markdown] _cell_guid="b1b3b56f-2f5f-47d6-9375-62c11e49ce79" _uuid="e9d52d5b182c0a01218982e844e53d5278e0d98a"
-# As we see, eight variables were kept.
-
-# %% _cell_guid="08986ec4-79ff-466b-b763-61bf84a0879b" _uuid="3f6950a7c24c629b72e17e54c556f3c183b3f779"
-Selected_features = [
-    'Age', 'TravelAlone', 'Pclass_1', 'Pclass_2', 'Embarked_C', 'Embarked_S',
-    'Sex_male', 'IsMinor'
-]
-X = final_train[Selected_features]
-
-plt.subplots(figsize=(8, 5))
-sns.heatmap(X.corr(), annot=True, cmap="RdYlGn")
-plt.show()
-
-# %% [markdown] _cell_guid="a7455afe-9716-4189-b207-f1cc9facce12" _uuid="46b76691c5f109b17f805f233a5ad5ba900b353b"
-# <a id="t4.2."></a>
-# ## 4.2. Review of model evaluation procedures
-#
-# Motivation: Need a way to choose between machine learning models
-# * Goal is to estimate likely performance of a model on out-of-sample data
-#
-# Initial idea: Train and test on the same data
-# * But, maximizing training accuracy rewards overly complex models which overfit the training data
-#
-# Alternative idea: Train/test split
-# * Split the dataset into two pieces, so that the model can be trained and tested on different data
-# * Testing accuracy is a better estimate than training accuracy of out-of-sample performance
-# * Problem with train/test split
-#     * It provides a high variance estimate since changing which observations happen to be in the testing set can significantly change testing accuracy
-#     * Testing accuracy can change a lot depending on a which observation happen to be in the testing set
-#
-# Reference: <br>
-# http://www.ritchieng.com/machine-learning-cross-validation/ <br>
-
-# %% [markdown] _cell_guid="b894002e-07cf-4d02-b708-a2ac387eed54" _uuid="e35125f8aa230d4875541aa4f6b5964d2f14a6a3"
-# <a id="t4.2.1."></a>
-# ### 4.2.1. Model evaluation based on simple train/test split using `train_test_split()` function
-
-# %% _cell_guid="84233f59-f3c7-4ea0-884d-96f8ad4d5b10" _uuid="46336228eeb864bc82e6739768122579d1c9634c"
-from sklearn.model_selection import train_test_split, cross_val_score
-from sklearn.metrics import accuracy_score, classification_report, precision_score, recall_score
-from sklearn.metrics import confusion_matrix, precision_recall_curve, roc_curve, auc, log_loss
-
-# create X (features) and y (response)
-X = final_train[Selected_features]
-y = final_train['Survived']
-
-# use train/test split with different random_state values
-# we can change the random_state values that changes the accuracy scores
-# the scores change a lot, this is why testing scores is a high-variance estimate
-X_train, X_test, y_train, y_test = train_test_split(X,
-                                                    y,
-                                                    test_size=0.2,
-                                                    random_state=2)
-
-# check classification scores of logistic regression
-logreg = LogisticRegression()
-logreg.fit(X_train, y_train)
-y_pred = logreg.predict(X_test)
-y_pred_proba = logreg.predict_proba(X_test)[:, 1]
-[fpr, tpr, thr] = roc_curve(y_test, y_pred_proba)
-print('Train/Test split results:')
-print(logreg.__class__.__name__ +
-      " accuracy is %2.3f" % accuracy_score(y_test, y_pred))
-print(logreg.__class__.__name__ +
-      " log_loss is %2.3f" % log_loss(y_test, y_pred_proba))
-print(logreg.__class__.__name__ + " auc is %2.3f" % auc(fpr, tpr))
-
-idx = np.min(np.where(tpr > 0.95)
-             )  # index of the first threshold for which the sensibility > 0.95
-
-plt.figure()
-plt.plot(fpr,
-         tpr,
-         color='coral',
-         label='ROC curve (area = %0.3f)' % auc(fpr, tpr))
-plt.plot([0, 1], [0, 1], 'k--')
-plt.plot([0, fpr[idx]], [tpr[idx], tpr[idx]], 'k--', color='blue')
-plt.plot([fpr[idx], fpr[idx]], [0, tpr[idx]], 'k--', color='blue')
-plt.xlim([0.0, 1.0])
-plt.ylim([0.0, 1.05])
-plt.xlabel('False Positive Rate (1 - specificity)', fontsize=14)
-plt.ylabel('True Positive Rate (recall)', fontsize=14)
-plt.title('Receiver operating characteristic (ROC) curve')
-plt.legend(loc="lower right")
-plt.show()
-
-print("Using a threshold of %.3f " % thr[idx] +
-      "guarantees a sensitivity of %.3f " % tpr[idx] +
-      "and a specificity of %.3f" % (1 - fpr[idx]) +
-      ", i.e. a false positive rate of %.2f%%." % (np.array(fpr[idx]) * 100))
-
-# %% [markdown] _cell_guid="6292c3f2-6be9-45e4-be33-1dca41e604a7" _uuid="ba0017b461cea0b8849746e76475598bbba7c9ce"
-# <a id="t4.2.2."></a>
-# ### 4.2.2. Model evaluation based on K-fold cross-validation using `cross_val_score()` function
-
-# %% _cell_guid="32a611ae-b2b7-43e0-8fa8-3cc56e351bf6" _uuid="7f0aba7b861c3fa1748060b4733778851fb00a31"
-# 10-fold cross-validation logistic regression
-logreg = LogisticRegression()
-# Use cross_val_score function
-# We are passing the entirety of X and y, not X_train or y_train, it takes care of splitting the data
-# cv=10 for 10 folds
-# scoring = {'accuracy', 'neg_log_loss', 'roc_auc'} for evaluation metric - althought they are many
-scores_accuracy = cross_val_score(logreg, X, y, cv=10, scoring='accuracy')
-scores_log_loss = cross_val_score(logreg, X, y, cv=10, scoring='neg_log_loss')
-scores_auc = cross_val_score(logreg, X, y, cv=10, scoring='roc_auc')
-print('K-fold cross-validation results:')
-print(logreg.__class__.__name__ +
-      " average accuracy is %2.3f" % scores_accuracy.mean())
-print(logreg.__class__.__name__ +
-      " average log_loss is %2.3f" % -scores_log_loss.mean())
-print(logreg.__class__.__name__ + " average auc is %2.3f" % scores_auc.mean())
-
-# %% [markdown] _cell_guid="bf358561-2762-4b24-b33b-bfda3c5d725f" _uuid="9debd9d4281cf7893159aa2cb1f7993652d60228"
-# <a id="t4.2.3."></a>
-# ### 4.2.3. Model evaluation based on K-fold cross-validation using `cross_validate()` function
-
-# %% _cell_guid="9ea95aac-00b2-413e-ab86-c6b8782c40ef" _uuid="90840c89d42e9284c9480a256dc259778c3a5b1b"
-from sklearn.model_selection import cross_validate
-
-scoring = {
-    'accuracy': 'accuracy',
-    'log_loss': 'neg_log_loss',
-    'auc': 'roc_auc'
-}
-
-modelCV = LogisticRegression()
-
-results = cross_validate(modelCV,
-                         X,
-                         y,
-                         cv=10,
-                         scoring=list(scoring.values()),
-                         return_train_score=False)
-
-print('K-fold cross-validation results:')
-for sc in range(len(scoring)):
-    print(modelCV.__class__.__name__ + " average %s: %.3f (+/-%.3f)" %
-          (list(scoring.keys())[sc], -results['test_%s' % list(scoring.values(
-          ))[sc]].mean() if list(scoring.values())[sc] == 'neg_log_loss' else
-           results['test_%s' % list(scoring.values())[sc]].mean(),
-           results['test_%s' % list(scoring.values())[sc]].std()))
-
-# %% [markdown] _cell_guid="96fc0718-8678-44b5-ba1f-9155303a4ffe" _uuid="c29fd32bbe1fb2740587524d8ddbf08d85832d9c"
-# <font color=bleu>What happens when we add the feature "Fare"?<font>
-
-# %% _cell_guid="80b5d5c3-ead3-48f7-84eb-42c801e0370a" _uuid="0fdf4a257a49b0b47552a4ed307c86852c4ed271"
-cols = [
-    "Age", "Fare", "TravelAlone", "Pclass_1", "Pclass_2", "Embarked_C",
-    "Embarked_S", "Sex_male", "IsMinor"
-]
-X = final_train[cols]
-
-scoring = {
-    'accuracy': 'accuracy',
-    'log_loss': 'neg_log_loss',
-    'auc': 'roc_auc'
-}
-
-modelCV = LogisticRegression()
-
-results = cross_validate(modelCV,
-                         final_train[cols],
-                         y,
-                         cv=10,
-                         scoring=list(scoring.values()),
-                         return_train_score=False)
-
-print('K-fold cross-validation results:')
-for sc in range(len(scoring)):
-    print(modelCV.__class__.__name__ + " average %s: %.3f (+/-%.3f)" %
-          (list(scoring.keys())[sc], -results['test_%s' % list(scoring.values(
-          ))[sc]].mean() if list(scoring.values())[sc] == 'neg_log_loss' else
-           results['test_%s' % list(scoring.values())[sc]].mean(),
-           results['test_%s' % list(scoring.values())[sc]].std()))
-
-# %% [markdown] _cell_guid="e6d3b029-550e-4142-8d9a-c9670b8cd529" _uuid="99b2a4c72f6c50fd11abda7a6463aca5154cacd7"
-# <font color=red>We notice that the model is slightly deteriorated. The "Fare" variable does not carry any useful information. Its presence is just a noise for the logistic regression model.<font>
-
-# %% [markdown] _cell_guid="f485ca4c-2172-4383-979e-21e78a66192c" _uuid="ec44fbaddbc23f03a8ac470391f41ce35f40cf72"
-# <a id="t4.3."></a>
-# ## 4.3. GridSearchCV evaluating using multiple scorers simultaneously
-
-# %% _cell_guid="4a39cb49-b446-4ffa-88a2-e37b627eb5e5" _uuid="765695a9712d3fe1ecff10f17dcc077a80ed7682"
-from sklearn.model_selection import GridSearchCV
-
-X = final_train[Selected_features]
-
-param_grid = {'C': np.arange(1e-05, 3, 0.1)}
-scoring = {
-    'Accuracy': 'accuracy',
-    'AUC': 'roc_auc',
-    'Log_loss': 'neg_log_loss'
-}
-
-gs = GridSearchCV(LogisticRegression(),
-                  return_train_score=True,
-                  param_grid=param_grid,
-                  scoring=scoring,
-                  cv=10,
-                  refit='Accuracy')
-
-gs.fit(X, y)
-results = gs.cv_results_
-
-print('=' * 20)
-print("best params: " + str(gs.best_estimator_))
-print("best params: " + str(gs.best_params_))
-print('best score:', gs.best_score_)
-print('=' * 20)
-
-plt.figure(figsize=(10, 10))
-plt.title("GridSearchCV evaluating using multiple scorers simultaneously",
-          fontsize=16)
-
-plt.xlabel("Inverse of regularization strength: C")
-plt.ylabel("Score")
-plt.grid()
-
-ax = plt.axes()
-ax.set_xlim(0, param_grid['C'].max())
-ax.set_ylim(0.35, 0.95)
-
-# Get the regular numpy array from the MaskedArray
-X_axis = np.array(results['param_C'].data, dtype=float)
-
-for scorer, color in zip(list(scoring.keys()), ['g', 'k', 'b']):
-    for sample, style in (('train', '--'), ('test', '-')):
-        sample_score_mean = -results['mean_%s_%s' % (
-            sample, scorer)] if scoring[scorer] == 'neg_log_loss' else results[
-                'mean_%s_%s' % (sample, scorer)]
-        sample_score_std = results['std_%s_%s' % (sample, scorer)]
-        ax.fill_between(X_axis,
-                        sample_score_mean - sample_score_std,
-                        sample_score_mean + sample_score_std,
-                        alpha=0.1 if sample == 'test' else 0,
-                        color=color)
-        ax.plot(X_axis,
-                sample_score_mean,
-                style,
-                color=color,
-                alpha=1 if sample == 'test' else 0.7,
-                label="%s (%s)" % (scorer, sample))
-
-    best_index = np.nonzero(results['rank_test_%s' % scorer] == 1)[0][0]
-    best_score = -results['mean_test_%s' % scorer][best_index] if scoring[
-        scorer] == 'neg_log_loss' else results['mean_test_%s' %
-                                               scorer][best_index]
-
-    # Plot a dotted vertical line at the best score for that scorer marked by x
-    ax.plot([
-        X_axis[best_index],
-    ] * 2, [0, best_score],
-            linestyle='-.',
-            color=color,
-            marker='x',
-            markeredgewidth=3,
-            ms=8)
-
-    # Annotate the best score for that scorer
-    ax.annotate("%0.2f" % best_score, (X_axis[best_index], best_score + 0.005))
-
-plt.legend(loc="best")
-plt.grid('off')
-plt.show()
-
-# %% [markdown] _cell_guid="4c32848c-e879-4ff0-9da4-7c572e80369e" _uuid="d5e88e25cf05dc5c2ad7dcb574a91115c186047e"
-# <a id="t4.4."></a>
-# ## 4.4. GridSearchCV evaluating using multiple scorers, RepeatedStratifiedKFold and pipeline for preprocessing simultaneously
-#
-# We can applied many tasks together for more in-depth evaluation like gridsearch using cross-validation based on k-folds repeated many times, that can be scaled or no with respect to many scorers and tunning on parameter for a given estimator!
-
-# %% _cell_guid="a777c645-7413-4e1d-855a-0b416be1b48e" _uuid="514d0e3aabe57aaad38d10d6101dbdac5af89ca4"
-from sklearn.preprocessing import StandardScaler
-from sklearn.model_selection import RepeatedStratifiedKFold
-from sklearn.pipeline import Pipeline
-
-#Define simple model
-###############################################################################
-C = np.arange(1e-05, 5.5, 0.1)
-scoring = {
-    'Accuracy': 'accuracy',
-    'AUC': 'roc_auc',
-    'Log_loss': 'neg_log_loss'
-}
-log_reg = LogisticRegression()
-
-#Simple pre-processing estimators
-###############################################################################
-std_scale = StandardScaler(with_mean=False, with_std=False)
-#std_scale = StandardScaler()
-
-#Defining the CV method: Using the Repeated Stratified K Fold
-###############################################################################
-
-n_folds = 5
-n_repeats = 5
-
-rskfold = RepeatedStratifiedKFold(n_splits=n_folds,
-                                  n_repeats=n_repeats,
-                                  random_state=2)
-
-#Creating simple pipeline and defining the gridsearch
-###############################################################################
-
-log_clf_pipe = Pipeline(steps=[('scale', std_scale), ('clf', log_reg)])
-
-log_clf = GridSearchCV(estimator=log_clf_pipe,
-                       cv=rskfold,
-                       scoring=scoring,
-                       return_train_score=True,
-                       param_grid=dict(clf__C=C),
-                       refit='Accuracy')
-
-log_clf.fit(X, y)
-results = log_clf.cv_results_
-
-print('=' * 20)
-print("best params: " + str(log_clf.best_estimator_))
-print("best params: " + str(log_clf.best_params_))
-print('best score:', log_clf.best_score_)
-print('=' * 20)
-
-plt.figure(figsize=(10, 10))
-plt.title("GridSearchCV evaluating using multiple scorers simultaneously",
-          fontsize=16)
-
-plt.xlabel("Inverse of regularization strength: C")
-plt.ylabel("Score")
-plt.grid()
-
-ax = plt.axes()
-ax.set_xlim(0, C.max())
-ax.set_ylim(0.35, 0.95)
-
-# Get the regular numpy array from the MaskedArray
-X_axis = np.array(results['param_clf__C'].data, dtype=float)
-
-for scorer, color in zip(list(scoring.keys()), ['g', 'k', 'b']):
-    for sample, style in (('train', '--'), ('test', '-')):
-        sample_score_mean = -results['mean_%s_%s' % (
-            sample, scorer)] if scoring[scorer] == 'neg_log_loss' else results[
-                'mean_%s_%s' % (sample, scorer)]
-        sample_score_std = results['std_%s_%s' % (sample, scorer)]
-        ax.fill_between(X_axis,
-                        sample_score_mean - sample_score_std,
-                        sample_score_mean + sample_score_std,
-                        alpha=0.1 if sample == 'test' else 0,
-                        color=color)
-        ax.plot(X_axis,
-                sample_score_mean,
-                style,
-                color=color,
-                alpha=1 if sample == 'test' else 0.7,
-                label="%s (%s)" % (scorer, sample))
-
-    best_index = np.nonzero(results['rank_test_%s' % scorer] == 1)[0][0]
-    best_score = -results['mean_test_%s' % scorer][best_index] if scoring[
-        scorer] == 'neg_log_loss' else results['mean_test_%s' %
-                                               scorer][best_index]
-
-    # Plot a dotted vertical line at the best score for that scorer marked by x
-    ax.plot([
-        X_axis[best_index],
-    ] * 2, [0, best_score],
-            linestyle='-.',
-            color=color,
-            marker='x',
-            markeredgewidth=3,
-            ms=8)
-
-    # Annotate the best score for that scorer
-    ax.annotate("%0.2f" % best_score, (X_axis[best_index], best_score + 0.005))
-
-plt.legend(loc="best")
-plt.grid('off')
-plt.show()
-
-# %% _cell_guid="e52c3bfb-4325-4c27-9f8e-5514dae799c2" _uuid="b3d12ca129d816d2434e74ed6574f829f826c3fc"
-final_test['Survived'] = log_clf.predict(final_test[Selected_features])
-final_test['PassengerId'] = test_df['PassengerId']
-
-submission = final_test[['PassengerId', 'Survived']]
-
-submission.to_csv("submission.csv", index=False)
-
-submission.tail()
diff --git a/setup.py b/setup.py
index c0ebe1c..aa6b1c7 100644
--- a/setup.py
+++ b/setup.py
@@ -38,6 +38,8 @@
     'ploomber',
     # to download data for running _kaggle notebooks
     'kaggle',
+    # to fetch from github repo
+    'pygithub',
     # to run some of the examples
     'pandas',
     'scikit-learn',
diff --git a/src/soorgeon/_kaggle.py b/src/soorgeon/_kaggle.py
index c8496d2..55d3ac9 100644
--- a/src/soorgeon/_kaggle.py
+++ b/src/soorgeon/_kaggle.py
@@ -1,7 +1,6 @@
 """
 CLI for downloading Kaggle notebooks for integration testing
 """
-from functools import partial
 import zipfile
 import shutil
 from pathlib import PurePosixPath, Path
@@ -12,28 +11,6 @@
 from kaggle import api
 
 
-def process_index(index):
-    return {PurePosixPath(i['url']).name: _add_partial(i) for i in index}
-
-
-def _add_partial(d):
-    url_data = PurePosixPath(d['data'])
-    is_competition = 'c' in url_data.parts
-    fn = download_from_competition if is_competition else download_from_dataset
-
-    if is_competition:
-        name = url_data.name
-    else:
-        name = str(PurePosixPath(*url_data.parts[-2:]))
-
-    kwargs = dict(name=name)
-
-    if d.get('files'):
-        kwargs['files'] = d['files']
-
-    return {**d, **dict(partial=partial(fn, **kwargs))}
-
-
 def download_from_competition(name, files=None):
     # FIXME: add support for more than one file
     api.competition_download_cli(name, file_name=files)
diff --git a/src/soorgeon/_pygithub.py b/src/soorgeon/_pygithub.py
new file mode 100644
index 0000000..a7834bf
--- /dev/null
+++ b/src/soorgeon/_pygithub.py
@@ -0,0 +1,29 @@
+import base64
+from pathlib import Path
+import shutil
+from github import Github
+# import os
+
+
+def download_directory(dir):
+    """
+    Download all contents from a dir in a repository,
+    store data in a sub_dir called input
+    """
+    g = Github()
+    repo = g.get_repo("ploomber/ci-notebooks")
+    contents = repo.get_contents(dir)
+    Path('input').mkdir()
+
+    for file_content in contents:
+        try:
+            print(f"handling {file_content.path}")
+            file_data = base64.b64decode(file_content.content)
+            file_out = open(file_content.name, "wb")
+            file_out.write(file_data)
+            file_out.close()
+            if file_content.name != 'nb.py':
+                path = Path(file_content.name)
+                shutil.move(path, Path('input', file_content.name))
+        except IOError as exc:
+            print('Error processing %s: %s', file_content.path, exc)


Test patch:
diff --git a/tests/please_delete_me.yaml b/tests/please_delete_me.yaml
new file mode 100644
index 0000000..5706614
--- /dev/null
+++ b/tests/please_delete_me.yaml
@@ -0,0 +1,703 @@
+# Run secret-dependent integration tests only after /ok-to-test approval
+on:
+  pull_request:
+  push:
+  schedule:
+    - cron: '0 0 * * *'
+  repository_dispatch:
+    types: [ok-to-test-command]
+
+name: Integration tests
+
+jobs:
+  # Branch-based pull request
+  integration-trusted-linux:
+    runs-on: ubuntu-latest
+    name: Linux-integration
+    if: (github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository) || github.event_name == 'schedule' || github.event_name == 'push'
+    strategy:
+      matrix:
+        python-version: [3.6, 3.7, 3.8, 3.9]
+    
+    services:
+      postgres:
+        image: postgres
+        env:
+          POSTGRES_PASSWORD: postgres
+        options: >-
+          --health-cmd pg_isready
+          --health-interval 10s
+          --health-timeout 5s
+          --health-retries 5
+        ports:
+          - 5432:5432
+
+    steps:
+
+    - name: Branch based PR checkout
+      uses: actions/checkout@v2
+
+    - name: Set up Python ${{ matrix.python-version }}
+      uses: actions/setup-python@v2
+      with:
+        python-version: ${{ matrix.python-version }}
+
+    - name: Set up R
+      uses: r-lib/actions/setup-r@v2
+
+    - name: Config
+      run: |
+        conda create -q -n conda-env python=${{ matrix.python-version }}
+
+        # for the tests that depend on git repos
+        git config --global user.email "someone@example.com"
+        git config --global user.name "Someone"
+
+    - name: Install dependencies
+      run: |
+
+        eval "$(conda shell.bash hook)"
+        conda activate conda-env
+
+        # make the build fail quickly on flake8 errors
+        pip install flake8
+        flake8
+
+        # for some reason, conda does not resolve the env correctly and
+        # installs and old version of pygraphviz (1.3) which breaks in
+        # Python 3.7
+        conda install -y "pygraphviz>=1.6" -c conda-forge
+
+        # Test vanilla installation, make sure core modules are importable
+        pip install .
+        python -c 'import ploomber'
+        python -c 'import ploomber.tasks'
+        python -c 'import ploomber.products'
+        python -c 'import ploomber.clients'
+        python -c 'import ploomber.io'
+
+        # install dev requirements
+        pip install .[dev]
+
+        # install IR kernel
+        echo "install.packages('IRkernel', repos='http://cran.us.r-project.org'); IRkernel::installspec()" | Rscript -
+
+        # install sample package for tests
+        pip install --editable tests/assets/test_pkg
+
+    - name: Run tests
+      env:
+        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+        PLOOMBER_CLOUD_KEY: ${{ secrets.PLOOMBER_CLOUD_KEY }}
+        PLOOMBER_STATS_ENABLED: false
+      run: |
+        eval "$(conda shell.bash hook)"
+        conda activate conda-env
+
+        # doc tests
+        export PY_IGNORE_IMPORTMISMATCH=1
+        pytest src/ploomber --doctest-modules --ignore src/ploomber/resources
+        unset PY_IGNORE_IMPORTMISMATCH
+
+        pytest tests/cli/test_cloud_execution.py --cov=ploomber 
+        pytest tests/cloud --cov=ploomber
+        pytest tests/telemetry --cov=ploomber
+
+  # Repo owner has commented /ok-to-test on a (fork-based) pull request
+  Linux-integration-forked:
+    runs-on: ubuntu-latest
+    name: Linux-integration-forked
+    if: 
+      github.event_name == 'repository_dispatch' &&
+      github.event.client_payload.slash_command.sha != '' &&
+      contains(github.event.client_payload.pull_request.head.sha, github.event.client_payload.slash_command.sha)
+
+    strategy:
+      matrix:
+        python-version: [3.9]
+    
+    services:
+      postgres:
+        image: postgres
+        env:
+          POSTGRES_PASSWORD: postgres
+        options: >-
+          --health-cmd pg_isready
+          --health-interval 10s
+          --health-timeout 5s
+          --health-retries 5
+        ports:
+          - 5432:5432
+
+    steps:
+
+    # Check out merge commit
+    - name: Fork based /ok-to-test checkout
+      uses: actions/checkout@v2
+      with:
+        ref: 'refs/pull/${{ github.event.client_payload.pull_request.number }}/merge'
+
+    - name: Set up Python ${{ matrix.python-version }}
+      uses: actions/setup-python@v2
+      with:
+        python-version: ${{ matrix.python-version }}
+
+    - name: Set up R
+      uses: r-lib/actions/setup-r@v2
+
+    - name: Config
+      run: |
+        conda create -q -n conda-env python=${{ matrix.python-version }}
+
+        # for the tests that depend on git repos
+        git config --global user.email "someone@example.com"
+        git config --global user.name "Someone"
+
+    - name: Install dependencies
+      run: |
+
+        eval "$(conda shell.bash hook)"
+        conda activate conda-env
+
+        # make the build fail quickly on flake8 errors
+        pip install flake8
+        flake8
+
+        # for some reason, conda does not resolve the env correctly and
+        # installs and old version of pygraphviz (1.3) which breaks in
+        # Python 3.7
+        conda install -y "pygraphviz>=1.6" -c conda-forge
+
+        # Test vanilla installation, make sure core modules are importable
+        pip install .
+        python -c 'import ploomber'
+        python -c 'import ploomber.tasks'
+        python -c 'import ploomber.products'
+        python -c 'import ploomber.clients'
+        python -c 'import ploomber.io'
+
+        # install dev requirements
+        pip install .[dev]
+
+        # install IR kernel
+        echo "install.packages('IRkernel', repos='http://cran.us.r-project.org'); IRkernel::installspec()" | Rscript -
+
+        # install sample package for tests
+        pip install --editable tests/assets/test_pkg
+
+    - name: Run tests
+      id: test-linux-fork
+      env:
+        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+        PLOOMBER_CLOUD_KEY: ${{ secrets.PLOOMBER_CLOUD_KEY }}
+        PLOOMBER_STATS_ENABLED: false
+      run: |
+        eval "$(conda shell.bash hook)"
+        conda activate conda-env
+
+        # doc tests
+        export PY_IGNORE_IMPORTMISMATCH=1
+        pytest src/ploomber --doctest-modules --ignore src/ploomber/resources
+        unset PY_IGNORE_IMPORTMISMATCH
+
+        # echo "::set-output name=PYTEST_LOG::$(pytest tests/cli/test_cloud_execution.py --cov=ploomber | pytest tests/cloud --cov=ploomber | pytest tests/telemetry --cov=ploomber)"
+        
+        pytest tests/cli/test_cloud_execution.py --cov=ploomber 
+        pytest tests/cloud --cov=ploomber
+        pytest tests/telemetry --cov=ploomber
+  
+    - run: |
+        echo "Integration tests... success! ;-)"
+
+    # Update check run called "integration-fork"
+    - uses: actions/github-script@v5
+      # id: update-check-run
+      if: ${{ always() }}
+      env:
+        number: ${{ github.event.client_payload.pull_request.number }}
+        job: ${{ github.job }}
+        # Conveniently, job.status maps to https://developer.github.com/v3/checks/runs/#update-a-check-run
+        conclusion: ${{ job.status }} 
+      with:
+        github-token: ${{ secrets.GITHUB_TOKEN }}
+        script: |
+          const { data: pull } = await github.rest.pulls.get({
+            ...context.repo,
+            pull_number: process.env.number
+          });
+          const ref = pull.head.sha;
+
+          const { data: checks } = await github.rest.checks.listForRef({
+            ...context.repo,
+            ref
+          });
+          console.log('checks')
+          console.log(checks)
+          console.log('================================================')
+          const check = checks.check_runs.filter(c => c.name === process.env.job);
+          console.log(check)
+          console.log('================================================')
+          console.log(process.env.job)
+          console.log('================================================')
+          console.log('Hello!')
+          console.log(${{ steps.test-linux-fork.outputs.value }})
+          console.log('================================================')
+          const { data: result } = await github.rest.checks.update({
+            ...context.repo,
+            check_run_id: check[0].id,
+            status: 'completed',
+            conclusion: 'success',
+            output: ${{ steps.test-linux-fork.outputs }}
+          });
+
+          return result;
+
+
+  integration-trusted-macos:
+    runs-on: macos-latest
+    name: Macos-integration
+    strategy:
+      matrix:
+        python-version: [3.6, 3.7, 3.8, 3.9]
+    if: (github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository) || github.event_name == 'schedule' || github.event_name == 'push'
+    steps:
+    - name: Start PostgreSQL
+      run: |
+        brew services start postgresql
+        echo "Check PostgreSQL service is running"
+        i=10
+        COMMAND='pg_isready'
+        while [ $i -gt 0 ]; do
+            echo "Check PostgreSQL service status"
+            eval $COMMAND && break
+            ((i--))
+            if [ $i == 0 ]; then
+                echo "PostgreSQL service not ready, all attempts exhausted"
+                exit 1
+            fi
+            echo "PostgreSQL service not ready, wait 10 more sec, attempts left: $i"
+            sleep 10
+        done
+
+    - name: Create PostgreSQL user
+      run: psql --command="CREATE USER postgres PASSWORD 'postgres'" postgres
+
+    - name: Update PostgreSQL permissions
+      run: psql --command="ALTER DATABASE postgres OWNER TO postgres" postgres
+
+    - name: Check PostgreSQL authentication
+      run: PGPASSWORD=postgres pg_isready --dbname postgres --username postgres
+
+    - name: Branch based PR checkout
+      uses: actions/checkout@v2
+
+    - name: Set up Python ${{ matrix.python-version }}
+      uses: conda-incubator/setup-miniconda@v2
+      with:
+        miniconda-version: "latest"
+        activate-environment: conda-env
+        python-version: ${{ matrix.python-version }}
+
+    - name: Set up R
+      uses: r-lib/actions/setup-r@v2
+
+    - name: Install dependencies
+      run: |
+
+        eval "$(conda shell.bash hook)"
+        conda activate conda-env
+
+        # make the build fail quickly on flake8 errors
+        pip install flake8
+        flake8
+
+        conda install -y pygraphviz -c conda-forge
+
+        # Test vanilla installation, make sure core modules are importable
+        # note: I needed to add "--ignore-installed certifi" because the
+        # installation started failing suddenly
+        pip install .  --ignore-installed certifi
+        python -c 'import ploomber'
+        python -c 'import ploomber.tasks'
+        python -c 'import ploomber.products'
+        python -c 'import ploomber.clients'
+
+        # install dev requirements
+        pip install .[dev]
+
+        # install IR kernel
+        echo "install.packages('IRkernel', repos='http://cran.us.r-project.org'); IRkernel::installspec()" | Rscript -
+
+        # install sample package for tests
+        pip install --editable tests/assets/test_pkg
+
+    - name: Run tests
+      env:
+        PLOOMBER_STATS_ENABLED: false
+        PLOOMBER_CLOUD_KEY: ${{ secrets.PLOOMBER_CLOUD_KEY }}
+      run: |
+        eval "$(conda shell.bash hook)"
+        conda activate conda-env
+        pytest tests/cli/test_cloud_execution.py
+        pytest tests/cloud
+        pytest tests/telemetry
+
+  # Repo owner has commented /ok-to-test on a (fork-based) pull request
+  integration-fork-macos:
+    runs-on: macos-latest
+    name: Macos-integration-forked
+    strategy:
+      matrix:
+        python-version: [3.6, 3.7, 3.8, 3.9]
+    if: 
+      github.event_name == 'repository_dispatch' &&
+      github.event.client_payload.slash_command.sha != '' &&
+      contains(github.event.client_payload.pull_request.head.sha, github.event.client_payload.slash_command.sha)
+    steps:
+    - name: Start PostgreSQL
+      run: |
+        brew services start postgresql
+        echo "Check PostgreSQL service is running"
+        i=10
+        COMMAND='pg_isready'
+        while [ $i -gt 0 ]; do
+            echo "Check PostgreSQL service status"
+            eval $COMMAND && break
+            ((i--))
+            if [ $i == 0 ]; then
+                echo "PostgreSQL service not ready, all attempts exhausted"
+                exit 1
+            fi
+            echo "PostgreSQL service not ready, wait 10 more sec, attempts left: $i"
+            sleep 10
+        done
+
+    - name: Create PostgreSQL user
+      run: psql --command="CREATE USER postgres PASSWORD 'postgres'" postgres
+
+    - name: Update PostgreSQL permissions
+      run: psql --command="ALTER DATABASE postgres OWNER TO postgres" postgres
+
+    - name: Check PostgreSQL authentication
+      run: PGPASSWORD=postgres pg_isready --dbname postgres --username postgres
+
+    # Check out merge commit
+    - name: Fork based /ok-to-test checkout
+      uses: actions/checkout@v2
+      with:
+        ref: 'refs/pull/${{ github.event.client_payload.pull_request.number }}/merge'
+
+    - name: Set up Python ${{ matrix.python-version }}
+      uses: conda-incubator/setup-miniconda@v2
+      with:
+        miniconda-version: "latest"
+        activate-environment: conda-env
+        python-version: ${{ matrix.python-version }}
+
+    - name: Set up R
+      uses: r-lib/actions/setup-r@v2
+
+    - name: Install dependencies
+      run: |
+
+        eval "$(conda shell.bash hook)"
+        conda activate conda-env
+
+        # make the build fail quickly on flake8 errors
+        pip install flake8
+        flake8
+
+        conda install -y pygraphviz -c conda-forge
+
+        # Test vanilla installation, make sure core modules are importable
+        # note: I needed to add "--ignore-installed certifi" because the
+        # installation started failing suddenly
+        pip install .  --ignore-installed certifi
+        python -c 'import ploomber'
+        python -c 'import ploomber.tasks'
+        python -c 'import ploomber.products'
+        python -c 'import ploomber.clients'
+
+        # install dev requirements
+        pip install .[dev]
+
+        # install IR kernel
+        echo "install.packages('IRkernel', repos='http://cran.us.r-project.org'); IRkernel::installspec()" | Rscript -
+
+        # install sample package for tests
+        pip install --editable tests/assets/test_pkg
+
+    - name: Run tests
+      env:
+        PLOOMBER_STATS_ENABLED: false
+        PLOOMBER_CLOUD_KEY: ${{ secrets.PLOOMBER_CLOUD_KEY }}
+      run: |
+        eval "$(conda shell.bash hook)"
+        conda activate conda-env
+        pytest tests/cli/test_cloud_execution.py
+        pytest tests/cloud
+        pytest tests/telemetry
+
+    - run: |
+        echo "Integration tests... success! ;-)"
+
+    # Update check run called "integration-fork"
+    - uses: actions/github-script@v5
+      # id: update-check-run
+      if: ${{ always() }}
+      env:
+        number: ${{ github.event.client_payload.pull_request.number }}
+        job: ${{ github.job }}
+        # Conveniently, job.status maps to https://developer.github.com/v3/checks/runs/#update-a-check-run
+        conclusion: ${{ job.status }} 
+      with:
+        github-token: ${{ secrets.GITHUB_TOKEN }}
+        script: |
+          const { data: pull } = await github.rest.pulls.get({
+            ...context.repo,
+            pull_number: process.env.number
+          });
+          const ref = pull.head.sha;
+
+          const { data: checks } = await github.rest.checks.listForRef({
+            ...context.repo,
+            ref
+          });
+
+          const check = checks.check_runs.filter(c => c.name === process.env.job);
+
+          const { data: result } = await github.rest.checks.update({
+            ...context.repo,
+            check_run_id: check[0].id,
+            status: 'completed',
+            conclusion: process.env.conclusion
+          });
+
+          return result;
+
+
+  integration-trusted-windows:
+    runs-on: windows-latest
+    name: Windows-integration
+    strategy:
+      matrix:
+        python-version: [3.7, 3.8, 3.9]
+
+    if: (github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository) || github.event_name == 'schedule' || github.event_name == 'push'
+    steps:
+
+    - name: Start PostgreSQL
+      run: |
+        $pgService = Get-Service -Name postgresql*
+        Set-Service -InputObject $pgService -Status running -StartupType automatic
+        Start-Process -FilePath "$env:PGBIN\pg_isready" -Wait -PassThru
+    - name: Change PostgreSQL password
+      run: |
+        & $env:PGBIN\psql --command  "ALTER USER postgres PASSWORD 'postgres'"
+    - name: Check PostgreSQL authentication
+      run: |
+        $env:PGPASSWORD = 'postgres'
+        & $env:PGBIN\pg_isready --dbname postgres --username postgres
+
+    - name: Branch based PR checkout
+      uses: actions/checkout@v2
+
+    - name: Set up Python ${{ matrix.python-version }}
+      uses: conda-incubator/setup-miniconda@v2
+      with:
+        miniconda-version: "latest"
+        activate-environment: conda-env
+        python-version: ${{ matrix.python-version }}
+    - name: Install dependencies
+      shell: bash -l {0}
+      run: |
+
+        eval "$(conda shell.bash hook)"
+        conda activate conda-env
+
+        # make the build fail quickly on flake8 errors
+        pip install flake8
+        flake8
+
+        # there's a bug in jupyter, we need to install this first
+        conda install pywin32
+
+        # install graphviz (pygraphviz not available in conda for win-64,
+        # it will be installed from pip)
+        # conda install -y graphviz -c conda-forge
+        # note: this installation still gives trouble (error when saving png
+        # files but it makes pygraphviz importable), we can get rid of this
+        # once we update the tests to mock the @requires decorator to make
+        # it think that pygraphviz is installed when it's not. the tests
+        # currently do not call pygraphviz (it's mocked), but @requires imports
+        conda install -y -c alubbock pygraphviz 
+        # seems like pygraphviz just doesnt work on windows
+        # https://github.com/pygraphviz/pygraphviz/issues/40
+        # pip install pygraphviz
+
+        # Test vanilla installation, make sure core modules are importable
+        pip install .
+        python -c 'import ploomber'
+        python -c 'import ploomber.tasks'
+        python -c 'import ploomber.products'
+        python -c 'import ploomber.clients'
+
+        # install dev requirements
+        pip install .[dev]
+
+        # https://github.com/ipython/ipython/issues/12049
+        pip install prompt-toolkit==2.0.10
+
+        # install R and IR kernel
+        conda install -y r-base r-irkernel -c conda-forge
+        echo "IRkernel::installspec()" | Rscript -
+
+        # install sample package for tests
+        pip install --editable tests/assets/test_pkg
+
+    - name: Run tests
+      env:
+        PLOOMBER_STATS_ENABLED: false
+        PLOOMBER_CLOUD_KEY: ${{ secrets.PLOOMBER_CLOUD_KEY }}
+      shell: bash -l {0}
+      run: |
+        eval "$(conda shell.bash hook)"
+        conda activate conda-env
+        pytest tests/cli/test_cloud_execution.py
+        pytest tests/cloud
+        pytest tests/telemetry
+
+  # Repo owner has commented /ok-to-test on a (fork-based) pull request
+  integration-fork-windows:
+    runs-on: windows-latest
+    name: Windows-integration-forked
+    strategy:
+      matrix:
+        python-version: [3.7, 3.8, 3.9]
+
+    if: 
+      github.event_name == 'repository_dispatch' &&
+      github.event.client_payload.slash_command.sha != '' &&
+      contains(github.event.client_payload.pull_request.head.sha, github.event.client_payload.slash_command.sha)
+    steps:
+
+    # Check out merge commit
+    - name: Fork based /ok-to-test checkout
+      uses: actions/checkout@v2
+      with:
+        ref: 'refs/pull/${{ github.event.client_payload.pull_request.number }}/merge'
+
+    - name: Start PostgreSQL
+      run: |
+        $pgService = Get-Service -Name postgresql*
+        Set-Service -InputObject $pgService -Status running -StartupType automatic
+        Start-Process -FilePath "$env:PGBIN\pg_isready" -Wait -PassThru
+    - name: Change PostgreSQL password
+      run: |
+        & $env:PGBIN\psql --command  "ALTER USER postgres PASSWORD 'postgres'"
+    - name: Check PostgreSQL authentication
+      run: |
+        $env:PGPASSWORD = 'postgres'
+        & $env:PGBIN\pg_isready --dbname postgres --username postgres
+
+    - name: Set up Python ${{ matrix.python-version }}
+      uses: conda-incubator/setup-miniconda@v2
+      with:
+        miniconda-version: "latest"
+        activate-environment: conda-env
+        python-version: ${{ matrix.python-version }}
+    - name: Install dependencies
+      shell: bash -l {0}
+      run: |
+
+        eval "$(conda shell.bash hook)"
+        conda activate conda-env
+
+        # make the build fail quickly on flake8 errors
+        pip install flake8
+        flake8
+
+        # there's a bug in jupyter, we need to install this first
+        conda install pywin32
+
+        # install graphviz (pygraphviz not available in conda for win-64,
+        # it will be installed from pip)
+        # conda install -y graphviz -c conda-forge
+        # note: this installation still gives trouble (error when saving png
+        # files but it makes pygraphviz importable), we can get rid of this
+        # once we update the tests to mock the @requires decorator to make
+        # it think that pygraphviz is installed when it's not. the tests
+        # currently do not call pygraphviz (it's mocked), but @requires imports
+        conda install -y -c alubbock pygraphviz 
+        # seems like pygraphviz just doesnt work on windows
+        # https://github.com/pygraphviz/pygraphviz/issues/40
+        # pip install pygraphviz
+
+        # Test vanilla installation, make sure core modules are importable
+        pip install .
+        python -c 'import ploomber'
+        python -c 'import ploomber.tasks'
+        python -c 'import ploomber.products'
+        python -c 'import ploomber.clients'
+
+        # install dev requirements
+        pip install .[dev]
+
+        # https://github.com/ipython/ipython/issues/12049
+        pip install prompt-toolkit==2.0.10
+
+        # install R and IR kernel
+        conda install -y r-base r-irkernel -c conda-forge
+        echo "IRkernel::installspec()" | Rscript -
+
+        # install sample package for tests
+        pip install --editable tests/assets/test_pkg
+
+    - name: Run tests
+      env:
+        PLOOMBER_STATS_ENABLED: false
+        PLOOMBER_CLOUD_KEY: ${{ secrets.PLOOMBER_CLOUD_KEY }}
+      shell: bash -l {0}
+      run: |
+        eval "$(conda shell.bash hook)"
+        conda activate conda-env
+        pytest tests/cli/test_cloud_execution.py
+        pytest tests/cloud
+        pytest tests/telemetry
+
+    - run: |
+        echo "Integration tests... success! ;-)"
+
+    # Update check run called "integration-fork"
+    - uses: actions/github-script@v5
+      # id: update-check-run
+      if: ${{ always() }}
+      env:
+        number: ${{ github.event.client_payload.pull_request.number }}
+        job: ${{ github.job }}
+        # Conveniently, job.status maps to https://developer.github.com/v3/checks/runs/#update-a-check-run
+        conclusion: ${{ job.status }} 
+      with:
+        github-token: ${{ secrets.GITHUB_TOKEN }}
+        script: |
+          const { data: pull } = await github.rest.pulls.get({
+            ...context.repo,
+            pull_number: process.env.number
+          });
+          const ref = pull.head.sha;
+
+          const { data: checks } = await github.rest.checks.listForRef({
+            ...context.repo,
+            ref
+          });
+
+          const check = checks.check_runs.filter(c => c.name === process.env.job);
+
+          const { data: result } = await github.rest.checks.update({
+            ...context.repo,
+            check_run_id: check[0].id,
+            status: 'completed',
+            conclusion: process.env.conclusion
+          });
+
+          return result;
\ No newline at end of file
diff --git a/tests/test_sample_notebooks.py b/tests/test_sample_notebooks.py
index 03c24ec..24590df 100644
--- a/tests/test_sample_notebooks.py
+++ b/tests/test_sample_notebooks.py
@@ -1,27 +1,26 @@
+import os
 from pathlib import Path
 
-import yaml
+
 import pytest
 from ploomber.spec import DAGSpec
-from conftest import PATH_TO_TESTS
 
 from soorgeon import export
-from soorgeon._kaggle import process_index
-
-_kaggle = Path(PATH_TO_TESTS, '..', '_kaggle')
-path_to_index = _kaggle / 'index.yaml'
-index_raw = yaml.safe_load(path_to_index.read_text())
+from soorgeon._pygithub import download_directory
 
-index = process_index(index_raw)
-path_to_nbs = [_kaggle / name for name in index]
 
+dir_names = ['titanic-logistic-regression-with-python',
+             'customer-segmentation-clustering',
+             'intro-to-time-series-forecasting',
+             'feature-selection-and-data-visualization',
+             'linear-regression-house-price-prediction',
+             'look-at-this-note-feature-engineering-is-easy']
 
-@pytest.mark.parametrize('path', path_to_nbs, ids=list(index))
-def test_notebooks(tmp_empty, path):
-    name = Path(path).name
-    download_fn = index[name]['partial']
-    download_fn()
 
+@pytest.mark.parametrize('dir', dir_names, ids=list(dir_names))
+def test_notebooks(tmp_empty, dir):
+    download_directory(dir)
+    path = os.getcwd()
     export.from_path(Path(path, 'nb.py'), py=True)
 
     dag = DAGSpec('pipeline.yaml').to_dag()