아래는 어떤 과제인가?
{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 0 - Introduction\n",
    "\n",
    "Welcome to the <span style=\"color:yellowgreen\">Foundations of Machine Learning</span> (ECE5984_41) course!\n",
    "\n",
    "This is the <span style=\"color:red\">2nd</span> lab practice. From now on, you will going to implement logistic regression and L2 regularization. \n",
    "\n",
    "### Outline\n",
    "- [ 1 - Packages ](#1-packages)\n",
    "- [ 2 - Problem Statement and Dataset ](#2-problem-statement-and-dataset)\n",
    "- [ 3 - Logistic Regression ](#3-logistic-regression)\n",
    "  - [ 3.1 Sigmoid function ](#31-sigmoid-function) - Exercise 1\n",
    "  - [ 3.2 Binary Cross Entropy Loss](#32-binary-cross-entropy-loss) - Exercise 2\n",
    "  - [ 3.3 Gradient Descent ](#33-gradient-descent) - Exercise 3, Exercise 4\n",
    "- [ 4 - L2 Regularization ](#4-l2-regularization) - Exercise 5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Packages \n",
    "\n",
    "You have to install and use below packages for HW#2.\n",
    "- [numpy](https://www.numpy.org): Fundamental package for matrix computation with python.\n",
    "- [matplotlib](https://matplotlib.org): Package for visualization of the graph with python.\n",
    "- [pandas](https://pandas.pydata.org): Open source package for data analysis and manipulation.\n",
    "\n",
    "**Do not use other machine learning packages in this homework, e.g., *tensorflow*, *pytorch*, *jax*, etc.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import math\n",
    "import copy\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Problem Statement and Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Problem Statement\n",
    "\n",
    "[Fake Bills Dataset](https://www.kaggle.com/code/unmoved/classify-fake-bills)\n",
    "\n",
    "The *Fake Bills Dataset* consists \n",
    "The dataset consists 1500 samples of bills. Some of the bills are real, and some are fake.\n",
    "\n",
    "In this homework, we would like to fit the logistic regression model on this dataset to determine if the given bill is fake or real!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dataset\n",
    "\n",
    "You will start by loading the dataset. Make sure the attached 'fake_bills.csv' file is in your current directory(can be downloaded from icampus).\n",
    "\n",
    "Each column describes the below attributes:\n",
    "- \\[ is_genuine \\]: if the given bill is real or not.\n",
    "- \\[ diagonal \\]: the diagonal of the banknote in mm.\n",
    "- \\[ height_left \\]: the height of the left side of the banknote in mm.\n",
    "- \\[ height_right \\]: the height of the right side of the banknote in mm.\n",
    "- \\[ margin_low \\]: lower side margin of the banknote in mm.\n",
    "- \\[ margin_up \\]: upper side margin of the banknote in mm.\n",
    "- \\[ length \\]: the length of the banknote in mm.\n",
    "\n",
    "Run below cells and take a look of the dataset!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load the dataset\n",
    "dataset = pd.read_csv(\"./fake_bills.csv\")\n",
    "\n",
    "# Display top 5 elements\n",
    "dataset.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# check dataset info\n",
    "dataset.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this lab practice, we will not going to use \"margin_low\" and \"margin_up\" features. \n",
    "\n",
    "(There are some missing-value cells on those attributes. We can use mean value or apply knn algorithm to fill in those cells, but in this lab practice, let's just simply drop those attributes.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset.drop(labels=['margin_low', 'margin_up'], axis=1, inplace=True)\n",
    "dataset.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Seperate train data and label from the dataset.\n",
    "data = dataset.drop(labels=['is_genuine'], axis=1, inplace=False).to_numpy()\n",
    "label = dataset[\"is_genuine\"].to_numpy()\n",
    "label = label.astype(float)\n",
    "\n",
    "# Check data statistics\n",
    "print(\"The shape of data is\", data.shape)\n",
    "print(\"The shape of label is\", label.shape)\n",
    "print()\n",
    "\n",
    "print(\"The first Five elements of data are\", data[:5])\n",
    "print(\"The first Five elements of label are\", label[:5])\n",
    "print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Normalizing data equalizes the scale of each features, which encourages fast and stable convergence for gradient descent. \n",
    "\n",
    "Run below code and normalize your data!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Normalize the data\n",
    "data = (data - np.mean(data, axis=0)) / np.std(data, axis=0)\n",
    "print(\"The first Five elements of data are\", data[:5])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Visualize two features (1st & 2nd dimension of the feature)\n",
    "real = (label == 1)\n",
    "fake = (label == 0)\n",
    "\n",
    "plt.scatter(data[real, 0], data[real, 1], marker='o', c='g', label=\"real\")\n",
    "plt.scatter(data[fake, 0], data[fake, 1], marker='x', c='r', label=\"fake\")\n",
    "plt.xlabel(\"Feature 1\")\n",
    "plt.ylabel(\"Feature 2\")\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Logistic Regression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.1 Sigmoid Function\n",
    "\n",
    "\$\$\\sigma(x) = \\frac{1}{1+e^{-x}}\$\$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\> In Exercise 1, you will implement the logistic function, i.e., sigmoid function. Below <span style=\"color:skyblue\">sigmoid()</span> returns sigmoid function value of given input x.\n",
    "\n",
    "Do not change the method name and its argument."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <span style=\"color:#ffd33d\"> Exercise 1 </span>: Write the code for sigmoid function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Complete the code for sigmoid function\n",
    "def sigmoid(x):\n",
    "    \"\"\"\n",
    "    compute the sigmoid of x.\n",
    "    \n",
    "    Args:\n",
    "        x    (np.ndarray): Shape (N, )\n",
    "    \n",
    "    Return:\n",
    "        sigma    (np.ndarray): Shape (N, ) Sigmoid values of given input x\n",
    "    \"\"\"\n",
    "    sigma = 0\n",
    "    return sigma\n",
    "\n",
    "x = np.array([-1, 0, 1])\n",
    "sigmoid(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.2 Binary Cross-Entropy Loss\n",
    "\n",
    "\$\$\\hat{y}^{(i)} = \\sigma(w^T x^{(i)} + b)\$\$\n",
    "\n",
    "\$\$BCE(w, b) = - \\sum_{i=1}^{N} y^{(i)} \\cdot log(\\hat{y}^{(i)}) + (1-y^{(i)}) \\cdot log(1-\\hat{y}^{(i)})\$\$\n",
    "\n",
    "* N is the number of training examples in the dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <span style=\"color:#ffd33d\"> Exercise 2 </span>: Complete the code for BCELoss"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\> Here, the <span style=\"color:skyblue\">compute_bceloss()</span> method computes the binary cross-entropy loss of the logistic regression model. \n",
    "\n",
    "Feel free to change or add variables, but do not modify the method name and its arguements."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def compute_bceloss(data, label, w, b):\n",
    "    \"\"\"\n",
    "    Compute the bceloss of the logistic regression model, given data and label.\n",
    "\n",
    "    Args:\n",
    "        data    (np.ndarray): Shape (N, ) Input to the model\n",
    "        label   (np.ndarray): Shape (N, ) Label of the data\n",
    "        w       (np.ndarray): Weight of the logistic regression model\n",
    "        b       (float): Bias of the logistic regression model\n",
    "    \n",
    "    Return:\n",
    "        total_loss (float): Total bceloss of the logistic regression model, given data and label\n",
    "    \"\"\"\n",
    "    total_loss = 0\n",
    "\n",
    "    #[NOTE] write your code here!\n",
    "\n",
    "    return total_loss\n",
    "\n",
    "w = np.zeros(4)\n",
    "b = 0.0\n",
    "\n",
    "compute_bceloss(data, label, w, b)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.3 Gradient Descent\n",
    "\n",
    "\$\$\n",
    "\\frac{\\partial BCE(\\mathbf{w},b)}{\\partial w_j}  = \\sum\\limits_{i = 1}^{N} (\\hat{y}^{(i)} - {y}^{(i)})x_{j}^{(i)}\n",
    "\$\$\n",
    "\$\$\n",
    "\\frac{\\partial BCE(\\mathbf{w},b)}{\\partial b}  = \\sum\\limits_{i = 1}^{N} (\\hat{y}^{(i)} - {y}^{(i)})\n",
    "\$\$\n",
    "* N is the number of training examples in the dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <span style=\"color:#ffd33d\"> Exercise 3 </span>: Complete the code for computing the gradient of the loss w.r.t weights and bias"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\> Here, the <span style=\"color:skyblue\">compute_gradient()</span> method computes the gradient of the loss with respect to the parameters of the logistic regression model.\n",
    "\n",
    "Feel free to change or add variables, but do not modify the method name and its arguments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def compute_gradient(data, label, w, b):\n",
    "    \"\"\"\n",
    "    Compute the gradient of the loss w.r.t weight & bias of model, given data and label.\n",
    "\n",
    "    Args:\n",
    "        data    (np.ndarray): Shape (N, ) Input to the model\n",
    "        label   (np.ndarray): Shape (N, ) Label of the data\n",
    "        w       (np.ndarray): Weight of the logistic regression model\n",
    "        b       (float): Bias of the logistic regression model\n",
    "    \n",
    "    Return:\n",
    "       grad_w   (np.ndarray): The gradient of the loss w.r.t weight w\n",
    "       grad_b   (float): The gradient of the loss w.r.t bias b\n",
    "    \"\"\"\n",
    "    grad_w, grad_b = 0, 0\n",
    "\n",
    "    #[NOTE] write your code here!\n",
    "\n",
    "    return grad_w, grad_b\n",
    "\n",
    "compute_gradient(data, label, w, b)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <span style=\"color:#ffd33d\"> Exercise 4 </span>: Complete the code of gradient descent algorithm"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\> <span style=\"color:skyblue\">gradient_descent()</span> method applies gradient descent on logistic regression model on the given dataset. You should make 'loop' inside this method to iteratively update w and b. You __**must**__ use <span style=\"color:skyblue\">compute_bceloss()</span> and <span style=\"color:skyblue\">compute_gradient()</span> method to calculate bceloss and gradients, respectively.\n",
    "\n",
    "Also, for the purpose of visualization of the training curve, this method returns some 'history' lists. You can just simply append updated values to the corresponding 'history' list.\n",
    "\n",
    "Feel free to change or add variables, but do not modify the method name, its arguments, and return values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def gradient_descent(data, label, w_init, b_init, iters=1500, lr=0.0001):\n",
    "    \"\"\"\n",
    "    Performs batch gradient descent to obatain weight and bias of the logistic regression model.\n",
    "    \n",
    "    Args:\n",
    "      data    (np.ndarray): Shape (N,)\n",
    "      label   (np.ndarray): Shape (N,)\n",
    "      w_init  (np.ndarray): Initial value of weight of the model\n",
    "      b_init  (float): Initial values of bias of the model\n",
    "      lr      (float): Learning rate\n",
    "      iters   (int): Number of iterations to run gradient descent\n",
    "    Returns\n",
    "      w             (np.ndarray): Weight of the logistic regression model obtained with BGD\n",
    "      b             (float): Bias of the logistic regression model obtained with BGD\n",
    "      loss_history  (list): loss values of every iteration steps\n",
    "      w_history     (list): w values of every iteration steps\n",
    "      b_history     (list): b values of every iteration steps\n",
    "    \"\"\"\n",
    "    loss_history = []\n",
    "    w_history = []\n",
    "    b_history = []\n",
    "    w = w_init\n",
    "    b = b_init\n",
    "\n",
    "    for i in range(iters):\n",
    "        #[NOTE] write your code here!\n",
    "\n",
    "        # print loss for every 100 iterations\n",
    "        if i % 100 == 99:\n",
    "            print(f\"[ {i + 1:4}/{iters} ] BCELoss: {loss:.4f}\")\n",
    "    \n",
    "    return w, b, loss_history, w_history, b_history"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now apply your gradient descent algorithm!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# initial values of w and b\n",
    "w_init, b_init = np.random.randn(4), .0\n",
    "\n",
    "# hyperparameters for gradient descent algorithm\n",
    "iters = 1500\n",
    "learning_rate = 0.0001\n",
    "\n",
    "# obtain w and b with gradient descent\n",
    "w, b, loss_history, w_history, b_history = gradient_descent(\n",
    "                                                data, label, w_init, b_init,\n",
    "                                                iters=iters,\n",
    "                                                lr=learning_rate,\n",
    "                                            )\n",
    "\n",
    "print(\"w,b found by gradient descent:\", w, b)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot training curve!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "loss_history = loss_history\n",
    "\n",
    "# plot training curve\n",
    "plt.plot(range(len(loss_history)), loss_history)\n",
    "plt.title(\"Training Curve\")\n",
    "plt.xlabel(\"Iter\")\n",
    "plt.ylabel(\"Loss\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot decision boundary of your logistic regression model!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_decision_boundary(data, label, w, b, feature_dims=(0, 1)):\n",
    "    # scatter data along the feature_dims\n",
    "    dim_1, dim_2 = feature_dims\n",
    "\n",
    "    real = (label == 1)\n",
    "    fake = (label == 0)\n",
    "\n",
    "    plt.scatter(data[real, dim_1], data[real, dim_2], marker='o', c='g', label=\"real\")\n",
    "    plt.scatter(data[fake, dim_1], data[fake, dim_2], marker='x', c='r', label=\"fake\")\n",
    "    plt.xlabel(f'Feature {dim_1 + 1}')\n",
    "    plt.ylabel(f'Feature {dim_2 + 1}')\n",
    "\n",
    "    db_x = np.array([min(data[:, dim_1]), max(data[:, dim_1])])\n",
    "    db_y = ((-1. / w[dim_2]) * (w[dim_1] * db_x + b))\n",
    "\n",
    "    plt.plot(db_x, db_y)\n",
    "\n",
    "# feel free to change 'feature_dims' argument to the other feature dimension indices,\n",
    "#   and check if your model fits well on the dataset!\n",
    "plot_decision_boundary(data, label, w, b, feature_dims=(0, 3))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. L2 Regularization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\$\$\\hat{y}^{(i)} = \\sigma(w^T x^{(i)} + b)\$\$\n",
    "\n",
    "\$\$BCE(w, b) = - \\sum_{i=1}^{N} y^{(i)} \\cdot log(\\hat{y}^{(i)}) + (1-y^{(i)}) \\cdot log(1-\\hat{y}^{(i)})\$\$\n",
    "\n",
    "\$\$L(w, b) = BCE(w, b) + \\frac{\\lambda}{2} \\| w\\|^2\$\$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <span style=\"color:#ffd33d\"> Exercise 5 </span>: Complete the below codes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\> Here, the <span style=\"color:skyblue\">compute_l2_reg_loss()</span> method computes the l2 regularized loss of the logistic regression model.\n",
    "\n",
    "Feel free to change or add variables, but do not modify the method name and its arguements."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def compute_l2_reg_loss(data, label, w, b, l2_lambda=0.5):\n",
    "    \"\"\"\n",
    "    Compute the bceloss of the logistic regression model, given data and label.\n",
    "\n",
    "    Args:\n",
    "        data    (np.ndarray): Shape (N, ) Input to the model\n",
    "        label   (np.ndarray): Shape (N, ) Label of the data\n",
    "        w       (np.ndarray): Weight of the logistic regression model\n",
    "        b       (float): Bias of the logistic regression model\n",
    "    \n",
    "    Return:\n",
    "        total_loss (float): Total bceloss of the logistic regression model with l2 regularization, given data and label\n",
    "    \"\"\"\n",
    "    total_loss = 0\n",
    "\n",
    "    #[NOTE] write your code here!\n",
    "\n",
    "    return total_loss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def gradient_descent(data, label, w_init, b_init, iters=1500, lr=0.0001, l2_lambda=0.1):\n",
    "    \"\"\"\n",
    "    Performs batch gradient descent to obatain weight and bias of the logistic regression model.\n",
    "    \n",
    "    Args:\n",
    "      data    (np.ndarray): Shape (N,)\n",
    "      label   (np.ndarray): Shape (N,)\n",
    "      w_init  (np.ndarray): Initial value of weight of the model\n",
    "      b_init  (float): Initial values of bias of the model\n",
    "      lr      (float): Learning rate\n",
    "      iters   (int): Number of iterations to run gradient descent\n",
    "    Returns\n",
    "      w             (np.ndarray): Weight of the logistic regression model obtained with BGD\n",
    "      b             (float): Bias of the logistic regression model obtained with BGD\n",
    "      loss_history  (list): loss values of every iteration steps\n",
    "      w_history     (list): w values of every iteration steps\n",
    "      b_history     (list): b values of every iteration steps\n",
    "    \"\"\"\n",
    "    loss_history = []\n",
    "    w_history = []\n",
    "    b_history = []\n",
    "    w = w_init\n",
    "    b = b_init\n",
    "\n",
    "    for i in range(iters):\n",
    "        #[NOTE] write your code here!\n",
    "\n",
    "        # print loss for every 100 iterations\n",
    "        if i % 100 == 99:\n",
    "            print(f\"[ {i + 1:4}/{iters} ] BCELoss: {loss:.4f}\")\n",
    "    \n",
    "    return w, b, loss_history, w_history, b_history"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# initial values of w and b\n",
    "w_init, b_init = np.zeros(4), .0\n",
    "\n",
    "# hyperparameters for gradient descent algorithm\n",
    "iters = 1500\n",
    "learning_rate = 0.0001\n",
    "l2_lambda = 0.2\n",
    "\n",
    "# obtain w and b with gradient descent\n",
    "w, b, loss_history, w_history, b_history = gradient_descent(\n",
    "                                                data, label, w_init, b_init,\n",
    "                                                iters=iters,\n",
    "                                                lr=learning_rate,\n",
    "                                                l2_lambda=l2_lambda\n",
    "                                            )\n",
    "\n",
    "print(\"w,b found by gradient descent:\", w, b)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot training_curve!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "loss_history = loss_history\n",
    "\n",
    "# plot training curve\n",
    "plt.plot(range(len(loss_history)), loss_history)\n",
    "plt.title(\"Training Curve\")\n",
    "plt.xlabel(\"Iter\")\n",
    "plt.ylabel(\"Loss\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot decision boundary of your logistic regression model, trained with l2 regularized BCE loss!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# feel free to change 'feature_dims' argument to the other feature dimension indices,\n",
    "#   and check if your model fits well on the dataset!\n",
    "plot_decision_boundary(data, label, w, b, feature_dims=(0, 3))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "fid",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}