Is everything okay?

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 0 - Introduction\n",
    "\n",
    "Welcome to the <span style=\"color:yellowgreen\">Foundations of Machine Learning</span> (ECE5984_41) course!\n",
    "\n",
    "This is the <span style=\"color:red\">2nd</span> lab practice. From now on, you will going to implement logistic regression and L2 regularization. \n",
    "\n",
    "### Outline\n",
    "- [ 1 - Packages ](#1-packages)\n",
    "- [ 2 - Problem Statement and Dataset ](#2-problem-statement-and-dataset)\n",
    "- [ 3 - Logistic Regression ](#3-logistic-regression)\n",
    "  - [ 3.1 Sigmoid function ](#31-sigmoid-function) - Exercise 1\n",
    "  - [ 3.2 Binary Cross Entropy Loss](#32-binary-cross-entropy-loss) - Exercise 2\n",
    "  - [ 3.3 Gradient Descent ](#33-gradient-descent) - Exercise 3, Exercise 4\n",
    "- [ 4 - L2 Regularization ](#4-l2-regularization) - Exercise 5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Packages \n",
    "\n",
    "You have to install and use below packages for HW#2.\n",
    "- [numpy](https://www.numpy.org): Fundamental package for matrix computation with python.\n",
    "- [matplotlib](https://matplotlib.org): Package for visualization of the graph with python.\n",
    "- [pandas](https://pandas.pydata.org): Open source package for data analysis and manipulation.\n",
    "\n",
    "**Do not use other machine learning packages in this homework, e.g., *tensorflow*, *pytorch*, *jax*, etc.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import math\n",
    "import copy\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Problem Statement and Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Problem Statement\n",
    "\n",
    "[Fake Bills Dataset](https://www.kaggle.com/code/unmoved/classify-fake-bills)\n",
    "\n",
    "The *Fake Bills Dataset* consists \n",
    "The dataset consists 1500 samples of bills. Some of the bills are real, and some are fake.\n",
    "\n",
    "In this homework, we would like to fit the logistic regression model on this dataset to determine if the given bill is fake or real!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dataset\n",
    "\n",
    "You will start by loading the dataset. Make sure the attached 'fake_bills.csv' file is in your current directory(can be downloaded from icampus).\n",
    "\n",
    "Each column describes the below attributes:\n",
    "- \\[ is_genuine \\]: if the given bill is real or not.\n",
    "- \\[ diagonal \\]: the diagonal of the banknote in mm.\n",
    "- \\[ height_left \\]: the height of the left side of the banknote in mm.\n",
    "- \\[ height_right \\]: the height of the right side of the banknote in mm.\n",
    "- \\[ margin_low \\]: lower side margin of the banknote in mm.\n",
    "- \\[ margin_up \\]: upper side margin of the banknote in mm.\n",
    "- \\[ length \\]: the length of the banknote in mm.\n",
    "\n",
    "Run below cells and take a look of the dataset!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>is_genuine</th>\n",
       "      <th>diagonal</th>\n",
       "      <th>height_left</th>\n",
       "      <th>height_right</th>\n",
       "      <th>margin_low</th>\n",
       "      <th>margin_up</th>\n",
       "      <th>length</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>True</td>\n",
       "      <td>171.81</td>\n",
       "      <td>104.86</td>\n",
       "      <td>104.95</td>\n",
       "      <td>4.52</td>\n",
       "      <td>2.89</td>\n",
       "      <td>112.83</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>True</td>\n",
       "      <td>171.46</td>\n",
       "      <td>103.36</td>\n",
       "      <td>103.66</td>\n",
       "      <td>3.77</td>\n",
       "      <td>2.99</td>\n",
       "      <td>113.09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>True</td>\n",
       "      <td>172.69</td>\n",
       "      <td>104.48</td>\n",
       "      <td>103.50</td>\n",
       "      <td>4.40</td>\n",
       "      <td>2.94</td>\n",
       "      <td>113.16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>True</td>\n",
       "      <td>171.36</td>\n",
       "      <td>103.91</td>\n",
       "      <td>103.94</td>\n",
       "      <td>3.62</td>\n",
       "      <td>3.01</td>\n",
       "      <td>113.51</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>True</td>\n",
       "      <td>171.73</td>\n",
       "      <td>104.28</td>\n",
       "      <td>103.46</td>\n",
       "      <td>4.04</td>\n",
       "      <td>3.48</td>\n",
       "      <td>112.54</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   is_genuine  diagonal  height_left  height_right  margin_low  margin_up  \\\n",
       "0        True    171.81       104.86        104.95        4.52       2.89   \n",
       "1        True    171.46       103.36        103.66        3.77       2.99   \n",
       "2        True    172.69       104.48        103.50        4.40       2.94   \n",
       "3        True    171.36       103.91        103.94        3.62       3.01   \n",
       "4        True    171.73       104.28        103.46        4.04       3.48   \n",
       "\n",
       "   length  \n",
       "0  112.83  \n",
       "1  113.09  \n",
       "2  113.16  \n",
       "3  113.51  \n",
       "4  112.54  "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Load the dataset\n",
    "dataset = pd.read_csv(\"./fake_bills.csv\")\n",
    "\n",
    "# Display top 5 elements\n",
    "dataset.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 1500 entries, 0 to 1499\n",
      "Data columns (total 7 columns):\n",
      " #   Column        Non-Null Count  Dtype  \n",
      "---  ------        --------------  -----  \n",
      " 0   is_genuine    1500 non-null   bool   \n",
      " 1   diagonal      1500 non-null   float64\n",
      " 2   height_left   1500 non-null   float64\n",
      " 3   height_right  1500 non-null   float64\n",
      " 4   margin_low    1463 non-null   float64\n",
      " 5   margin_up     1500 non-null   float64\n",
      " 6   length        1500 non-null   float64\n",
      "dtypes: bool(1), float64(6)\n",
      "memory usage: 71.9 KB\n"
     ]
    }
   ],
   "source": [
    "# check dataset info\n",
    "dataset.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this lab practice, we will not going to use \"margin_low\" and \"margin_up\" features. \n",
    "\n",
    "(There are some missing-value cells on those attributes. We can use mean value or apply knn algorithm to fill in those cells, but in this lab practice, let's just simply drop those attributes.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 1500 entries, 0 to 1499\n",
      "Data columns (total 5 columns):\n",
      " #   Column        Non-Null Count  Dtype  \n",
      "---  ------        --------------  -----  \n",
      " 0   is_genuine    1500 non-null   bool   \n",
      " 1   diagonal      1500 non-null   float64\n",
      " 2   height_left   1500 non-null   float64\n",
      " 3   height_right  1500 non-null   float64\n",
      " 4   length        1500 non-null   float64\n",
      "dtypes: bool(1), float64(4)\n",
      "memory usage: 48.5 KB\n"
     ]
    }
   ],
   "source": [
    "dataset.drop(labels=['margin_low', 'margin_up'], axis=1, inplace=True)\n",
    "dataset.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The shape of data is (1500, 4)\n",
      "The shape of label is (1500,)\n",
      "\n",
      "The first Five elements of data are [[171.81 104.86 104.95 112.83]\n",
      " [171.46 103.36 103.66 113.09]\n",
      " [172.69 104.48 103.5  113.16]\n",
      " [171.36 103.91 103.94 113.51]\n",
      " [171.73 104.28 103.46 112.54]]\n",
      "The first Five elements of label are [1. 1. 1. 1. 1.]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Seperate train data and label from the dataset.\n",
    "data = dataset.drop(labels=['is_genuine'], axis=1, inplace=False).to_numpy()\n",
    "label = dataset[\"is_genuine\"].to_numpy()\n",
    "label = label.astype(float)\n",
    "\n",
    "# Check data statistics\n",
    "print(\"The shape of data is\", data.shape)\n",
    "print(\"The shape of label is\", label.shape)\n",
    "print()\n",
    "\n",
    "print(\"The first Five elements of data are\", data[:5])\n",
    "print(\"The first Five elements of label are\", label[:5])\n",
    "print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Normalizing data equalizes the scale of each features, which encourages fast and stable convergence for gradient descent. \n",
    "\n",
    "Run below code and normalize your data!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The first Five elements of data are [[-0.48653957  2.77412327  3.16324041  0.17365114]\n",
      " [-1.63372936 -2.23653528 -0.79966777  0.47166629]\n",
      " [ 2.3978233   1.50475644 -1.29119126  0.55190114]\n",
      " [-1.96149786 -0.39929381  0.06049835  0.95307539]\n",
      " [-0.74875438  0.83666863 -1.41407214 -0.15875038]]\n"
     ]
    }
   ],
   "source": [
    "# Normalize the data\n",
    "data = (data - np.mean(data, axis=0)) / np.std(data, axis=0)\n",
    "print(\"The first Five elements of data are\", data[:5])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Visualize two features (1st & 2nd dimension of the feature)\n",
    "real = (label == 1)\n",
    "fake = (label == 0)\n",
    "\n",
    "plt.scatter(data[real, 0], data[real, 1], marker='o', c='g', label=\"real\")\n",
    "plt.scatter(data[fake, 0], data[fake, 1], marker='x', c='r', label=\"fake\")\n",
    "plt.xlabel(\"Feature 1\")\n",
    "plt.ylabel(\"Feature 2\")\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Logistic Regression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.1 Sigmoid Function\n",
    "\n",
    "\$\$\\sigma(x) = \\frac{1}{1+e^{-x}}\$\$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\> In Exercise 1, you will implement the logistic function, i.e., sigmoid function. Below <span style=\"color:skyblue\">sigmoid()</span> returns sigmoid function value of given input x.\n",
    "\n",
    "Do not change the method name and its argument."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <span style=\"color:#ffd33d\"> Exercise 1 </span>: Write the code for sigmoid function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.26894142, 0.5       , 0.73105858])"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Complete the code for sigmoid function\n",
    "def sigmoid(x):\n",
    "    \"\"\"\n",
    "    compute the sigmoid of x.\n",
    "    \n",
    "    Args:\n",
    "        x    (np.ndarray): Shape (N, )\n",
    "    \n",
    "    Return:\n",
    "        sigma    (np.ndarray): Shape (N, ) Sigmoid values of given input x\n",
    "    \"\"\"\n",
    "    sigma = 1 / (1 + np.exp(-x))\n",
    "    return sigma\n",
    "\n",
    "x = np.array([-1, 0, 1])\n",
    "sigmoid(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.2 Binary Cross-Entropy Loss\n",
    "\n",
    "\$\$\\hat{y}^{(i)} = \\sigma(w^T x^{(i)} + b)\$\$\n",
    "\n",
    "\$\$BCE(w, b) = - \\sum_{i=1}^{N} y^{(i)} \\cdot log(\\hat{y}^{(i)}) + (1-y^{(i)}) \\cdot log(1-\\hat{y}^{(i)})\$\$\n",
    "\n",
    "* N is the number of training examples in the dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <span style=\"color:#ffd33d\"> Exercise 2 </span>: Complete the code for BCELoss"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\> Here, the <span style=\"color:skyblue\">compute_bceloss()</span> method computes the binary cross-entropy loss of the logistic regression model. \n",
    "\n",
    "Feel free to change or add variables, but do not modify the method name and its arguements."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1039.7207708398773"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def compute_bceloss(data, label, w, b):\n",
    "    \"\"\"\n",
    "    Compute the bceloss of the logistic regression model, given data and label.\n",
    "\n",
    "    Args:\n",
    "        data    (np.ndarray): Shape (N, ) Input to the model\n",
    "        label   (np.ndarray): Shape (N, ) Label of the data\n",
    "        w       (np.ndarray): Weight of the logistic regression model\n",
    "        b       (float): Bias of the logistic regression model\n",
    "    \n",
    "    Return:\n",
    "        total_loss (float): Total bceloss of the logistic regression model, given data and label\n",
    "    \"\"\"\n",
    "    total_loss = 0\n",
    "\n",
    "    #[NOTE] write your code here!\n",
    "    # Implement the loss calculation\n",
    "    N = data.shape[0]  # Number of examples\n",
    "    \n",
    "    for i in range(N):\n",
    "        # Compute the prediction using sigmoid function\n",
    "        y_hat = sigmoid(np.dot(w, data[i]) + b)\n",
    "        # Compute the loss for the ith example\n",
    "        loss_i = -label[i] * np.log(y_hat) - (1 - label[i]) * np.log(1 - y_hat)\n",
    "        total_loss += loss_i\n",
    "\n",
    "    return total_loss\n",
    "\n",
    "# Assuming the shapes of ‘data’ and ‘label’ match the requirements (N, D) and (N, ) respectively\n",
    "# Initialize some dummy ‘data’ and ‘label’ if needed\n",
    "# Here is an example assuming we have a dataset with 3 samples and 4 features:\n",
    "# Example inputs just for illustration\n",
    "# data = np.array([[0.2, 0.4, 0.6, 0.8],\n",
    "#                  [0.1, 0.3, 0.5, 0.7],\n",
    "#                  [0.9, 0.8, 0.7, 0.6]])\n",
    "# label = np.array([1, 0, 1])\n",
    "\n",
    "# Initialize weights and bias\n",
    "w = np.zeros(4)\n",
    "b = 0.0\n",
    "\n",
    "compute_bceloss(data, label, w, b)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.3 Gradient Descent\n",
    "\n",
    "\$\$\n",
    "\\frac{\\partial BCE(\\mathbf{w},b)}{\\partial w_j}  = \\sum\\limits_{i = 1}^{N} (\\hat{y}^{(i)} - {y}^{(i)})x_{j}^{(i)}\n",
    "\$\$\n",
    "\$\$\n",
    "\\frac{\\partial BCE(\\mathbf{w},b)}{\\partial b}  = \\sum\\limits_{i = 1}^{N} (\\hat{y}^{(i)} - {y}^{(i)})\n",
    "\$\$\n",
    "* N is the number of training examples in the dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <span style=\"color:#ffd33d\"> Exercise 3 </span>: Complete the code for computing the gradient of the loss w.r.t weights and bias"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\> Here, the <span style=\"color:skyblue\">compute_gradient()</span> method computes the gradient of the loss with respect to the parameters of the logistic regression model.\n",
    "\n",
    "Feel free to change or add variables, but do not modify the method name and its arguments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([-0.06258193,  0.17905496,  0.22867448, -0.40035662]),\n",
       " -0.16666666666666666)"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def compute_gradient(data, label, w, b):\n",
    "    \"\"\"\n",
    "    Compute the gradient of the loss w.r.t weight & bias of model, given data and label.\n",
    "\n",
    "    Args:\n",
    "        data    (np.ndarray): Shape (N, ) Input to the model\n",
    "        label   (np.ndarray): Shape (N, ) Label of the data\n",
    "        w       (np.ndarray): Weight of the logistic regression model\n",
    "        b       (float): Bias of the logistic regression model\n",
    "    \n",
    "    Return:\n",
    "       grad_w   (np.ndarray): The gradient of the loss w.r.t weight w\n",
    "       grad_b   (float): The gradient of the loss w.r.t bias b\n",
    "    \"\"\"\n",
    "    grad_w, grad_b = 0, 0\n",
    "\n",
    "    #[NOTE] write your code here!\n",
    "    N = data.shape[0]\n",
    "\n",
    "    # Predicted probability\n",
    "    y_hat = sigmoid(np.dot(data, w) + b)\n",
    "    \n",
    "    # Error\n",
    "    error = y_hat - label\n",
    "    \n",
    "    # Gradient w.r.t weights\n",
    "    grad_w = np.dot(data.T, error) / N\n",
    "    \n",
    "    # Gradient w.r.t bias\n",
    "    grad_b = np.sum(error) / N\n",
    "\n",
    "    return grad_w, grad_b\n",
    "\n",
    "compute_gradient(data, label, w, b)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <span style=\"color:#ffd33d\"> Exercise 4 </span>: Complete the code of gradient descent algorithm"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\> <span style=\"color:skyblue\">gradient_descent()</span> method applies gradient descent on logistic regression model on the given dataset. You should make 'loop' inside this method to iteratively update w and b. You __**must**__ use <span style=\"color:skyblue\">compute_bceloss()</span> and <span style=\"color:skyblue\">compute_gradient()</span> method to calculate bceloss and gradients, respectively.\n",
    "\n",
    "Also, for the purpose of visualization of the training curve, this method returns some 'history' lists. You can just simply append updated values to the corresponding 'history' list.\n",
    "\n",
    "Feel free to change or add variables, but do not modify the method name, its arguments, and return values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "def gradient_descent(data, label, w_init, b_init, iters=1500, lr=0.0001):\n",
    "    \"\"\"\n",
    "    Performs batch gradient descent to obatain weight and bias of the logistic regression model.\n",
    "    \n",
    "    Args:\n",
    "      data    (np.ndarray): Shape (N,)\n",
    "      label   (np.ndarray): Shape (N,)\n",
    "      w_init  (np.ndarray): Initial value of weight of the model\n",
    "      b_init  (float): Initial values of bias of the model\n",
    "      lr      (float): Learning rate\n",
    "      iters   (int): Number of iterations to run gradient descent\n",
    "    Returns\n",
    "      w             (np.ndarray): Weight of the logistic regression model obtained with BGD\n",
    "      b             (float): Bias of the logistic regression model obtained with BGD\n",
    "      loss_history  (list): loss values of every iteration steps\n",
    "      w_history     (list): w values of every iteration steps\n",
    "      b_history     (list): b values of every iteration steps\n",
    "    \"\"\"\n",
    "    loss_history = []\n",
    "    w_history = []\n",
    "    b_history = []\n",
    "    w = w_init\n",
    "    b = b_init\n",
    "\n",
    "    for i in range(iters):\n",
    "        #[NOTE] write your code here!\n",
    "        # Calculate current loss and gradients\n",
    "        loss = compute_bceloss(data, label, w, b)\n",
    "        dw, db = compute_gradient(data, label, w, b)\n",
    "        \n",
    "        # Update parameters\n",
    "        w -= lr * dw\n",
    "        b -= lr * db\n",
    "\n",
    "        # Append current loss and parameters to their history lists\n",
    "        loss_history.append(loss)\n",
    "        w_history.append(w)\n",
    "        b_history.append(b)\n",
    "\n",
    "        # print loss for every 100 iterations\n",
    "        if i % 100 == 99:\n",
    "            print(f\"[ {i + 1:4}/{iters} ] BCELoss: {loss:.4f}\")\n",
    "    \n",
    "    return w, b, loss_history, w_history, b_history"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now apply your gradient descent algorithm!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[  100/1500 ] BCELoss: 1135.4708\n",
      "[  200/1500 ] BCELoss: 1132.8894\n",
      "[  300/1500 ] BCELoss: 1130.3160\n",
      "[  400/1500 ] BCELoss: 1127.7507\n",
      "[  500/1500 ] BCELoss: 1125.1933\n",
      "[  600/1500 ] BCELoss: 1122.6439\n",
      "[  700/1500 ] BCELoss: 1120.1024\n",
      "[  800/1500 ] BCELoss: 1117.5688\n",
      "[  900/1500 ] BCELoss: 1115.0432\n",
      "[ 1000/1500 ] BCELoss: 1112.5254\n",
      "[ 1100/1500 ] BCELoss: 1110.0154\n",
      "[ 1200/1500 ] BCELoss: 1107.5133\n",
      "[ 1300/1500 ] BCELoss: 1105.0190\n",
      "[ 1400/1500 ] BCELoss: 1102.5324\n",
      "[ 1500/1500 ] BCELoss: 1100.0536\n",
      "w,b found by gradient descent: [-1.26121716 -0.80893992 -1.59510877 -0.0129695 ] 0.02487778352746521\n"
     ]
    }
   ],
   "source": [
    "# initial values of w and b\n",
    "w_init, b_init = np.random.randn(4), .0\n",
    "\n",
    "# hyperparameters for gradient descent algorithm\n",
    "iters = 1500\n",
    "learning_rate = 0.0001\n",
    "\n",
    "# obtain w and b with gradient descent\n",
    "w, b, loss_history, w_history, b_history = gradient_descent(\n",
    "                                                data, label, w_init, b_init,\n",
    "                                                iters=iters,\n",
    "                                                lr=learning_rate,\n",
    "                                            )\n",
    "\n",
    "print(\"w,b found by gradient descent:\", w, b)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot training curve!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "loss_history = loss_history\n",
    "\n",
    "# plot training curve\n",
    "plt.plot(range(len(loss_history)), loss_history)\n",
    "plt.title(\"Training Curve\")\n",
    "plt.xlabel(\"Iter\")\n",
    "plt.ylabel(\"Loss\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot decision boundary of your logistic regression model!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "def plot_decision_boundary(data, label, w, b, feature_dims=(0, 1)):\n",
    "    # scatter data along the feature_dims\n",
    "    dim_1, dim_2 = feature_dims\n",
    "\n",
    "    real = (label == 1)\n",
    "    fake = (label == 0)\n",
    "\n",
    "    plt.scatter(data[real, dim_1], data[real, dim_2], marker='o', c='g', label=\"real\")\n",
    "    plt.scatter(data[fake, dim_1], data[fake, dim_2], marker='x', c='r', label=\"fake\")\n",
    "    plt.xlabel(f'Feature {dim_1 + 1}')\n",
    "    plt.ylabel(f'Feature {dim_2 + 1}')\n",
    "\n",
    "    db_x = np.array([min(data[:, dim_1]), max(data[:, dim_1])])\n",
    "    db_y = ((-1. / w[dim_2]) * (w[dim_1] * db_x + b))\n",
    "\n",
    "    plt.plot(db_x, db_y)\n",
    "\n",
    "# feel free to change 'feature_dims' argument to the other feature dimension indices,\n",
    "#   and check if your model fits well on the dataset!\n",
    "plot_decision_boundary(data, label, w, b, feature_dims=(0, 3))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. L2 Regularization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\$\$\\hat{y}^{(i)} = \\sigma(w^T x^{(i)} + b)\$\$\n",
    "\n",
    "\$\$BCE(w, b) = - \\sum_{i=1}^{N} y^{(i)} \\cdot log(\\hat{y}^{(i)}) + (1-y^{(i)}) \\cdot log(1-\\hat{y}^{(i)})\$\$\n",
    "\n",
    "\$\$L(w, b) = BCE(w, b) + \\frac{\\lambda}{2} \\| w\\|^2\$\$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <span style=\"color:#ffd33d\"> Exercise 5 </span>: Complete the below codes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\> Here, the <span style=\"color:skyblue\">compute_l2_reg_loss()</span> method computes the l2 regularized loss of the logistic regression model.\n",
    "\n",
    "Feel free to change or add variables, but do not modify the method name and its arguements."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "def compute_l2_reg_loss(data, label, w, b, l2_lambda=0.5):\n",
    "    \"\"\"\n",
    "    Compute the bceloss of the logistic regression model, given data and label.\n",
    "\n",
    "    Args:\n",
    "        data    (np.ndarray): Shape (N, ) Input to the model\n",
    "        label   (np.ndarray): Shape (N, ) Label of the data\n",
    "        w       (np.ndarray): Weight of the logistic regression model\n",
    "        b       (float): Bias of the logistic regression model\n",
    "    \n",
    "    Return:\n",
    "        total_loss (float): Total bceloss of the logistic regression model with l2 regularization, given data and label\n",
    "    \"\"\"\n",
    "    total_loss = 0\n",
    "\n",
    "    #[NOTE] write your code here!\n",
    "    # Ensure data is in the correct dimension\n",
    "    if len(data.shape) == 1:\n",
    "        data = data[:, np.newaxis]\n",
    "    \n",
    "    # Calculate predictions\n",
    "    predictions = sigmoid(np.dot(data, w) + b)\n",
    "    \n",
    "    # Calculate BCE loss\n",
    "    bce_loss = -np.mean(label * np.log(predictions) + (1 - label) * np.log(1 - predictions))\n",
    "    \n",
    "    # Calculate L2 Regularization term\n",
    "    l2_reg = l2_lambda / 2 * np.sum(w ** 2)\n",
    "    \n",
    "    # Calculate total loss\n",
    "    total_loss = bce_loss + l2_reg\n",
    "\n",
    "    return total_loss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "def gradient_descent(data, label, w_init, b_init, iters=1500, lr=0.0001, l2_lambda=0.1):\n",
    "    \"\"\"\n",
    "    Performs batch gradient descent to obatain weight and bias of the logistic regression model.\n",
    "    \n",
    "    Args:\n",
    "      data    (np.ndarray): Shape (N,)\n",
    "      label   (np.ndarray): Shape (N,)\n",
    "      w_init  (np.ndarray): Initial value of weight of the model\n",
    "      b_init  (float): Initial values of bias of the model\n",
    "      lr      (float): Learning rate\n",
    "      iters   (int): Number of iterations to run gradient descent\n",
    "    Returns\n",
    "      w             (np.ndarray): Weight of the logistic regression model obtained with BGD\n",
    "      b             (float): Bias of the logistic regression model obtained with BGD\n",
    "      loss_history  (list): loss values of every iteration steps\n",
    "      w_history     (list): w values of every iteration steps\n",
    "      b_history     (list): b values of every iteration steps\n",
    "    \"\"\"\n",
    "    loss_history = []\n",
    "    w_history = []\n",
    "    b_history = []\n",
    "    w = w_init\n",
    "    b = b_init\n",
    "\n",
    "    for i in range(iters):\n",
    "        #[NOTE] write your code here!\n",
    "        # Calculate current loss and gradients\n",
    "        loss = compute_bceloss(data, label, w, b)\n",
    "        dw, db = compute_gradient(data, label, w, b)\n",
    "        \n",
    "        # Update parameters\n",
    "        w -= lr * dw\n",
    "        b -= lr * db\n",
    "\n",
    "        # Append current loss and parameters to their history lists\n",
    "        loss_history.append(loss)\n",
    "        w_history.append(w)\n",
    "        b_history.append(b)\n",
    "\n",
    "        # print loss for every 100 iterations\n",
    "        if i % 100 == 99:\n",
    "            print(f\"[ {i + 1:4}/{iters} ] BCELoss: {loss:.4f}\")\n",
    "    \n",
    "    return w, b, loss_history, w_history, b_history"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[  100/1500 ] BCELoss: 1035.6326\n",
      "[  200/1500 ] BCELoss: 1031.5343\n",
      "[  300/1500 ] BCELoss: 1027.4670\n",
      "[  400/1500 ] BCELoss: 1023.4306\n",
      "[  500/1500 ] BCELoss: 1019.4248\n",
      "[  600/1500 ] BCELoss: 1015.4492\n",
      "[  700/1500 ] BCELoss: 1011.5038\n",
      "[  800/1500 ] BCELoss: 1007.5882\n",
      "[  900/1500 ] BCELoss: 1003.7022\n",
      "[ 1000/1500 ] BCELoss: 999.8456\n",
      "[ 1100/1500 ] BCELoss: 996.0181\n",
      "[ 1200/1500 ] BCELoss: 992.2194\n",
      "[ 1300/1500 ] BCELoss: 988.4494\n",
      "[ 1400/1500 ] BCELoss: 984.7077\n",
      "[ 1500/1500 ] BCELoss: 980.9943\n",
      "w,b found by gradient descent: [ 0.00910144 -0.02586466 -0.03310694  0.05852207] 0.02453753318900997\n"
     ]
    }
   ],
   "source": [
    "# initial values of w and b\n",
    "w_init, b_init = np.zeros(4), .0\n",
    "\n",
    "# hyperparameters for gradient descent algorithm\n",
    "iters = 1500\n",
    "learning_rate = 0.0001\n",
    "l2_lambda = 0.2\n",
    "\n",
    "# obtain w and b with gradient descent\n",
    "w, b, loss_history, w_history, b_history = gradient_descent(\n",
    "                                                data, label, w_init, b_init,\n",
    "                                                iters=iters,\n",
    "                                                lr=learning_rate,\n",
    "                                                l2_lambda=l2_lambda\n",
    "                                            )\n",
    "\n",
    "print(\"w,b found by gradient descent:\", w, b)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot training_curve!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "loss_history = loss_history\n",
    "\n",
    "# plot training curve\n",
    "plt.plot(range(len(loss_history)), loss_history)\n",
    "plt.title(\"Training Curve\")\n",
    "plt.xlabel(\"Iter\")\n",
    "plt.ylabel(\"Loss\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot decision boundary of your logistic regression model, trained with l2 regularized BCE loss!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# feel free to change 'feature_dims' argument to the other feature dimension indices,\n",
    "#   and check if your model fits well on the dataset!\n",
    "plot_decision_boundary(data, label, w, b, feature_dims=(0, 3))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "fid",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}