text: {
   "cells": [
    {
     "cell_type": "markdown",
     "metadata": {
      "colab_type": "text",
      "id": "view-in-github"
     },
     "source": [
      "<a href=\"https://colab.research.google.com/github/Tessellate-Imaging/monk_v1/blob/master/study_roadmaps/1_getting_started_roadmap/6_hyperparameter_tuning/1)%20Analyse%20Learning%20Rates.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
     ]
    },
    {
     "cell_type": "code",
     "execution_count": null,
     "metadata": {},
     "outputs": [],
     "source": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "# Goals\n",
      "\n",
      "\n",
      "### Learn how to use hyper parameter analyser for learning rates"
     ]
    },
    {
     "cell_type": "code",
     "execution_count": null,
     "metadata": {},
     "outputs": [],
     "source": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "# Table of Contents\n",
      "\n",
      "\n",
      "## [Install](#0)\n",
      "\n",
      "\n",
      "## [Load experiment in default mode](#1)\n",
      "\n",
      "\n",
      "## [Run Analyser](#2)"
     ]
    },
    {
     "cell_type": "code",
     "execution_count": null,
     "metadata": {},
     "outputs": [],
     "source": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "<a id='0'></a>\n",
      "# Install Monk"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "## Using pip (Recommended)\n",
      "\n",
      "  - colab (gpu) \n",
      "      - All bakcends: `pip install -U monk-colab`\n",
      "      \n",
      "\n",
      "  - kaggle (gpu) \n",
      "      - All backends: `pip install -U monk-kaggle`\n",
      "      \n",
      "\n",
      "  - cuda 10.2\t\n",
      "      - All backends: `pip install -U monk-cuda102`\n",
      "      - Gluon bakcned: `pip install -U monk-gluon-cuda102`\n",
      "\t  - Pytorch backend: `pip install -U monk-pytorch-cuda102`\n",
      "      - Keras backend: `pip install -U monk-keras-cuda102`\n",
      "      \n",
      "\n",
      "  - cuda 10.1\t\n",
      "      - All backend: `pip install -U monk-cuda101`\n",
      "\t  - Gluon bakcned: `pip install -U monk-gluon-cuda101`\n",
      "\t  - Pytorch backend: `pip install -U monk-pytorch-cuda101`\n",
      "\t  - Keras backend: `pip install -U monk-keras-cuda101`\n",
      "      \n",
      "\n",
      "  - cuda 10.0\t\n",
      "      - All backend: `pip install -U monk-cuda100`\n",
      "\t  - Gluon bakcned: `pip install -U monk-gluon-cuda100`\n",
      "\t  - Pytorch backend: `pip install -U monk-pytorch-cuda100`\n",
      "\t  - Keras backend: `pip install -U monk-keras-cuda100`\n",
      "      \n",
      "\n",
      "  - cuda 9.2\t\n",
      "      - All backend: `pip install -U monk-cuda92`\n",
      "\t  - Gluon bakcned: `pip install -U monk-gluon-cuda92`\n",
      "\t  - Pytorch backend: `pip install -U monk-pytorch-cuda92`\n",
      "\t  - Keras backend: `pip install -U monk-keras-cuda92`\n",
      "      \n",
      "\n",
      "  - cuda 9.0\t\n",
      "      - All backend: `pip install -U monk-cuda90`\n",
      "\t  - Gluon bakcned: `pip install -U monk-gluon-cuda90`\n",
      "\t  - Pytorch backend: `pip install -U monk-pytorch-cuda90`\n",
      "\t  - Keras backend: `pip install -U monk-keras-cuda90`\n",
      "      \n",
      "\n",
      "  - cpu \t\t\n",
      "      - All backend: `pip install -U monk-cpu`\n",
      "\t  - Gluon bakcned: `pip install -U monk-gluon-cpu`\n",
      "\t  - Pytorch backend: `pip install -U monk-pytorch-cpu`\n",
      "\t  - Keras backend: `pip install -U monk-keras-cpu`"
     ]
    },
    {
     "cell_type": "code",
     "execution_count": null,
     "metadata": {},
     "outputs": [],
     "source": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "## Install Monk Manually (Not recommended)\n",
      " \n",
      "### Step 1: Clone the library\n",
      " - git clone https://github.com/Tessellate-Imaging/monk_v1.git\n",
      " \n",
      " \n",
      " \n",
      " \n",
      "### Step 2: Install requirements \n",
      " - Linux\n",
      "     - Cuda 9.0\n",
      "         - `cd monk_v1/installation/Linux && pip install -r requirements_cu90.txt`\n",
      "     - Cuda 9.2\n",
      "         - `cd monk_v1/installation/Linux && pip install -r requirements_cu92.txt`\n",
      "     - Cuda 10.0\n",
      "         - `cd monk_v1/installation/Linux && pip install -r requirements_cu100.txt`\n",
      "     - Cuda 10.1\n",
      "         - `cd monk_v1/installation/Linux && pip install -r requirements_cu101.txt`\n",
      "     - Cuda 10.2\n",
      "         - `cd monk_v1/installation/Linux && pip install -r requirements_cu102.txt`\n",
      "     - CPU (Non gpu system)\n",
      "         - `cd monk_v1/installation/Linux && pip install -r requirements_cpu.txt`\n",
      " \n",
      " \n",
      "  - Windows\n",
      "     - Cuda 9.0 (Experimental support)\n",
      "         - `cd monk_v1/installation/Windows && pip install -r requirements_cu90.txt`\n",
      "     - Cuda 9.2 (Experimental support)\n",
      "         - `cd monk_v1/installation/Windows && pip install -r requirements_cu92.txt`\n",
      "     - Cuda 10.0 (Experimental support)\n",
      "         - `cd monk_v1/installation/Windows && pip install -r requirements_cu100.txt`\n",
      "     - Cuda 10.1 (Experimental support)\n",
      "         - `cd monk_v1/installation/Windows && pip install -r requirements_cu101.txt`\n",
      "     - Cuda 10.2 (Experimental support)\n",
      "         - `cd monk_v1/installation/Windows && pip install -r requirements_cu102.txt`\n",
      "     - CPU (Non gpu system)\n",
      "         - `cd monk_v1/installation/Windows && pip install -r requirements_cpu.txt`\n",
      "         \n",
      "         \n",
      "  - Mac\n",
      "     - CPU (Non gpu system)\n",
      "         - `cd monk_v1/installation/Mac && pip install -r requirements_cpu.txt`\n",
      "         \n",
      "         \n",
      "  - Misc\n",
      "      - Colab (GPU)\n",
      "          - `cd monk_v1/installation/Misc && pip install -r requirements_colab.txt`\n",
      "      - Kaggle (GPU)\n",
      "          - `cd monk_v1/installation/Misc && pip install -r requirements_kaggle.txt`\n",
      " \n",
      " \n",
      " \n",
      "### Step 3: Add to system path (Required for every terminal or kernel run)\n",
      " - `import sys`\n",
      " - `sys.path.append(\"monk_v1/\");`"
     ]
    },
    {
     "cell_type": "code",
     "execution_count": null,
     "metadata": {},
     "outputs": [],
     "source": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "## Dataset - Caltech-256\n",
      "    - https://www.kaggle.com/jessicali9530/caltech256"
     ]
    },
    {
     "cell_type": "code",
     "execution_count": null,
     "metadata": {},
     "outputs": [],
     "source": [
      "! wget --load-cookies /tmp/cookies.txt \"https://docs.google.com/uc?export=download&confirm=$(wget --save-cookies /tmp/cookies.txt --keep-session-cookies --no-check-certificate 'https://docs.google.com/uc?export=download&id=1Lltrl2U4L8WJkyBjMBFHSaoK8dLhoItl' -O- | sed -rn 's/.*confirm=([0-9A-Za-z_]+).*/\\1\\n/p')&id=1Lltrl2U4L8WJkyBjMBFHSaoK8dLhoItl\" -O caltech256.zip && rm -rf /tmp/cookies.txt"
     ]
    },
    {
     "cell_type": "code",
     "execution_count": 3,
     "metadata": {},
     "outputs": [],
     "source": [
      "! unzip -qq caltech256.zip"
     ]
    },
    {
     "cell_type": "code",
     "execution_count": null,
     "metadata": {},
     "outputs": [],
     "source": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "# Imports"
     ]
    },
    {
     "cell_type": "code",
     "execution_count": 1,
     "metadata": {},
     "outputs": [],
     "source": [
      "#Using gluon backend \n",
      "\n",
      "# When installed using pip\n",
      "from monk.gluon_prototype import prototype\n",
      "\n",
      "\n",
      "# When installed manually (Uncomment the following)\n",
      "#import os\n",
      "#import sys\n",
      "#sys.path.append(\"monk_v1/\");\n",
      "#sys.path.append(\"monk_v1/monk/\");\n",
      "#from monk.gluon_prototype import prototype"
     ]
    },
    {
     "cell_type": "code",
     "execution_count": null,
     "metadata": {},
     "outputs": [],
     "source": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "<a id='1'></a>\n",
      "# Load experiment in default mode"
     ]
    },
    {
     "cell_type": "code",
     "execution_count": 3,
     "metadata": {},
     "outputs": [
      {
       "name": "stdout",
       "output_type": "stream",
       "text": [
        "Mxnet Version: 1.5.0\n",
        "\n",
        "Experiment Details\n",
        "    Project: Project\n",
        "    Experiment: analyser_lr\n",
        "    Dir: /home/abhi/Desktop/Work/tess_tool/gui/v0.3/finetune_models/Organization/development/v5.3_roadmaps/1_getting_started_roadmap/6_hyperparameter_tuning/workspace/Project/analyser_lr/\n",
        "\n"
       ]
      }
     ],
     "source": [
      "gtf = prototype(verbose=1);\n",
      "gtf.Prototype(\"Project\", \"analyser_lr\");"
     ]
    },
    {
     "cell_type": "code",
     "execution_count": 4,
     "metadata": {},
     "outputs": [
      {
       "name": "stdout",
       "output_type": "stream",
       "text": [
        "Dataset Details\n",
        "    Train path:     caltech256/train\n",
        "    Val path:       None\n",
        "    CSV train path: None\n",
        "    CSV val path:   None\n",
        "\n",
        "Dataset Params\n",
        "    Input Size:   224\n",
        "    Batch Size:   4\n",
        "    Data Shuffle: True\n",
        "    Processors:   4\n",
        "    Train-val split:   0.7\n",
        "\n"
       ]
      },
      {
       "name": "stderr",
       "output_type": "stream",
       "text": [
        "/home/abhi/.virtualenvs/finetune_py36/lib/python3.6/site-packages/mxnet/gluon/data/vision/datasets.py:312: UserWarning: Ignoring caltech256/train/198.spider/RENAME2 of type . Only support .jpg, .jpeg, .png\n",
        "  filename, ext, ', '.join(self._exts)))\n"
       ]
      },
      {
       "name": "stdout",
       "output_type": "stream",
       "text": [
        "Pre-Composed Train Transforms\n",
        "[{'RandomHorizontalFlip': {'p': 0.8}}, {'Normalize': {'mean': [0.485, 0.456, 0.406], 'std': [0.229, 0.224, 0.225]}}]\n",
        "\n",
        "Pre-Composed Val Transforms\n",
        "[{'RandomHorizontalFlip': {'p': 0.8}}, {'Normalize': {'mean': [0.485, 0.456, 0.406], 'std': [0.229, 0.224, 0.225]}}]\n",
        "\n",
        "Dataset Numbers\n",
        "    Num train images: 21424\n",
        "    Num val images:   9183\n",
        "    Num classes:      257\n",
        "\n",
        "Model Params\n",
        "    Model name:           resnet18_v1\n",
        "    Use Gpu:              True\n",
        "    Use pretrained:       True\n",
        "    Freeze base network:  False\n",
        "\n",
        "Model Details\n",
        "    Loading pretrained model\n",
        "    Model Loaded on device\n",
        "        Model name:                           resnet18_v1\n",
        "        Num of potentially trainable layers:  41\n",
        "        Num of actual trainable layers:       41\n",
        "\n",
        "Optimizer\n",
        "    Name:          sgd\n",
        "    Learning rate: 0.01\n",
        "    Params:        {'lr': 0.01, 'momentum': 0, 'weight_decay': 0, 'momentum_dampening_rate': 0, 'clipnorm': 0.0, 'clipvalue': 0.0}\n",
        "\n",
        "\n",
        "\n",
        "Learning rate scheduler\n",
        "    Name:   steplr\n",
        "    Params: {'step_size': 1, 'gamma': 0.98, 'last_epoch': -1}\n",
        "\n",
        "Loss\n",
        "    Name:          softmaxcrossentropy\n",
        "    Params:        {'weight': None, 'batch_axis': 0, 'axis_to_sum_over': -1, 'label_as_categories': True, 'label_smoothing': False}\n",
        "\n",
        "Training params\n",
        "    Num Epochs: 5\n",
        "\n",
        "Display params\n",
        "    Display progress:          True\n",
        "    Display progress realtime: True\n",
        "    Save Training logs:        True\n",
        "    Save Intermediate models:  True\n",
        "    Intermediate model prefix: intermediate_model_\n",
        "\n"
       ]
      },
      {
       "name": "stderr",
       "output_type": "stream",
       "text": [
        "monk_v1/monk/system/imports.py:160: UserWarning: ArgumentWarning: clipnorm and clipvalue are active only for keras in current version of Monk\n",
        "  warnings.warn(msg)\n",
        "monk_v1/monk/system/imports.py:160: UserWarning: ArgumentWarning: momentum_dampening_rate is active only for pytorch in current version of Monk\n",
        "  warnings.warn(msg)\n"
       ]
      }
     ],
     "source": [
      "gtf.Default(dataset_path=\"caltech256/train\", \n",
      "            model_name=\"resnet18_v1\", \n",
      "            freeze_base_network=False,\n",
      "            num_epochs=5);\n",
      "\n",
      "#Read the summary generated once you run this cell. "
     ]
    },
    {
     "cell_type": "code",
     "execution_count": null,
     "metadata": {},
     "outputs": [],
     "source": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "<a id='2'></a>\n",
      "# Analyse Learning Rates"
     ]
    },
    {
     "cell_type": "code",
     "execution_count": 5,
     "metadata": {},
     "outputs": [],
     "source": [
      "# Analysis Project Name\n",
      "analysis_name = \"analyse_learning_rates\""
     ]
    },
    {
     "cell_type": "code",
     "execution_count": 6,
     "metadata": {},
     "outputs": [],
     "source": [
      "# Learning rates to explore\n",
      "lrs = [0.1, 0.05, 0.01, 0.005, 0.0001];"
     ]
    },
    {
     "cell_type": "code",
     "execution_count": 7,
     "metadata": {},
     "outputs": [],
     "source": [
      "# Num epochs for each sub-experiment to run\n",
      "epochs=10"
     ]
    },
    {
     "cell_type": "code",
     "execution_count": 8,
     "metadata": {},
     "outputs": [],
     "source": [
      "# Percentage of original dataset to take in for experimentation\n",
      "percent_data=10"
     ]
    },
    {
     "cell_type": "code",
     "execution_count": 9,
     "metadata": {},
     "outputs": [
      {
       "name": "stdout",
       "output_type": "stream",
       "text": [
        "\n",
        "Running Learning rate analysis\n",
        "Analysis Name      : analyse_learning_rates\n",
        "\n",
        "Running experiment : 1/5\n",
        "Experiment name    : Learning_Rate_0.1\n"
       ]
      },
      {
       "name": "stderr",
       "output_type": "stream",
       "text": [
        "/home/abhi/.virtualenvs/finetune_py36/lib/python3.6/site-packages/mxnet/gluon/data/vision/datasets.py:312: UserWarning: Ignoring caltech256/train/198.spider/RENAME2 of type . Only support .jpg, .jpeg, .png\n",
        "  filename, ext, ', '.join(self._exts)))\n"
       ]
      },
      {
       "name": "stdout",
       "output_type": "stream",
       "text": [
        "Estimated time     : 3 min\n",
        "Experiment Complete\n",
        "\n",
        "\n",
        "Running experiment : 2/5\n",
        "Experiment name    : Learning_Rate_0.05\n"
       ]
      },
      {
       "name": "stderr",
       "output_type": "stream",
       "text": [
        "/home/abhi/.virtualenvs/finetune_py36/lib/python3.6/site-packages/mxnet/gluon/data/vision/datasets.py:312: UserWarning: Ignoring caltech256/train/198.spider/RENAME2 of type . Only support .jpg, .jpeg, .png\n",
        "  filename, ext, ', '.join(self._exts)))\n"
       ]
      },
      {
       "name": "stdout",
       "output_type": "stream",
       "text": [
        "Estimated time     : 3 min\n",
        "Experiment Complete\n",
        "\n",
        "\n",
        "Running experiment : 3/5\n",
        "Experiment name    : Learning_Rate_0.01\n"
       ]
      },
      {
       "name": "stderr",
       "output_type": "stream",
       "text": [
        "/home/abhi/.virtualenvs/finetune_py36/lib/python3.6/site-packages/mxnet/gluon/data/vision/datasets.py:312: UserWarning: Ignoring caltech256/train/198.spider/RENAME2 of type . Only support .jpg, .jpeg, .png\n",
        "  filename, ext, ', '.join(self._exts)))\n"
       ]
      },
      {
       "name": "stdout",
       "output_type": "stream",
       "text": [
        "Estimated time     : 3 min\n",
        "Experiment Complete\n",
        "\n",
        "\n",
        "Running experiment : 4/5\n",
        "Experiment name    : Learning_Rate_0.005\n"
       ]
      },
      {
       "name": "stderr",
       "output_type": "stream",
       "text": [
        "/home/abhi/.virtualenvs/finetune_py36/lib/python3.6/site-packages/mxnet/gluon/data/vision/datasets.py:312: UserWarning: Ignoring caltech256/train/198.spider/RENAME2 of type . Only support .jpg, .jpeg, .png\n",
        "  filename, ext, ', '.join(self._exts)))\n"
       ]
      },
      {
       "name": "stdout",
       "output_type": "stream",
       "text": [
        "Estimated time     : 3 min\n",
        "Experiment Complete\n",
        "\n",
        "\n",
        "Running experiment : 5/5\n",
        "Experiment name    : Learning_Rate_0.0001\n"
       ]
      },
      {
       "name": "stderr",
       "output_type": "stream",
       "text": [
        "/home/abhi/.virtualenvs/finetune_py36/lib/python3.6/site-packages/mxnet/gluon/data/vision/datasets.py:312: UserWarning: Ignoring caltech256/train/198.spider/RENAME2 of type . Only support .jpg, .jpeg, .png\n",
        "  filename, ext, ', '.join(self._exts)))\n"
       ]
      },
      {
       "name": "stdout",
       "output_type": "stream",
       "text": [
        "Estimated time     : 3 min\n",
        "Experiment Complete\n",
        "\n",
        "\n",
        "Comparing Experiments\n",
        "Comparison ID:      Comparison_analyse_learning_rates\n",
        "Generated statistics post all epochs\n",
        "| Experiment Name      |   Train Acc |   Val Acc |   Train Loss |   Val Loss |\n",
        "|----------------------+-------------+-----------+--------------+------------|\n",
        "| Learning_Rate_0.1    |   0.0746437 | 0.0639731 |     5.10316  |    5.27679 |\n",
        "| Learning_Rate_0.05   |   0.358965  | 0.23569   |     2.78242  |    3.77081 |\n",
        "| Learning_Rate_0.01   |   0.978995  | 0.47138   |     0.2577   |    2.3881  |\n",
        "| Learning_Rate_0.005  |   0.945611  | 0.491582  |     0.599715 |    2.53947 |\n",
        "| Learning_Rate_0.0001 |   0.0982746 | 0.0808081 |     5.119    |    5.32924 |\n",
        "\n"
       ]
      },
      {
       "data": {
        "text/plain": [
         "<Figure size 432x288 with 0 Axes>"
        ]
       },
       "metadata": {},
       "output_type": "display_data"
      },
      {
       "data": {
        "text/plain": [
         "<Figure size 1440x720 with 0 Axes>"
        ]
       },
       "metadata": {},
       "output_type": "display_data"
      },
      {
       "data": {
        "text/plain": [
         "<Figure size 1440x720 with 0 Axes>"
        ]
       },
       "metadata": {},
       "output_type": "display_data"
      },
      {
       "data": {
        "text/plain": [
         "<Figure size 1440x720 with 0 Axes>"
        ]
       },
       "metadata": {},
       "output_type": "display_data"
      },
      {
       "data": {
        "text/plain": [
         "<Figure size 1440x720 with 0 Axes>"
        ]
       },
       "metadata": {},
       "output_type": "display_data"
      }
     ],
     "source": [
      "# \"keep_all\" - Keep all the sub experiments created\n",
      "# \"keep_non\" - Delete all sub experiments created\n",
      "analysis = gtf.Analyse_Learning_Rates(analysis_name, lrs, percent_data, \n",
      "                                      num_epochs=epochs, state=\"keep_none\"); "
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "## Analysis\n",
      "\n",
      "    - LR as 0.1 doesnt work\n",
      "    - Same is the case with 0.0001\n",
      "    \n",
      "    - Of the other's lr as 0.01 produces least validation loss"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "## Update learning rate"
     ]
    },
    {
     "cell_type": "code",
     "execution_count": 10,
     "metadata": {},
     "outputs": [
      {
       "name": "stdout",
       "output_type": "stream",
       "text": [
        "Update: Learning Rate - 0.01\n",
        "\n"
       ]
      },
      {
       "name": "stderr",
       "output_type": "stream",
       "text": [
        "/home/abhi/.virtualenvs/finetune_py36/lib/python3.6/site-packages/mxnet/gluon/data/vision/datasets.py:312: UserWarning: Ignoring caltech256/train/198.spider/RENAME2 of type . Only support .jpg, .jpeg, .png\n",
        "  filename, ext, ', '.join(self._exts)))\n"
       ]
      },
      {
       "name": "stdout",
       "output_type": "stream",
       "text": [
        "Pre-Composed Train Transforms\n",
        "[{'RandomHorizontalFlip': {'p': 0.8}}, {'Normalize': {'mean': [0.485, 0.456, 0.406], 'std': [0.229, 0.224, 0.225]}}]\n",
        "\n",
        "Pre-Composed Val Transforms\n",
        "[{'RandomHorizontalFlip': {'p': 0.8}}, {'Normalize': {'mean': [0.485, 0.456, 0.406], 'std': [0.229, 0.224, 0.225]}}]\n",
        "\n",
        "Dataset Numbers\n",
        "    Num train images: 21424\n",
        "    Num val images:   9183\n",
        "    Num classes:      257\n",
        "\n",
        "Model Details\n",
        "    Loading pretrained model\n",
        "    Model Loaded on device\n",
        "        Model name:                           resnet18_v1\n",
        "        Num of potentially trainable layers:  41\n",
        "        Num of actual trainable layers:       41\n",
        "\n"
       ]
      }
     ],
     "source": [
      "gtf.update_learning_rate(0.01);\n",
      "\n",
      "# Very important to reload post updates\n",
      "gtf.Reload();"
     ]
    },
    {
     "cell_type": "code",
     "execution_count": null,
     "metadata": {},
     "outputs": [],
     "source": [
      "#Start Training\n",
      "gtf.Train();\n",
      "\n",
      "#Read the training summary generated once you run the cell and training is completed"
     ]
    },
    {
     "cell_type": "code",
     "execution_count": null,
     "metadata": {},
     "outputs": [],
     "source": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "# Goals Completed\n",
      "\n",
      "### Learn how to use hyper parameter analyser for learning rates"
     ]
    },
    {
     "cell_type": "code",
     "execution_count": null,
     "metadata": {},
     "outputs": [],
     "source": []
    }
   ],
   "metadata": {
    "kernelspec": {
     "display_name": "Python 3",
     "language": "python",
     "name": "python3"
    },
    "language_info": {
     "codemirror_mode": {
      "name": "ipython",
      "version": 3
     },
     "file_extension": ".py",
     "mimetype": "text/x-python",
     "name": "python",
     "nbconvert_exporter": "python",
     "pygments_lexer": "ipython3",
     "version": "3.6.9"
    }
   },
   "nbformat": 4,
   "nbformat_minor": 2
  }
  