{ "cells": [ { "cell_type": "markdown", "metadata": { "id": "eBpjBBZc6IvA" }, "source": [ "# Fatima Fellowship Quick Coding Challenge (Pick 1)\n", "\n", "Thank you for applying to the Fatima Fellowship. To help us select the Fellows and assess your ability to do machine learning research, we are asking that you complete a short coding challenge. Please pick **1 of these 5** coding challenges, whichever is most aligned with your interests. \n", "\n", "**Due date: 1 week**\n", "\n", "**How to submit**: Please make a copy of this colab notebook, add your code and results, and submit your colab notebook to the submission link below. If you have never used a colab notebook, [check out this video](https://www.youtube.com/watch?v=i-HnvsehuSw).\n", "\n", "**Submission link**: https://airtable.com/shrXy3QKSsO2yALd3" ] }, { "cell_type": "markdown", "metadata": { "id": "braBzmRpMe7_" }, "source": [ "# 1. Deep Learning for Vision" ] }, { "cell_type": "markdown", "metadata": { "id": "1IWw-NZf5WfF" }, "source": [ "**Upside down detector**: Train a model to detect if images are upside down\n", "\n", "* Pick a dataset of natural images (we suggest looking at datasets on the [Hugging Face Hub](https://huggingface.co/datasets?task_categories=task_categories:image-classification&sort=downloads))\n", "* Synthetically turn some of images upside down. Create a training and test set.\n", "* Build a neural network (using Tensorflow, PyTorch, or any framework you like)\n", "* Train it to classify image orientation until a reasonable accuracy is reached\n", "* [Upload the the model to the Hugging Face Hub](https://huggingface.co/docs/hub/adding-a-model), and add a link to your model below.\n", "* Look at some of the images that were classified incorrectly. Please explain what you might do to improve your model's performance on these images in the future (you do not need to impelement these suggestions)\n", "\n", "**Submission instructions**: Please write your code below and include some examples of images that were classified" ] }, { "cell_type": "markdown", "metadata": { "id": "sFU9LTOyMiMj" }, "source": [ "# 2. Deep Learning for NLP\n", "\n", "**Fake news classifier**: Train a text classification model to detect fake news articles!\n", "\n", "* Download the dataset here: https://www.kaggle.com/clmentbisaillon/fake-and-real-news-dataset\n", "* Develop an NLP model for classification that uses a pretrained language model\n", "* Finetune your model on the dataset, and generate an AUC curve of your model on the test set of your choice. \n", "* [Upload the the model to the Hugging Face Hub](https://huggingface.co/docs/hub/adding-a-model), and add a link to your model below.\n", "* *Answer the following question*: Look at some of the news articles that were classified incorrectly. Please explain what you might do to improve your model's performance on these news articles in the future (you do not need to impelement these suggestions)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "K2GJaYBpw91T" }, "outputs": [], "source": [ "### WRITE YOUR CODE TO TRAIN THE MODEL HERE" ] }, { "cell_type": "markdown", "source": [ "Connecting to drive" ], "metadata": { "id": "_7iHl1-udgTx" } }, { "cell_type": "code", "execution_count": 1, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "_aryZSHzhGJJ", "outputId": "00b7a877-1745-4d52-f440-88e01ae4f70c" }, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "Mounted at /root/gdrive\n" ] } ], "source": [ "from google.colab import drive\n", "drive.mount('/root/gdrive', force_remount=True)" ] }, { "cell_type": "markdown", "source": [ "Going To Root directory where the dataset is stored" ], "metadata": { "id": "qu1CPJC_djfp" } }, { "cell_type": "code", "execution_count": 2, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "U1k3q6TDhd3Q", "outputId": "61b6bed0-e298-4497-fe60-f242ab1930be" }, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "/root/gdrive/MyDrive/Fake news detection\n" ] } ], "source": [ "%cd /root/gdrive/MyDrive/Fake news detection" ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "id": "NBta24m-hhSf" }, "outputs": [], "source": [ "import numpy as np\n", "import pandas as pd\n", "import matplotlib.pyplot as plt" ] }, { "cell_type": "markdown", "source": [ "Data preprocessing" ], "metadata": { "id": "d4BNzjBPduyw" } }, { "cell_type": "code", "execution_count": 4, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "p9-2ihR3hmdv", "outputId": "8f95258c-44b3-47c7-ac65-22afab6c31fa" }, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "(23481, 4) (21417, 4)\n" ] } ], "source": [ "fake=pd.read_csv('Fake.csv')\n", "true=pd.read_csv('True.csv')\n", "print(fake.shape, true.shape)" ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "0ioPOytHiP5h", "outputId": "44ab72e9-dca1-4496-9bb0-a1ab1c4282c8" }, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "(23481, 3) (21417, 3)\n" ] } ], "source": [ "fake.drop(['date', 'subject'], axis=1, inplace=True)\n", "true.drop(['date', 'subject'], axis=1, inplace=True)\n", "fake['class'] = 0 \n", "true['class'] = 1\n", "print(fake.shape, true.shape)" ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "Tndd2AbQjIgW", "outputId": "69e99818-bbb4-43c6-e5f6-5bd70885f859" }, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "(23481, 2) (21417, 2)\n" ] } ], "source": [ "fake['text'] = fake['title'] + fake['text']\n", "true['text'] = true['title'] + true['text']\n", "fake.drop('title', axis=1, inplace=True)\n", "true.drop('title', axis=1, inplace=True)\n", "print(fake.shape, true.shape)" ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "Aew_uoyUjm9d", "outputId": "3b4fbdc4-e857-4fd4-fcf0-7e1525d6a940" }, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "(44898, 2) text class\n", "30805 A discontented Michigan town: America should h... 1\n", "41251 Iran president defends Guards in show of unity... 1\n", "3629 White Man Dedicates His Racist Attack To Trum... 0\n", "24764 Trump healthcare order could run afoul of reti... 1\n", "23861 How McConnell kept Republicans in line to win ... 1\n" ] } ], "source": [ "data = pd.concat([fake, true], ignore_index=True, sort=False).sample(frac=1)\n", "print(data.shape,data.head())" ] }, { "cell_type": "markdown", "source": [ "Working for only 2000 data because of google colab RAM and disk issue" ], "metadata": { "id": "tscAcwbUdx6e" } }, { "cell_type": "code", "execution_count": 8, "metadata": { "id": "qCiZOCV9OCMk" }, "outputs": [], "source": [ "data = data[:2000]" ] }, { "cell_type": "code", "execution_count": 9, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "uzCBGsNHkS99", "outputId": "f27b9740-cdfe-4b63-e11e-1b70dbe6b51b" }, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "(2000,) \n" ] } ], "source": [ "x_text = data['text'].values.tolist()\n", "y_label = data['class'].values.tolist()\n", "print(np.shape(x_text), type(x_text))" ] }, { "cell_type": "markdown", "source": [ "Splitting into train-test set" ], "metadata": { "id": "DOGw_4owd8Vp" } }, { "cell_type": "code", "execution_count": 10, "metadata": { "id": "KX87qfSTloAX" }, "outputs": [], "source": [ "from sklearn.model_selection import train_test_split\n", "X_train, X_test, y_train, y_test = train_test_split(x_text,y_label, stratify=y_label)" ] }, { "cell_type": "code", "execution_count": 11, "metadata": { "id": "MITQpsrLltCs" }, "outputs": [], "source": [ "import tensorflow as tf\n", "from tensorflow.keras.optimizers import Adam\n", "from tensorflow.keras.callbacks import EarlyStopping\n", "from tensorflow.keras.initializers import TruncatedNormal\n", "from tensorflow.keras.losses import CategoricalCrossentropy\n", "from tensorflow.keras.metrics import CategoricalAccuracy\n", "from tensorflow.keras.utils import to_categorical\n", "from tensorflow.keras.layers import Input, Dense" ] }, { "cell_type": "markdown", "source": [ "Library installing for XLMRoBERTa model" ], "metadata": { "id": "ehs-ojoseBwh" } }, { "cell_type": "code", "execution_count": 12, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "yUzZk66Ilv6b", "outputId": "82cd4378-01e9-4898-f099-791a84832ab8" }, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "Collecting transformers\n", " Downloading transformers-4.17.0-py3-none-any.whl (3.8 MB)\n", "\u001b[K |████████████████████████████████| 3.8 MB 5.0 MB/s \n", "\u001b[?25hRequirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.7/dist-packages (from transformers) (4.63.0)\n", "Collecting sacremoses\n", " Downloading sacremoses-0.0.49-py3-none-any.whl (895 kB)\n", "\u001b[K |████████████████████████████████| 895 kB 45.7 MB/s \n", "\u001b[?25hCollecting tokenizers!=0.11.3,>=0.11.1\n", " Downloading tokenizers-0.11.6-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (6.5 MB)\n", "\u001b[K |████████████████████████████████| 6.5 MB 30.8 MB/s \n", "\u001b[?25hRequirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.7/dist-packages (from transformers) (2019.12.20)\n", "Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.7/dist-packages (from transformers) (1.21.5)\n", "Collecting huggingface-hub<1.0,>=0.1.0\n", " Downloading huggingface_hub-0.4.0-py3-none-any.whl (67 kB)\n", "\u001b[K |████████████████████████████████| 67 kB 4.9 MB/s \n", "\u001b[?25hRequirement already satisfied: importlib-metadata in /usr/local/lib/python3.7/dist-packages (from transformers) (4.11.3)\n", "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.7/dist-packages (from transformers) (21.3)\n", "Requirement already satisfied: requests in /usr/local/lib/python3.7/dist-packages (from transformers) (2.23.0)\n", "Requirement already satisfied: filelock in /usr/local/lib/python3.7/dist-packages (from transformers) (3.6.0)\n", "Collecting pyyaml>=5.1\n", " Downloading PyYAML-6.0-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (596 kB)\n", "\u001b[K |████████████████████████████████| 596 kB 46.4 MB/s \n", "\u001b[?25hRequirement already satisfied: typing-extensions>=3.7.4.3 in /usr/local/lib/python3.7/dist-packages (from huggingface-hub<1.0,>=0.1.0->transformers) (3.10.0.2)\n", "Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.7/dist-packages (from packaging>=20.0->transformers) (3.0.7)\n", "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata->transformers) (3.7.0)\n", "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests->transformers) (3.0.4)\n", "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests->transformers) (1.24.3)\n", "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests->transformers) (2.10)\n", "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests->transformers) (2021.10.8)\n", "Requirement already satisfied: joblib in /usr/local/lib/python3.7/dist-packages (from sacremoses->transformers) (1.1.0)\n", "Requirement already satisfied: six in /usr/local/lib/python3.7/dist-packages (from sacremoses->transformers) (1.15.0)\n", "Requirement already satisfied: click in /usr/local/lib/python3.7/dist-packages (from sacremoses->transformers) (7.1.2)\n", "Installing collected packages: pyyaml, tokenizers, sacremoses, huggingface-hub, transformers\n", " Attempting uninstall: pyyaml\n", " Found existing installation: PyYAML 3.13\n", " Uninstalling PyYAML-3.13:\n", " Successfully uninstalled PyYAML-3.13\n", "Successfully installed huggingface-hub-0.4.0 pyyaml-6.0 sacremoses-0.0.49 tokenizers-0.11.6 transformers-4.17.0\n" ] } ], "source": [ "!pip install transformers\n", "import transformers" ] }, { "cell_type": "code", "execution_count": 13, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "NEvK8sEhmWRi", "outputId": "bc07a4de-5b3e-4260-d459-b931757c2066" }, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "Collecting sentencepiece\n", " Downloading sentencepiece-0.1.96-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.2 MB)\n", "\u001b[?25l\r\u001b[K |▎ | 10 kB 25.5 MB/s eta 0:00:01\r\u001b[K |▌ | 20 kB 17.9 MB/s eta 0:00:01\r\u001b[K |▉ | 30 kB 11.6 MB/s eta 0:00:01\r\u001b[K |█ | 40 kB 9.8 MB/s eta 0:00:01\r\u001b[K |█▍ | 51 kB 4.4 MB/s eta 0:00:01\r\u001b[K |█▋ | 61 kB 5.2 MB/s eta 0:00:01\r\u001b[K |██ | 71 kB 5.8 MB/s eta 0:00:01\r\u001b[K |██▏ | 81 kB 5.9 MB/s eta 0:00:01\r\u001b[K |██▍ | 92 kB 6.6 MB/s eta 0:00:01\r\u001b[K |██▊ | 102 kB 5.3 MB/s eta 0:00:01\r\u001b[K |███ | 112 kB 5.3 MB/s eta 0:00:01\r\u001b[K |███▎ | 122 kB 5.3 MB/s eta 0:00:01\r\u001b[K |███▌ | 133 kB 5.3 MB/s eta 0:00:01\r\u001b[K |███▉ | 143 kB 5.3 MB/s eta 0:00:01\r\u001b[K |████ | 153 kB 5.3 MB/s eta 0:00:01\r\u001b[K |████▎ | 163 kB 5.3 MB/s eta 0:00:01\r\u001b[K |████▋ | 174 kB 5.3 MB/s eta 0:00:01\r\u001b[K |████▉ | 184 kB 5.3 MB/s eta 0:00:01\r\u001b[K |█████▏ | 194 kB 5.3 MB/s eta 0:00:01\r\u001b[K |█████▍ | 204 kB 5.3 MB/s eta 0:00:01\r\u001b[K |█████▊ | 215 kB 5.3 MB/s eta 0:00:01\r\u001b[K |██████ | 225 kB 5.3 MB/s eta 0:00:01\r\u001b[K |██████▏ | 235 kB 5.3 MB/s eta 0:00:01\r\u001b[K |██████▌ | 245 kB 5.3 MB/s eta 0:00:01\r\u001b[K |██████▊ | 256 kB 5.3 MB/s eta 0:00:01\r\u001b[K |███████ | 266 kB 5.3 MB/s eta 0:00:01\r\u001b[K |███████▎ | 276 kB 5.3 MB/s eta 0:00:01\r\u001b[K |███████▋ | 286 kB 5.3 MB/s eta 0:00:01\r\u001b[K |███████▉ | 296 kB 5.3 MB/s eta 0:00:01\r\u001b[K |████████ | 307 kB 5.3 MB/s eta 0:00:01\r\u001b[K |████████▍ | 317 kB 5.3 MB/s eta 0:00:01\r\u001b[K |████████▋ | 327 kB 5.3 MB/s eta 0:00:01\r\u001b[K |█████████ | 337 kB 5.3 MB/s eta 0:00:01\r\u001b[K |█████████▏ | 348 kB 5.3 MB/s eta 0:00:01\r\u001b[K |█████████▌ | 358 kB 5.3 MB/s eta 0:00:01\r\u001b[K |█████████▊ | 368 kB 5.3 MB/s eta 0:00:01\r\u001b[K |██████████ | 378 kB 5.3 MB/s eta 0:00:01\r\u001b[K |██████████▎ | 389 kB 5.3 MB/s eta 0:00:01\r\u001b[K |██████████▌ | 399 kB 5.3 MB/s eta 0:00:01\r\u001b[K |██████████▉ | 409 kB 5.3 MB/s eta 0:00:01\r\u001b[K |███████████ | 419 kB 5.3 MB/s eta 0:00:01\r\u001b[K |███████████▍ | 430 kB 5.3 MB/s eta 0:00:01\r\u001b[K |███████████▋ | 440 kB 5.3 MB/s eta 0:00:01\r\u001b[K |███████████▉ | 450 kB 5.3 MB/s eta 0:00:01\r\u001b[K |████████████▏ | 460 kB 5.3 MB/s eta 0:00:01\r\u001b[K |████████████▍ | 471 kB 5.3 MB/s eta 0:00:01\r\u001b[K |████████████▊ | 481 kB 5.3 MB/s eta 0:00:01\r\u001b[K |█████████████ | 491 kB 5.3 MB/s eta 0:00:01\r\u001b[K |█████████████▎ | 501 kB 5.3 MB/s eta 0:00:01\r\u001b[K |█████████████▌ | 512 kB 5.3 MB/s eta 0:00:01\r\u001b[K |█████████████▊ | 522 kB 5.3 MB/s eta 0:00:01\r\u001b[K |██████████████ | 532 kB 5.3 MB/s eta 0:00:01\r\u001b[K |██████████████▎ | 542 kB 5.3 MB/s eta 0:00:01\r\u001b[K |██████████████▋ | 552 kB 5.3 MB/s eta 0:00:01\r\u001b[K |██████████████▉ | 563 kB 5.3 MB/s eta 0:00:01\r\u001b[K |███████████████▏ | 573 kB 5.3 MB/s eta 0:00:01\r\u001b[K |███████████████▍ | 583 kB 5.3 MB/s eta 0:00:01\r\u001b[K |███████████████▋ | 593 kB 5.3 MB/s eta 0:00:01\r\u001b[K |████████████████ | 604 kB 5.3 MB/s eta 0:00:01\r\u001b[K |████████████████▏ | 614 kB 5.3 MB/s eta 0:00:01\r\u001b[K |████████████████▌ | 624 kB 5.3 MB/s eta 0:00:01\r\u001b[K |████████████████▊ | 634 kB 5.3 MB/s eta 0:00:01\r\u001b[K |█████████████████ | 645 kB 5.3 MB/s eta 0:00:01\r\u001b[K |█████████████████▎ | 655 kB 5.3 MB/s eta 0:00:01\r\u001b[K |█████████████████▌ | 665 kB 5.3 MB/s eta 0:00:01\r\u001b[K |█████████████████▉ | 675 kB 5.3 MB/s eta 0:00:01\r\u001b[K |██████████████████ | 686 kB 5.3 MB/s eta 0:00:01\r\u001b[K |██████████████████▍ | 696 kB 5.3 MB/s eta 0:00:01\r\u001b[K |██████████████████▋ | 706 kB 5.3 MB/s eta 0:00:01\r\u001b[K |███████████████████ | 716 kB 5.3 MB/s eta 0:00:01\r\u001b[K |███████████████████▏ | 727 kB 5.3 MB/s eta 0:00:01\r\u001b[K |███████████████████▍ | 737 kB 5.3 MB/s eta 0:00:01\r\u001b[K |███████████████████▊ | 747 kB 5.3 MB/s eta 0:00:01\r\u001b[K |████████████████████ | 757 kB 5.3 MB/s eta 0:00:01\r\u001b[K |████████████████████▎ | 768 kB 5.3 MB/s eta 0:00:01\r\u001b[K |████████████████████▌ | 778 kB 5.3 MB/s eta 0:00:01\r\u001b[K |████████████████████▉ | 788 kB 5.3 MB/s eta 0:00:01\r\u001b[K |█████████████████████ | 798 kB 5.3 MB/s eta 0:00:01\r\u001b[K |█████████████████████▎ | 808 kB 5.3 MB/s eta 0:00:01\r\u001b[K |█████████████████████▋ | 819 kB 5.3 MB/s eta 0:00:01\r\u001b[K |█████████████████████▉ | 829 kB 5.3 MB/s eta 0:00:01\r\u001b[K |██████████████████████▏ | 839 kB 5.3 MB/s eta 0:00:01\r\u001b[K |██████████████████████▍ | 849 kB 5.3 MB/s eta 0:00:01\r\u001b[K |██████████████████████▊ | 860 kB 5.3 MB/s eta 0:00:01\r\u001b[K |███████████████████████ | 870 kB 5.3 MB/s eta 0:00:01\r\u001b[K |███████████████████████▏ | 880 kB 5.3 MB/s eta 0:00:01\r\u001b[K |███████████████████████▌ | 890 kB 5.3 MB/s eta 0:00:01\r\u001b[K |███████████████████████▊ | 901 kB 5.3 MB/s eta 0:00:01\r\u001b[K |████████████████████████ | 911 kB 5.3 MB/s eta 0:00:01\r\u001b[K |████████████████████████▎ | 921 kB 5.3 MB/s eta 0:00:01\r\u001b[K |████████████████████████▋ | 931 kB 5.3 MB/s eta 0:00:01\r\u001b[K |████████████████████████▉ | 942 kB 5.3 MB/s eta 0:00:01\r\u001b[K |█████████████████████████ | 952 kB 5.3 MB/s eta 0:00:01\r\u001b[K |█████████████████████████▍ | 962 kB 5.3 MB/s eta 0:00:01\r\u001b[K |█████████████████████████▋ | 972 kB 5.3 MB/s eta 0:00:01\r\u001b[K |██████████████████████████ | 983 kB 5.3 MB/s eta 0:00:01\r\u001b[K |██████████████████████████▏ | 993 kB 5.3 MB/s eta 0:00:01\r\u001b[K |██████████████████████████▌ | 1.0 MB 5.3 MB/s eta 0:00:01\r\u001b[K |██████████████████████████▊ | 1.0 MB 5.3 MB/s eta 0:00:01\r\u001b[K |███████████████████████████ | 1.0 MB 5.3 MB/s eta 0:00:01\r\u001b[K |███████████████████████████▎ | 1.0 MB 5.3 MB/s eta 0:00:01\r\u001b[K |███████████████████████████▌ | 1.0 MB 5.3 MB/s eta 0:00:01\r\u001b[K |███████████████████████████▉ | 1.1 MB 5.3 MB/s eta 0:00:01\r\u001b[K |████████████████████████████ | 1.1 MB 5.3 MB/s eta 0:00:01\r\u001b[K |████████████████████████████▍ | 1.1 MB 5.3 MB/s eta 0:00:01\r\u001b[K |████████████████████████████▋ | 1.1 MB 5.3 MB/s eta 0:00:01\r\u001b[K |████████████████████████████▉ | 1.1 MB 5.3 MB/s eta 0:00:01\r\u001b[K |█████████████████████████████▏ | 1.1 MB 5.3 MB/s eta 0:00:01\r\u001b[K |█████████████████████████████▍ | 1.1 MB 5.3 MB/s eta 0:00:01\r\u001b[K |█████████████████████████████▊ | 1.1 MB 5.3 MB/s eta 0:00:01\r\u001b[K |██████████████████████████████ | 1.1 MB 5.3 MB/s eta 0:00:01\r\u001b[K |██████████████████████████████▎ | 1.1 MB 5.3 MB/s eta 0:00:01\r\u001b[K |██████████████████████████████▌ | 1.2 MB 5.3 MB/s eta 0:00:01\r\u001b[K |██████████████████████████████▊ | 1.2 MB 5.3 MB/s eta 0:00:01\r\u001b[K |███████████████████████████████ | 1.2 MB 5.3 MB/s eta 0:00:01\r\u001b[K |███████████████████████████████▎| 1.2 MB 5.3 MB/s eta 0:00:01\r\u001b[K |███████████████████████████████▋| 1.2 MB 5.3 MB/s eta 0:00:01\r\u001b[K |███████████████████████████████▉| 1.2 MB 5.3 MB/s eta 0:00:01\r\u001b[K |████████████████████████████████| 1.2 MB 5.3 MB/s \n", "\u001b[?25hInstalling collected packages: sentencepiece\n", "Successfully installed sentencepiece-0.1.96\n" ] } ], "source": [ "!pip install sentencepiece" ] }, { "cell_type": "markdown", "source": [ "Loading pretrained XLM Roberta tokenizer and Encoder" ], "metadata": { "id": "5dMy0niueKJi" } }, { "cell_type": "code", "execution_count": 14, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 220, "referenced_widgets": [ "c9d66cc4ead940a1babe93fe3011abe2", "bf4ea0b231e44d75abee70860652f6da", "c3b86577ff2d49eaad2785112d107e95", "86e941cca4c148a5a8fb3faed3e1c058", "bd7fdc49334142818f3fd69c467b3665", "78baedc6e7a0480aa824576ec549a7ff", "e0e3f3fe05b1414088d93c0a0f32db02", "4f7ade9b563249f99dad7d9e36bdbba3", "de92d95bb4954949bd6a051b595fb688", "64b2f8229c4d406a95ec2054bcae1ebf", "966c0379ec81464d97cf165d3ddc05bc", "5620a2691dc24031a5db44bf549f2e6a", "174a6b221a9346809b334a0690732e5a", "2e9b0ba33faa4ab6957ffbda64e8e35f", "ab7609a1a4c0436bb5c8bd35884504f0", "b96db63588f3454e910ab6457e2a67d9", "453493c9d9854e2283e68a9fb0dbe8b6", "2eed6379fe3e4b3b8c483a78da05a4bc", "4b9f7cd7403d4499b7360d2d1fce57c5", "f6182cd27c8040089a65be7e84ef3403", "8f78b817454d4fd2a30b1a7b8d89b57f", "fdfef297b3d44d1486025664c257afd6", "40aa7f9c6a894d26bdee2352ec269ddb", "4c8162e8f9264b33bb2bb625c764fe85", "1c78711b6c9a476fa140d54637c5cc51", "8e2eb87c9572426abb0f47883b5849a9", "6f87ef566e644da784f889d617b455eb", "303792e631314bb0b9500cd59f338bbf", "c6261873f8d1455fa8ac6a5968383ece", "ec257f57f17e421490440b2202941db4", "a78f48f50f994746b0343efda278bf43", "b8b252e5acf74dba83fdeb21323d6296", "5457220bdb004116b87e596ab725a1b6" ] }, "id": "w4F2LfWpmXEm", "outputId": "d356c0d8-000a-45b5-a1f3-1cffddd8162f" }, "outputs": [ { "output_type": "display_data", "data": { "text/plain": [ "Downloading: 0%| | 0.00/512 [00:00\n", "[1 0 0 0 0 0 1 0 0 1 0 1 0 0 0 1 0 0 0 1 1 1 1 0 0 0 0 0 1 1 1 1 0 0 0 1 1\n", " 1 0 0 1 0 0 1 1 1 0 1 1 1 0 1 0 1 0 0 0 0 1 1 1 0 0 0 1 1 1 0 1 0 1 0 1 0\n", " 1 1 0 0 0 1 1 0 1 0 0 1 0 1 0 0 1 1 0 1 0 1 1 1 1 1 0 1 1 0 0 0 0 1 1 1 0\n", " 1 1 0 0 0 1 1 0 1 0 1 1 1 0 1 0 0 1 0 1 0 0 1 0 1 1 1 1 1 0 0 0 1 0 1 1 1\n", " 0 1 0 0 1 0 0 0 0 1 0 0 0 1 1 0 0 0 1 1 0 1 0 0 0 0 1 1 0 0 1 1 0 0 1 0 1\n", " 0 0 0 1 0 1 0 1 1 1 1 0 0 0 0 1 0 1 1 0 1 1 0 1 0 1 1 0 0 1 0 0 1 0 0 1 1\n", " 1 1 1 1 0 0 0 1 1 0 0 1 0 0 0 0 0 0 0 1 1 1 0 1 1 0 1 0 1 1 1 0 1 1 0 1 1\n", " 0 0 1 0 0 1 0 0 0 1 0 0 1 1 0 1 1 0 0 0 0 1 0 0 1 0 0 0 1 1 0 0 0 0 1 0 0\n", " 1 0 1 0 0 0 0 1 1 1 0 0 1 1 0 1 1 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 1 1 0 0 0\n", " 0 1 0 1 1 0 1 0 0 1 0 0 1 0 0 1 1 1 0 1 1 0 0 0 0 1 1 1 1 1 1 0 1 1 1 1 1\n", " 0 1 1 0 1 0 1 1 1 1 0 0 1 1 1 1 1 0 0 1 1 0 0 1 0 0 0 0 0 1 0 1 0 0 0 1 1\n", " 1 0 0 0 0 0 0 0 0 1 1 1 0 0 0 0 1 1 1 0 1 1 0 0 1 1 0 0 1 1 0 1 1 0 0 0 0\n", " 1 1 0 1 0 1 0 1 1 0 1 0 1 1 0 1 1 0 1 0 0 0 0 1 0 1 1 0 0 1 1 0 1 1 0 0 1\n", " 1 0 0 0 0 1 1 0 1 1 1 1 1 1 1 0 1 0 1] 500 (500,)\n", "{0, 1}\n", "[[0. 1.]\n", " [1. 0.]\n", " [1. 0.]\n", " [1. 0.]\n", " [1. 0.]\n", " [1. 0.]\n", " [0. 1.]\n", " [1. 0.]\n", " [1. 0.]\n", " [0. 1.]\n", " [1. 0.]\n", " [0. 1.]\n", " [1. 0.]\n", " [1. 0.]\n", " [1. 0.]\n", " [0. 1.]\n", " [1. 0.]\n", " [1. 0.]\n", " [1. 0.]\n", " [0. 1.]\n", " [0. 1.]\n", " [0. 1.]\n", " [0. 1.]\n", " [1. 0.]\n", " [1. 0.]\n", " [1. 0.]\n", " [1. 0.]\n", " [1. 0.]\n", " [0. 1.]\n", " [0. 1.]\n", " [0. 1.]\n", " [0. 1.]\n", " [1. 0.]\n", " [1. 0.]\n", " [1. 0.]\n", " [0. 1.]\n", " [0. 1.]\n", " [0. 1.]\n", " [1. 0.]\n", " [1. 0.]\n", " [0. 1.]\n", " [1. 0.]\n", " [1. 0.]\n", " [0. 1.]\n", " [0. 1.]\n", " [0. 1.]\n", " [1. 0.]\n", " [0. 1.]\n", " [0. 1.]\n", " [0. 1.]\n", " [1. 0.]\n", " [0. 1.]\n", " [1. 0.]\n", " [0. 1.]\n", " [1. 0.]\n", " [1. 0.]\n", " [1. 0.]\n", " [1. 0.]\n", " [0. 1.]\n", " [0. 1.]\n", " [0. 1.]\n", " [1. 0.]\n", " [1. 0.]\n", " [1. 0.]\n", " [0. 1.]\n", " [0. 1.]\n", " [0. 1.]\n", " [1. 0.]\n", " [0. 1.]\n", " [1. 0.]\n", " [0. 1.]\n", " [1. 0.]\n", " [0. 1.]\n", " [1. 0.]\n", " [0. 1.]\n", " [0. 1.]\n", " [1. 0.]\n", " [1. 0.]\n", " [1. 0.]\n", " [0. 1.]\n", " [0. 1.]\n", " [1. 0.]\n", " [0. 1.]\n", " [1. 0.]\n", " [1. 0.]\n", " [0. 1.]\n", " [1. 0.]\n", " [0. 1.]\n", " [1. 0.]\n", " [1. 0.]\n", " [0. 1.]\n", " [0. 1.]\n", " [1. 0.]\n", " [0. 1.]\n", " [1. 0.]\n", " [0. 1.]\n", " [0. 1.]\n", " [0. 1.]\n", " [0. 1.]\n", " [0. 1.]\n", " [1. 0.]\n", " [0. 1.]\n", " [0. 1.]\n", " [1. 0.]\n", " [1. 0.]\n", " [1. 0.]\n", " [1. 0.]\n", " [0. 1.]\n", " [0. 1.]\n", " [0. 1.]\n", " [1. 0.]\n", " [0. 1.]\n", " [0. 1.]\n", " [1. 0.]\n", " [1. 0.]\n", " [1. 0.]\n", " [0. 1.]\n", " [0. 1.]\n", " [1. 0.]\n", " [0. 1.]\n", " [1. 0.]\n", " [0. 1.]\n", " [0. 1.]\n", " [0. 1.]\n", " [1. 0.]\n", " [0. 1.]\n", " [1. 0.]\n", " [1. 0.]\n", " [0. 1.]\n", " [1. 0.]\n", " [0. 1.]\n", " [1. 0.]\n", " [1. 0.]\n", " [0. 1.]\n", " [1. 0.]\n", " [0. 1.]\n", " [0. 1.]\n", " [0. 1.]\n", " [0. 1.]\n", " [0. 1.]\n", " [1. 0.]\n", " [1. 0.]\n", " [1. 0.]\n", " [0. 1.]\n", " [1. 0.]\n", " [0. 1.]\n", " [0. 1.]\n", " [0. 1.]\n", " [1. 0.]\n", " [0. 1.]\n", " [1. 0.]\n", " [1. 0.]\n", " [0. 1.]\n", " [1. 0.]\n", " [1. 0.]\n", " [1. 0.]\n", " [1. 0.]\n", " [0. 1.]\n", " [1. 0.]\n", " [1. 0.]\n", " [1. 0.]\n", " [0. 1.]\n", " [0. 1.]\n", " [1. 0.]\n", " [1. 0.]\n", " [1. 0.]\n", " [0. 1.]\n", " [0. 1.]\n", " [1. 0.]\n", " [0. 1.]\n", " [1. 0.]\n", " [1. 0.]\n", " [1. 0.]\n", " [1. 0.]\n", " [0. 1.]\n", " [0. 1.]\n", " [1. 0.]\n", " [1. 0.]\n", " [0. 1.]\n", " [0. 1.]\n", " [1. 0.]\n", " [1. 0.]\n", " [0. 1.]\n", " [1. 0.]\n", " [0. 1.]\n", " [1. 0.]\n", " [1. 0.]\n", " [1. 0.]\n", " [0. 1.]\n", " [1. 0.]\n", " [0. 1.]\n", " [1. 0.]\n", " [0. 1.]\n", " [0. 1.]\n", " [0. 1.]\n", " [0. 1.]\n", " [1. 0.]\n", " [1. 0.]\n", " [1. 0.]\n", " [1. 0.]\n", " [0. 1.]\n", " [1. 0.]\n", " [0. 1.]\n", " [0. 1.]\n", " [1. 0.]\n", " [0. 1.]\n", " [0. 1.]\n", " [1. 0.]\n", " [0. 1.]\n", " [1. 0.]\n", " [0. 1.]\n", " [0. 1.]\n", " [1. 0.]\n", " [1. 0.]\n", " [0. 1.]\n", " [1. 0.]\n", " [1. 0.]\n", " [0. 1.]\n", " [1. 0.]\n", " [1. 0.]\n", " [0. 1.]\n", " [0. 1.]\n", " [0. 1.]\n", " [0. 1.]\n", " [0. 1.]\n", " [0. 1.]\n", " [1. 0.]\n", " [1. 0.]\n", " [1. 0.]\n", " [0. 1.]\n", " [0. 1.]\n", " [1. 0.]\n", " [1. 0.]\n", " [0. 1.]\n", " [1. 0.]\n", " [1. 0.]\n", " [1. 0.]\n", " [1. 0.]\n", " [1. 0.]\n", " [1. 0.]\n", " [1. 0.]\n", " [0. 1.]\n", " [0. 1.]\n", " [0. 1.]\n", " [1. 0.]\n", " [0. 1.]\n", " [0. 1.]\n", " [1. 0.]\n", " [0. 1.]\n", " [1. 0.]\n", " [0. 1.]\n", " [0. 1.]\n", " [0. 1.]\n", " [1. 0.]\n", " [0. 1.]\n", " [0. 1.]\n", " [1. 0.]\n", " [0. 1.]\n", " [0. 1.]\n", " [1. 0.]\n", " [1. 0.]\n", " [0. 1.]\n", " [1. 0.]\n", " [1. 0.]\n", " [0. 1.]\n", " [1. 0.]\n", " [1. 0.]\n", " [1. 0.]\n", " [0. 1.]\n", " [1. 0.]\n", " [1. 0.]\n", " [0. 1.]\n", " [0. 1.]\n", " [1. 0.]\n", " [0. 1.]\n", " [0. 1.]\n", " [1. 0.]\n", " [1. 0.]\n", " [1. 0.]\n", " [1. 0.]\n", " [0. 1.]\n", " [1. 0.]\n", " [1. 0.]\n", " [0. 1.]\n", " [1. 0.]\n", " [1. 0.]\n", " [1. 0.]\n", " [0. 1.]\n", " [0. 1.]\n", " [1. 0.]\n", " [1. 0.]\n", " [1. 0.]\n", " [1. 0.]\n", " [0. 1.]\n", " [1. 0.]\n", " [1. 0.]\n", " [0. 1.]\n", " [1. 0.]\n", " [0. 1.]\n", " [1. 0.]\n", " [1. 0.]\n", " [1. 0.]\n", " [1. 0.]\n", " [0. 1.]\n", " [0. 1.]\n", " [0. 1.]\n", " [1. 0.]\n", " [1. 0.]\n", " [0. 1.]\n", " [0. 1.]\n", " [1. 0.]\n", " [0. 1.]\n", " [0. 1.]\n", " [1. 0.]\n", " [1. 0.]\n", " [1. 0.]\n", " [1. 0.]\n", " [0. 1.]\n", " [1. 0.]\n", " [1. 0.]\n", " [1. 0.]\n", " [1. 0.]\n", " [1. 0.]\n", " [0. 1.]\n", " [1. 0.]\n", " [1. 0.]\n", " [1. 0.]\n", " [1. 0.]\n", " [0. 1.]\n", " [0. 1.]\n", " [1. 0.]\n", " [1. 0.]\n", " [1. 0.]\n", " [1. 0.]\n", " [0. 1.]\n", " [1. 0.]\n", " [0. 1.]\n", " [0. 1.]\n", " [1. 0.]\n", " [0. 1.]\n", " [1. 0.]\n", " [1. 0.]\n", " [0. 1.]\n", " [1. 0.]\n", " [1. 0.]\n", " [0. 1.]\n", " [1. 0.]\n", " [1. 0.]\n", " [0. 1.]\n", " [0. 1.]\n", " [0. 1.]\n", " [1. 0.]\n", " [0. 1.]\n", " [0. 1.]\n", " [1. 0.]\n", " [1. 0.]\n", " [1. 0.]\n", " [1. 0.]\n", " [0. 1.]\n", " [0. 1.]\n", " [0. 1.]\n", " [0. 1.]\n", " [0. 1.]\n", " [0. 1.]\n", " [1. 0.]\n", " [0. 1.]\n", " [0. 1.]\n", " [0. 1.]\n", " [0. 1.]\n", " [0. 1.]\n", " [1. 0.]\n", " [0. 1.]\n", " [0. 1.]\n", " [1. 0.]\n", " [0. 1.]\n", " [1. 0.]\n", " [0. 1.]\n", " [0. 1.]\n", " [0. 1.]\n", " [0. 1.]\n", " [1. 0.]\n", " [1. 0.]\n", " [0. 1.]\n", " [0. 1.]\n", " [0. 1.]\n", " [0. 1.]\n", " [0. 1.]\n", " [1. 0.]\n", " [1. 0.]\n", " [0. 1.]\n", " [0. 1.]\n", " [1. 0.]\n", " [1. 0.]\n", " [0. 1.]\n", " [1. 0.]\n", " [1. 0.]\n", " [1. 0.]\n", " [1. 0.]\n", " [1. 0.]\n", " [0. 1.]\n", " [1. 0.]\n", " [0. 1.]\n", " [1. 0.]\n", " [1. 0.]\n", " [1. 0.]\n", " [0. 1.]\n", " [0. 1.]\n", " [0. 1.]\n", " [1. 0.]\n", " [1. 0.]\n", " [1. 0.]\n", " [1. 0.]\n", " [1. 0.]\n", " [1. 0.]\n", " [1. 0.]\n", " [1. 0.]\n", " [0. 1.]\n", " [0. 1.]\n", " [0. 1.]\n", " [1. 0.]\n", " [1. 0.]\n", " [1. 0.]\n", " [1. 0.]\n", " [0. 1.]\n", " [0. 1.]\n", " [0. 1.]\n", " [1. 0.]\n", " [0. 1.]\n", " [0. 1.]\n", " [1. 0.]\n", " [1. 0.]\n", " [0. 1.]\n", " [0. 1.]\n", " [1. 0.]\n", " [1. 0.]\n", " [0. 1.]\n", " [0. 1.]\n", " [1. 0.]\n", " [0. 1.]\n", " [0. 1.]\n", " [1. 0.]\n", " [1. 0.]\n", " [1. 0.]\n", " [1. 0.]\n", " [0. 1.]\n", " [0. 1.]\n", " [1. 0.]\n", " [0. 1.]\n", " [1. 0.]\n", " [0. 1.]\n", " [1. 0.]\n", " [0. 1.]\n", " [0. 1.]\n", " [1. 0.]\n", " [0. 1.]\n", " [1. 0.]\n", " [0. 1.]\n", " [0. 1.]\n", " [1. 0.]\n", " [0. 1.]\n", " [0. 1.]\n", " [1. 0.]\n", " [0. 1.]\n", " [1. 0.]\n", " [1. 0.]\n", " [1. 0.]\n", " [1. 0.]\n", " [0. 1.]\n", " [1. 0.]\n", " [0. 1.]\n", " [0. 1.]\n", " [1. 0.]\n", " [1. 0.]\n", " [0. 1.]\n", " [0. 1.]\n", " [1. 0.]\n", " [0. 1.]\n", " [0. 1.]\n", " [1. 0.]\n", " [1. 0.]\n", " [0. 1.]\n", " [0. 1.]\n", " [1. 0.]\n", " [1. 0.]\n", " [1. 0.]\n", " [1. 0.]\n", " [0. 1.]\n", " [0. 1.]\n", " [1. 0.]\n", " [0. 1.]\n", " [0. 1.]\n", " [0. 1.]\n", " [0. 1.]\n", " [0. 1.]\n", " [0. 1.]\n", " [0. 1.]\n", " [1. 0.]\n", " [0. 1.]\n", " [1. 0.]\n", " [0. 1.]] (500, 2)\n", "\n" ] } ], "source": [ "y_train = make_ohe(y_train)\n", "print(type(y_train))\n", "y_test = make_ohe(y_test)\n", "print(type(y_test))" ] }, { "cell_type": "markdown", "source": [ "Classifier model Using XLMRoberta encoder and dense layer" ], "metadata": { "id": "bc1XGhEXeaUf" } }, { "cell_type": "code", "execution_count": 18, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "zCgXsZwCnKvr", "outputId": "68838e26-2ce9-4fa1-9684-efe1488d1eab" }, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "(None, 512, 768)\n", "Model: \"model\"\n", "__________________________________________________________________________________________________\n", " Layer (type) Output Shape Param # Connected to \n", "==================================================================================================\n", " input_ids (InputLayer) [(None, 512)] 0 [] \n", " \n", " attention_mask (InputLayer) [(None, 512)] 0 [] \n", " \n", " tfxlm_roberta_model (TFXLMRobe TFBaseModelOutputWi 278043648 ['input_ids[0][0]', \n", " rtaModel) thPoolingAndCrossAt 'attention_mask[0][0]'] \n", " tentions(last_hidde \n", " n_state=(None, 512, \n", " 768), \n", " pooler_output=(Non \n", " e, 768), \n", " past_key_values=No \n", " ne, hidden_states=N \n", " one, attentions=Non \n", " e, cross_attentions \n", " =None) \n", " \n", " global_max_pooling1d (GlobalMa (None, 768) 0 ['tfxlm_roberta_model[0][0]'] \n", " xPooling1D) \n", " \n", " dense (Dense) (None, 32) 24608 ['global_max_pooling1d[0][0]'] \n", " \n", " dense_1 (Dense) (None, 2) 66 ['dense[0][0]'] \n", " \n", "==================================================================================================\n", "Total params: 278,068,322\n", "Trainable params: 278,068,322\n", "Non-trainable params: 0\n", "__________________________________________________________________________________________________\n" ] } ], "source": [ "max_len = 512\n", "input_ids = Input(shape=(max_len,), dtype=tf.int32, name=\"input_ids\")\n", "input_mask = Input(shape=(max_len,), dtype=tf.int32, name=\"attention_mask\")\n", "embeddings = bert_model(input_ids,attention_mask = input_mask)[0]\n", "print(np.shape(embeddings))\n", "out = tf.keras.layers.GlobalMaxPool1D()(embeddings)\n", "# out = Dense(128, activation='relu')(out)\n", "# out = tf.keras.layers.Dropout(0.1)(out)\n", "out = Dense(32,activation = 'relu')(out)\n", "y = Dense(2, activation='sigmoid')(out)\n", "model = tf.keras.Model(inputs=[input_ids, input_mask], outputs=y)\n", "model.layers[2].trainable = True\n", "model.summary()" ] }, { "cell_type": "code", "execution_count": 19, "metadata": { "id": "O2E1VOhQnSP0" }, "outputs": [], "source": [ "optimizer = Adam(\n", " learning_rate=5e-05, # this learning rate is for bert model , taken from huggingface website \n", " epsilon=1e-08,\n", " decay=0.01,\n", " clipnorm=1.0)\n", "# Set loss and metrics\n", "# loss =CategoricalCrossentropy(from_logits = True)\n", "# metrics = CategoricalAccuracy('balanced_accuracy'),\n", "loss='binary_crossentropy'\n", "metrics=['accuracy']\n", "# Compile the model\n", "model.compile(\n", " optimizer = optimizer,\n", " loss = loss, \n", " metrics = metrics)" ] }, { "cell_type": "markdown", "source": [ "training model & saving checkpoint" ], "metadata": { "id": "S50BGlhUejj4" } }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "background_save": true, "base_uri": "https://localhost:8080/" }, "id": "gJPGGsXo-Ohi", "outputId": "68a0d2d5-295f-4570-8f4a-5a842edcf78f" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Epoch 1/2\n", "WARNING:tensorflow:Gradients do not exist for variables ['tfxlm_roberta_model/roberta/pooler/dense/kernel:0', 'tfxlm_roberta_model/roberta/pooler/dense/bias:0'] when minimizing the loss. If you're using `model.compile()`, did you forget to provide a `loss`argument?\n", "WARNING:tensorflow:Gradients do not exist for variables ['tfxlm_roberta_model/roberta/pooler/dense/kernel:0', 'tfxlm_roberta_model/roberta/pooler/dense/bias:0'] when minimizing the loss. If you're using `model.compile()`, did you forget to provide a `loss`argument?\n", "225/225 [==============================] - 10699s 47s/step - loss: 0.0038 - accuracy: 0.9993 - val_loss: 0.0310 - val_accuracy: 0.9933\n", "Epoch 2/2\n", "174/225 [======================>.......] - ETA: 44:18 - loss: 0.0077 - accuracy: 0.9990" ] } ], "source": [ "checkpoint_filepath = 'checkpoint_bert_base_2k_2'\n", "model_checkpoint_callback = tf.keras.callbacks.ModelCheckpoint(\n", " filepath=checkpoint_filepath,\n", " save_weights_only=True,\n", " monitor='val_accuracy',\n", " mode='max',\n", " save_best_only=True)\n", "early_stop = tf.keras.callbacks.EarlyStopping(monitor='val_loss', patience=2, restore_best_weights=True)\n", "train_history = model.fit(x ={'input_ids':x_train['input_ids'],'attention_mask':x_train['attention_mask']},y= y_train, epochs=2, validation_split=0.1, batch_size=6, shuffle=True, callbacks=[early_stop, model_checkpoint_callback])" ] }, { "cell_type": "markdown", "source": [ "Loading saved weights after 1 epoch" ], "metadata": { "id": "L97Zn5pHequx" } }, { "cell_type": "code", "source": [ "model.load_weights('checkpoint_bert_base_2k_2')" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "d8Cppvhbc-U2", "outputId": "125fcbef-4e57-4b6f-aad8-44d977425f88" }, "execution_count": 21, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "" ] }, "metadata": {}, "execution_count": 21 } ] }, { "cell_type": "markdown", "source": [ "Retraining the saved model for another epoch ( It started to ran for 2nd time so i stopped it interrupting)" ], "metadata": { "id": "5zlzHcWoewB4" } }, { "cell_type": "code", "source": [ "checkpoint_filepath = 'checkpoint_bert_base_2k_2'\n", "model_checkpoint_callback = tf.keras.callbacks.ModelCheckpoint(\n", " filepath=checkpoint_filepath,\n", " save_weights_only=True,\n", " monitor='val_accuracy',\n", " mode='max',\n", " save_best_only=True)\n", "early_stop = tf.keras.callbacks.EarlyStopping(monitor='val_loss', patience=2, restore_best_weights=True)\n", "train_history = model.fit(x ={'input_ids':x_train['input_ids'],'attention_mask':x_train['attention_mask']},y= y_train, epochs=1, validation_split=0.1, batch_size=6, shuffle=True, callbacks=[early_stop])" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 380 }, "id": "2hGMo4Jhi-35", "outputId": "1b229640-d130-45ed-c02c-0e5f933acd1b" }, "execution_count": 24, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "\r 1/225 [..............................] - ETA: 5:57:13 - loss: 4.4309e-04 - accuracy: 1.0000" ] }, { "output_type": "error", "ename": "KeyboardInterrupt", "evalue": "ignored", "traceback": [ "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", "\u001b[0;31mKeyboardInterrupt\u001b[0m Traceback (most recent call last)", "\u001b[0;32m\u001b[0m in \u001b[0;36m\u001b[0;34m()\u001b[0m\n\u001b[1;32m 7\u001b[0m save_best_only=True)\n\u001b[1;32m 8\u001b[0m \u001b[0mearly_stop\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mkeras\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcallbacks\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mEarlyStopping\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmonitor\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m'val_loss'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mpatience\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m2\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mrestore_best_weights\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mTrue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 9\u001b[0;31m \u001b[0mtrain_history\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mmodel\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m \u001b[0;34m=\u001b[0m\u001b[0;34m{\u001b[0m\u001b[0;34m'input_ids'\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0mx_train\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'input_ids'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m'attention_mask'\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0mx_train\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'attention_mask'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m}\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0my\u001b[0m\u001b[0;34m=\u001b[0m \u001b[0my_train\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mepochs\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mvalidation_split\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m0.1\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbatch_size\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m6\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mshuffle\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mTrue\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcallbacks\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mearly_stop\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m", "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/keras/utils/traceback_utils.py\u001b[0m in \u001b[0;36merror_handler\u001b[0;34m(*args, **kwargs)\u001b[0m\n\u001b[1;32m 62\u001b[0m \u001b[0mfiltered_tb\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 63\u001b[0m \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 64\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mfn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 65\u001b[0m \u001b[0;32mexcept\u001b[0m \u001b[0mException\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;31m# pylint: disable=broad-except\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 66\u001b[0m \u001b[0mfiltered_tb\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0m_process_traceback_frames\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0me\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__traceback__\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/keras/engine/training.py\u001b[0m in \u001b[0;36mfit\u001b[0;34m(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_batch_size, validation_freq, max_queue_size, workers, use_multiprocessing)\u001b[0m\n\u001b[1;32m 1382\u001b[0m _r=1):\n\u001b[1;32m 1383\u001b[0m \u001b[0mcallbacks\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mon_train_batch_begin\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mstep\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1384\u001b[0;31m \u001b[0mtmp_logs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtrain_function\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0miterator\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 1385\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mdata_handler\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mshould_sync\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1386\u001b[0m \u001b[0mcontext\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0masync_wait\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/tensorflow/python/util/traceback_utils.py\u001b[0m in \u001b[0;36merror_handler\u001b[0;34m(*args, **kwargs)\u001b[0m\n\u001b[1;32m 148\u001b[0m \u001b[0mfiltered_tb\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 149\u001b[0m \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 150\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mfn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 151\u001b[0m \u001b[0;32mexcept\u001b[0m \u001b[0mException\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 152\u001b[0m \u001b[0mfiltered_tb\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0m_process_traceback_frames\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0me\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__traceback__\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/def_function.py\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, *args, **kwds)\u001b[0m\n\u001b[1;32m 913\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 914\u001b[0m \u001b[0;32mwith\u001b[0m \u001b[0mOptionalXlaContext\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_jit_compile\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 915\u001b[0;31m \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwds\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 916\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 917\u001b[0m \u001b[0mnew_tracing_count\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mexperimental_get_tracing_count\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/def_function.py\u001b[0m in \u001b[0;36m_call\u001b[0;34m(self, *args, **kwds)\u001b[0m\n\u001b[1;32m 945\u001b[0m \u001b[0;31m# In this case we have created variables on the first call, so we run the\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 946\u001b[0m \u001b[0;31m# defunned version which is guaranteed to never create variables.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 947\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_stateless_fn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwds\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;31m# pylint: disable=not-callable\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 948\u001b[0m \u001b[0;32melif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_stateful_fn\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 949\u001b[0m \u001b[0;31m# Release the lock early so that multiple threads can perform the call\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/function.py\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m 2955\u001b[0m filtered_flat_args) = self._maybe_define_function(args, kwargs)\n\u001b[1;32m 2956\u001b[0m return graph_function._call_flat(\n\u001b[0;32m-> 2957\u001b[0;31m filtered_flat_args, captured_inputs=graph_function.captured_inputs) # pylint: disable=protected-access\n\u001b[0m\u001b[1;32m 2958\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 2959\u001b[0m \u001b[0;34m@\u001b[0m\u001b[0mproperty\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/function.py\u001b[0m in \u001b[0;36m_call_flat\u001b[0;34m(self, args, captured_inputs, cancellation_manager)\u001b[0m\n\u001b[1;32m 1852\u001b[0m \u001b[0;31m# No tape is watching; skip to running the function.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1853\u001b[0m return self._build_call_outputs(self._inference_function.call(\n\u001b[0;32m-> 1854\u001b[0;31m ctx, args, cancellation_manager=cancellation_manager))\n\u001b[0m\u001b[1;32m 1855\u001b[0m forward_backward = self._select_forward_and_backward_functions(\n\u001b[1;32m 1856\u001b[0m \u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/function.py\u001b[0m in \u001b[0;36mcall\u001b[0;34m(self, ctx, args, cancellation_manager)\u001b[0m\n\u001b[1;32m 502\u001b[0m \u001b[0minputs\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 503\u001b[0m \u001b[0mattrs\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mattrs\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 504\u001b[0;31m ctx=ctx)\n\u001b[0m\u001b[1;32m 505\u001b[0m \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 506\u001b[0m outputs = execute.execute_with_cancellation(\n", "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/execute.py\u001b[0m in \u001b[0;36mquick_execute\u001b[0;34m(op_name, num_outputs, inputs, attrs, ctx, name)\u001b[0m\n\u001b[1;32m 53\u001b[0m \u001b[0mctx\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mensure_initialized\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 54\u001b[0m tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,\n\u001b[0;32m---> 55\u001b[0;31m inputs, attrs, num_outputs)\n\u001b[0m\u001b[1;32m 56\u001b[0m \u001b[0;32mexcept\u001b[0m \u001b[0mcore\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_NotOkStatusException\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 57\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mname\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;31mKeyboardInterrupt\u001b[0m: " ] } ] }, { "cell_type": "markdown", "source": [ "Saved the model after 2 epoch" ], "metadata": { "id": "ATtQg-hMfLAO" } }, { "cell_type": "code", "source": [ "model.save(\"bert_xlmroberta_2k_2.h5\")" ], "metadata": { "id": "FfmnjFUwjHXZ" }, "execution_count": 23, "outputs": [] }, { "cell_type": "markdown", "source": [ "Evaluation of the model on train and test set" ], "metadata": { "id": "bUGje_6lfOER" } }, { "cell_type": "code", "execution_count": 25, "metadata": { "id": "U6yne3eZoU-G", "colab": { "base_uri": "https://localhost:8080/" }, "outputId": "21917e68-31ab-4b78-f190-07d460f23dc9" }, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ " precision recall f1-score support\n", "\n", " 0 1.00 1.00 1.00 781\n", " 1 1.00 1.00 1.00 719\n", "\n", " accuracy 1.00 1500\n", " macro avg 1.00 1.00 1.00 1500\n", "weighted avg 1.00 1.00 1.00 1500\n", "\n", " precision recall f1-score support\n", "\n", " 0 1.00 1.00 1.00 260\n", " 1 1.00 1.00 1.00 240\n", "\n", " accuracy 1.00 500\n", " macro avg 1.00 1.00 1.00 500\n", "weighted avg 1.00 1.00 1.00 500\n", "\n" ] } ], "source": [ "from sklearn.metrics import classification_report\n", "\n", "# Predict Train set result\n", "predicted_raw_train = model.predict({'input_ids':x_train['input_ids'],'attention_mask':x_train['attention_mask']})\n", "y_predicted_train = np.argmax(predicted_raw_train, axis = 1)\n", "y_true_train = np.argmax(y_train, axis = 1)\n", "print(classification_report(y_true_train, y_predicted_train))\n", "\n", "#Predict test set result\n", "predicted_raw = model.predict({'input_ids':x_test['input_ids'],'attention_mask':x_test['attention_mask']})\n", "y_predicted = np.argmax(predicted_raw, axis = 1)\n", "y_true = np.argmax(y_test, axis = 1)\n", "print(classification_report(y_true, y_predicted))\n" ] }, { "cell_type": "markdown", "source": [ "**Write up:**\n", "1. Link to hugging face model:\n", "https://huggingface.co/Sajib-006/fake_news_detection_xlmRoberta\n", "2. As we can see, there is almost 100% accuracy and F1-score for 2000 dataset, so i haven't tried to find misclassified data.\n", "3. I couldn't run the model for the whole dataset as i used google colab free version, there was RAM and disk restrictions. XLMRoberta is a heavy model, so training it for the full dataset tends to tke huge time. Colab doesn't provide GPU for long time. \n", "4. As one run for one epoch took huge time, i had to save checkpoint after 1 epoch and retrain the model loading weights for 2nd time. After 2 epoch it showed almost 100% accuracy, so i didn't continue to train again.\n", "5. A more clear picture could have been seen if it could be run for the full dataset. I thought of some ideas about better model but couldn't implement for hardware restriction as mentioned and time constraint. My ideas are-\n", "\n", "\n", " * Adding dense layer and dropout layer to reduce overfitting(Though in my result there is 100% accuracy on hold-out test set, so no overfitting seems to be there)\n", " * Adding convolutional layer after the bert encoder work even better.\n", " * Combination of different complex convolution layers can be added to check if accuracy increases further more.\n", " * Hyperparameter tuning of the layers to ensure best result.\n", "\n" ], "metadata": { "id": "WNHCZu6wfijY" } }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "E90i018KyJH3" }, "outputs": [], "source": [ "### WRITE YOUR CODE TO TRAIN THE MODEL HERE" ] }, { "cell_type": "markdown", "metadata": { "id": "kpInVUMLyJ24" }, "source": [ "**Write up**: \n", "* Link to the model on Hugging Face Hub: \n", "* Include some examples of misclassified news articles. Please explain what you might do to improve your model's performance on these news articles in the future (you do not need to impelement these suggestions)" ] }, { "cell_type": "markdown", "metadata": { "id": "jTfHpo6BOmE8" }, "source": [ "# 3. Deep RL / Robotics" ] }, { "cell_type": "markdown", "metadata": { "id": "saB64bbTXWgZ" }, "source": [ "**RL for Classical Control:** Using any of the [classical control](https://github.com/openai/gym/blob/master/docs/environments.md#classic-control) environments from OpenAI's `gym`, implement a deep NN that learns an optimal policy which maximizes the reward of the environment.\n", "\n", "* Describe the NN you implemented and the behavior you observe from the agent as the model converges (or diverges).\n", "* Plot the reward as a function of steps (or Epochs).\n", "Compare your results to a random agent.\n", "* Discuss whether you think your model has learned the optimal policy and potential methods for improving it and/or where it might fail.\n", "* (Optional) [Upload the the model to the Hugging Face Hub](https://huggingface.co/docs/hub/adding-a-model), and add a link to your model below.\n", "\n", "\n", "You may use any frameworks you like, but you must implement your NN on your own (no pre-defined/trained models like [`stable_baselines`](https://stable-baselines.readthedocs.io/en/master/)).\n", "\n", "You may use any simulator other than `gym` _however_:\n", "* The environment has to be similar to the classical control environments (or more complex like [`robosuite`](https://github.com/ARISE-Initiative/robosuite)).\n", "* You cannot choose a game/Atari/text based environment. The purpose of this challenge is to demonstrate an understanding of basic kinematic/dynamic systems." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "CUhkTcoeynVv" }, "outputs": [], "source": [ "### WRITE YOUR CODE TO TRAIN THE MODEL HERE" ] }, { "cell_type": "markdown", "metadata": { "id": "bWllPZhJyotg" }, "source": [ "**Write up**: \n", "* (Optional) link to the model on Hugging Face Hub: \n", "* Discuss whether you think your model has learned the optimal policy and potential methods for improving it and/or where it might fail." ] }, { "cell_type": "markdown", "metadata": { "id": "rbrRbrISa5J_" }, "source": [ "# 4. Theory / Linear Algebra " ] }, { "cell_type": "markdown", "metadata": { "id": "KFkLRCzTXTzL" }, "source": [ "**Implement Contrastive PCA** Read [this paper](https://www.nature.com/articles/s41467-018-04608-8) and implement contrastive PCA in Python.\n", "\n", "* First, please discuss what kind of dataset this would make sense to use this method on\n", "* Implement the method in Python (do not use previous implementations of the method if they already exist)\n", "* Then create a synthetic dataset and apply the method to the synthetic data. Compare with standard PCA.\n" ] }, { "cell_type": "markdown", "metadata": { "id": "TpyqWl-ly0wy" }, "source": [ "**Write up**: Discuss what kind of dataset it would make sense to use Contrastive PCA" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "1CQzUSfQywRk" }, "outputs": [], "source": [ "### WRITE YOUR CODE HERE" ] }, { "cell_type": "markdown", "metadata": { "id": "dlqmZS5Hy6q-" }, "source": [ "# 5. Systems" ] }, { "cell_type": "markdown", "metadata": { "id": "QW_eiDFw1QKm" }, "source": [ "**Inference on the edge**: Measure the inference times in various computationally-constrained settings\n", "\n", "* Pick a few different speech detection models (we suggest looking at models on the [Hugging Face Hub](https://huggingface.co/models?pipeline_tag=automatic-speech-recognition&sort=downloads))\n", "* Simulate different memory constraints and CPU allocations that are realistic for edge devices that might run such models, such as smart speakers or microcontrollers, and measure what is the average inference time of the models under these conditions \n", "* How does the inference time vary with (1) choice of model (2) available system memory (3) available CPU (4) size of input?\n", "\n", "Are there any surprising discoveries? (Note that this coding challenge is fairly open-ended, so we will be considering the amount of effort invested in discovering something interesting here)." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "OYp94wLP1kWJ" }, "outputs": [], "source": [ "### WRITE YOUR CODE HERE" ] }, { "cell_type": "markdown", "metadata": { "id": "yoHmutWx2jer" }, "source": [ "**Write up**: What surprising discoveries do you see?" ] } ], "metadata": { "colab": { "collapsed_sections": [], "name": "Coding Challenge for Fatima Fellowship-XLMRoberta", "provenance": [] }, "kernelspec": { "display_name": "Python 3", "name": "python3" }, "widgets": { "application/vnd.jupyter.widget-state+json": { "c9d66cc4ead940a1babe93fe3011abe2": { "model_module": "@jupyter-widgets/controls", "model_name": "HBoxModel", "model_module_version": "1.5.0", "state": { "_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HBoxModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HBoxView", "box_style": "", "children": [ "IPY_MODEL_bf4ea0b231e44d75abee70860652f6da", "IPY_MODEL_c3b86577ff2d49eaad2785112d107e95", "IPY_MODEL_86e941cca4c148a5a8fb3faed3e1c058" ], "layout": "IPY_MODEL_bd7fdc49334142818f3fd69c467b3665" } }, "bf4ea0b231e44d75abee70860652f6da": { "model_module": "@jupyter-widgets/controls", "model_name": "HTMLModel", "model_module_version": "1.5.0", "state": { "_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HTMLView", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_78baedc6e7a0480aa824576ec549a7ff", "placeholder": "​", "style": "IPY_MODEL_e0e3f3fe05b1414088d93c0a0f32db02", "value": "Downloading: 100%" } }, "c3b86577ff2d49eaad2785112d107e95": { "model_module": "@jupyter-widgets/controls", "model_name": "FloatProgressModel", "model_module_version": "1.5.0", "state": { "_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "FloatProgressModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "ProgressView", "bar_style": "success", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_4f7ade9b563249f99dad7d9e36bdbba3", "max": 512, "min": 0, "orientation": "horizontal", "style": "IPY_MODEL_de92d95bb4954949bd6a051b595fb688", "value": 512 } }, "86e941cca4c148a5a8fb3faed3e1c058": { "model_module": "@jupyter-widgets/controls", "model_name": "HTMLModel", "model_module_version": "1.5.0", "state": { "_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HTMLView", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_64b2f8229c4d406a95ec2054bcae1ebf", "placeholder": "​", "style": "IPY_MODEL_966c0379ec81464d97cf165d3ddc05bc", "value": " 512/512 [00:00<00:00, 11.3kB/s]" } }, "bd7fdc49334142818f3fd69c467b3665": { "model_module": "@jupyter-widgets/base", "model_name": "LayoutModel", "model_module_version": "1.2.0", "state": { "_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null } }, "78baedc6e7a0480aa824576ec549a7ff": { "model_module": "@jupyter-widgets/base", "model_name": "LayoutModel", "model_module_version": "1.2.0", "state": { "_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null } }, "e0e3f3fe05b1414088d93c0a0f32db02": { "model_module": "@jupyter-widgets/controls", "model_name": "DescriptionStyleModel", "model_module_version": "1.5.0", "state": { "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "DescriptionStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "description_width": "" } }, "4f7ade9b563249f99dad7d9e36bdbba3": { "model_module": "@jupyter-widgets/base", "model_name": "LayoutModel", "model_module_version": "1.2.0", "state": { "_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null } }, "de92d95bb4954949bd6a051b595fb688": { "model_module": "@jupyter-widgets/controls", "model_name": "ProgressStyleModel", "model_module_version": "1.5.0", "state": { "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "ProgressStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "bar_color": null, "description_width": "" } }, "64b2f8229c4d406a95ec2054bcae1ebf": { "model_module": "@jupyter-widgets/base", "model_name": "LayoutModel", "model_module_version": "1.2.0", "state": { "_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null } }, "966c0379ec81464d97cf165d3ddc05bc": { "model_module": "@jupyter-widgets/controls", "model_name": "DescriptionStyleModel", "model_module_version": "1.5.0", "state": { "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "DescriptionStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "description_width": "" } }, "5620a2691dc24031a5db44bf549f2e6a": { "model_module": "@jupyter-widgets/controls", "model_name": "HBoxModel", "model_module_version": "1.5.0", "state": { "_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HBoxModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HBoxView", "box_style": "", "children": [ "IPY_MODEL_174a6b221a9346809b334a0690732e5a", "IPY_MODEL_2e9b0ba33faa4ab6957ffbda64e8e35f", "IPY_MODEL_ab7609a1a4c0436bb5c8bd35884504f0" ], "layout": "IPY_MODEL_b96db63588f3454e910ab6457e2a67d9" } }, "174a6b221a9346809b334a0690732e5a": { "model_module": "@jupyter-widgets/controls", "model_name": "HTMLModel", "model_module_version": "1.5.0", "state": { "_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HTMLView", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_453493c9d9854e2283e68a9fb0dbe8b6", "placeholder": "​", "style": "IPY_MODEL_2eed6379fe3e4b3b8c483a78da05a4bc", "value": "Downloading: 100%" } }, "2e9b0ba33faa4ab6957ffbda64e8e35f": { "model_module": "@jupyter-widgets/controls", "model_name": "FloatProgressModel", "model_module_version": "1.5.0", "state": { "_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "FloatProgressModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "ProgressView", "bar_style": "success", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_4b9f7cd7403d4499b7360d2d1fce57c5", "max": 5069051, "min": 0, "orientation": "horizontal", "style": "IPY_MODEL_f6182cd27c8040089a65be7e84ef3403", "value": 5069051 } }, "ab7609a1a4c0436bb5c8bd35884504f0": { "model_module": "@jupyter-widgets/controls", "model_name": "HTMLModel", "model_module_version": "1.5.0", "state": { "_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HTMLView", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_8f78b817454d4fd2a30b1a7b8d89b57f", "placeholder": "​", "style": "IPY_MODEL_fdfef297b3d44d1486025664c257afd6", "value": " 4.83M/4.83M [00:00<00:00, 23.8MB/s]" } }, "b96db63588f3454e910ab6457e2a67d9": { "model_module": "@jupyter-widgets/base", "model_name": "LayoutModel", "model_module_version": "1.2.0", "state": { "_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null } }, "453493c9d9854e2283e68a9fb0dbe8b6": { "model_module": "@jupyter-widgets/base", "model_name": "LayoutModel", "model_module_version": "1.2.0", "state": { "_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null } }, "2eed6379fe3e4b3b8c483a78da05a4bc": { "model_module": "@jupyter-widgets/controls", "model_name": "DescriptionStyleModel", "model_module_version": "1.5.0", "state": { "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "DescriptionStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "description_width": "" } }, "4b9f7cd7403d4499b7360d2d1fce57c5": { "model_module": "@jupyter-widgets/base", "model_name": "LayoutModel", "model_module_version": "1.2.0", "state": { "_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null } }, "f6182cd27c8040089a65be7e84ef3403": { "model_module": "@jupyter-widgets/controls", "model_name": "ProgressStyleModel", "model_module_version": "1.5.0", "state": { "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "ProgressStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "bar_color": null, "description_width": "" } }, "8f78b817454d4fd2a30b1a7b8d89b57f": { "model_module": "@jupyter-widgets/base", "model_name": "LayoutModel", "model_module_version": "1.2.0", "state": { "_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null } }, "fdfef297b3d44d1486025664c257afd6": { "model_module": "@jupyter-widgets/controls", "model_name": "DescriptionStyleModel", "model_module_version": "1.5.0", "state": { "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "DescriptionStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "description_width": "" } }, "40aa7f9c6a894d26bdee2352ec269ddb": { "model_module": "@jupyter-widgets/controls", "model_name": "HBoxModel", "model_module_version": "1.5.0", "state": { "_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HBoxModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HBoxView", "box_style": "", "children": [ "IPY_MODEL_4c8162e8f9264b33bb2bb625c764fe85", "IPY_MODEL_1c78711b6c9a476fa140d54637c5cc51", "IPY_MODEL_8e2eb87c9572426abb0f47883b5849a9" ], "layout": "IPY_MODEL_6f87ef566e644da784f889d617b455eb" } }, "4c8162e8f9264b33bb2bb625c764fe85": { "model_module": "@jupyter-widgets/controls", "model_name": "HTMLModel", "model_module_version": "1.5.0", "state": { "_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HTMLView", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_303792e631314bb0b9500cd59f338bbf", "placeholder": "​", "style": "IPY_MODEL_c6261873f8d1455fa8ac6a5968383ece", "value": "Downloading: 100%" } }, "1c78711b6c9a476fa140d54637c5cc51": { "model_module": "@jupyter-widgets/controls", "model_name": "FloatProgressModel", "model_module_version": "1.5.0", "state": { "_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "FloatProgressModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "ProgressView", "bar_style": "success", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_ec257f57f17e421490440b2202941db4", "max": 1885418496, "min": 0, "orientation": "horizontal", "style": "IPY_MODEL_a78f48f50f994746b0343efda278bf43", "value": 1885418496 } }, "8e2eb87c9572426abb0f47883b5849a9": { "model_module": "@jupyter-widgets/controls", "model_name": "HTMLModel", "model_module_version": "1.5.0", "state": { "_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HTMLView", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_b8b252e5acf74dba83fdeb21323d6296", "placeholder": "​", "style": "IPY_MODEL_5457220bdb004116b87e596ab725a1b6", "value": " 1.76G/1.76G [01:30<00:00, 36.7MB/s]" } }, "6f87ef566e644da784f889d617b455eb": { "model_module": "@jupyter-widgets/base", "model_name": "LayoutModel", "model_module_version": "1.2.0", "state": { "_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null } }, "303792e631314bb0b9500cd59f338bbf": { "model_module": "@jupyter-widgets/base", "model_name": "LayoutModel", "model_module_version": "1.2.0", "state": { "_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null } }, "c6261873f8d1455fa8ac6a5968383ece": { "model_module": "@jupyter-widgets/controls", "model_name": "DescriptionStyleModel", "model_module_version": "1.5.0", "state": { "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "DescriptionStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "description_width": "" } }, "ec257f57f17e421490440b2202941db4": { "model_module": "@jupyter-widgets/base", "model_name": "LayoutModel", "model_module_version": "1.2.0", "state": { "_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null } }, "a78f48f50f994746b0343efda278bf43": { "model_module": "@jupyter-widgets/controls", "model_name": "ProgressStyleModel", "model_module_version": "1.5.0", "state": { "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "ProgressStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "bar_color": null, "description_width": "" } }, "b8b252e5acf74dba83fdeb21323d6296": { "model_module": "@jupyter-widgets/base", "model_name": "LayoutModel", "model_module_version": "1.2.0", "state": { "_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null } }, "5457220bdb004116b87e596ab725a1b6": { "model_module": "@jupyter-widgets/controls", "model_name": "DescriptionStyleModel", "model_module_version": "1.5.0", "state": { "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "DescriptionStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "description_width": "" } } } } }, "nbformat": 4, "nbformat_minor": 0 }