tutorials/asr/ASR_with_Transducers.ipynb

{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "SUOXg71A3w78"
      },
      "outputs": [],
      "source": [
        "\"\"\"\n",
        "You can run either this notebook locally (if you have all the dependencies and a GPU) or on Google Colab.\n",
        "\n",
        "Instructions for setting up Colab are as follows:\n",
        "1. Open a new Python 3 notebook.\n",
        "2. Import this notebook from GitHub (File -> Upload Notebook -> \"GITHUB\" tab -> copy/paste GitHub URL)\n",
        "3. Connect to an instance with a GPU (Runtime -> Change runtime type -> select \"GPU\" for hardware accelerator)\n",
        "4. Run this cell to set up dependencies.\n",
        "5. Restart the runtime (Runtime -> Restart Runtime) for any upgraded packages to take effect\n",
        "\"\"\"\n",
        "# If you're using Google Colab and not running locally, run this cell.\n",
        "import os\n",
        "\n",
        "# Install dependencies\n",
        "!pip install wget\n",
        "!apt-get install sox libsndfile1 ffmpeg\n",
        "!pip install text-unidecode\n",
        "!pip install matplotlib>=3.3.2\n",
        "\n",
        "## Install NeMo\n",
        "BRANCH = 'r1.17.0'\n",
        "!python -m pip install git+https://github.com/NVIDIA/NeMo.git@$BRANCH#egg=nemo_toolkit[all]\n",
        "\n",
        "## Grab the config we'll use in this example\n",
        "!mkdir configs\n",
        "!wget -P configs/ https://raw.githubusercontent.com/NVIDIA/NeMo/$BRANCH/examples/asr/conf/contextnet_rnnt/contextnet_rnnt.yaml\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "uj-UnhKk47oW"
      },
      "outputs": [],
      "source": [
        "# In a conda environment, you would use the following command\n",
        "# Update Numba to > 0.53\n",
        "# conda install -c conda-forge numba\n",
        "# or\n",
        "# conda update -c conda-forge numba\n",
        "\n",
        "# For pip based environments,\n",
        "# Update Numba to > 0.53\n",
        "!pip install --upgrade numba"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RGNuJWr66C38"
      },
      "source": [
        "# Automatic Speech Recognition with Transducer Models\n",
        "\n",
        "This notebook is a basic tutorial for creating a Transducer ASR model and then training it on a small dataset (AN4). It includes discussion relevant to reducing memory issues when training such models and demonstrates how to change the decoding strategy after training. Finally, it also provides a brief glimpse of extracting alignment information from a trained Transducer model.\n",
        "\n",
        "As we will see in this tutorial, apart from the differences in the config and the class used to instantiate the model, nearly all steps are precisely similar to any CTC-based model training. Many concepts such as data loader setup, optimization setup, pre-trained checkpoint weight loading will be nearly identical between CTC and Transducer models.\n",
        "\n",
        "In essence, NeMo makes it seamless to take a config for a CTC ASR model, add in a few components related to Transducers (often without any modifications) and use a different class to instantiate a Transducer model!\n",
        "\n",
        "--------\n",
        "\n",
        "**Note**: It is assumed that the previous tutorial - \"Intro-to-Transducers\" has been reviewed, and there is some familiarity with the config components of transducer models.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eAqCcJ-T6C6k"
      },
      "source": [
        "# Preparing the dataset\n",
        "\n",
        "In this tutorial, we will be utilizing the `AN4`dataset - also known as the Alphanumeric dataset, which was collected and published by Carnegie Mellon University. It consists of recordings of people spelling out addresses, names, telephone numbers, etc., one letter or number at a time and their corresponding transcripts. We choose to use AN4 for this tutorial because it is relatively small, with 948 training and 130 test utterances, and so it trains quickly. \n",
        "\n",
        "Let's first download the preparation script from NeMo's scripts directory -"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "pu4n4GkjAo9i"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "\n",
        "if not os.path.exists(\"scripts/\"):\n",
        "  os.makedirs(\"scripts\")\n",
        "\n",
        "if not os.path.exists(\"scripts/process_an4_data.py\"):\n",
        "  !wget -P scripts/ https://raw.githubusercontent.com/NVIDIA/NeMo/$BRANCH/scripts/dataset_processing/process_an4_data.py"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_QGyusFJBUqH"
      },
      "source": [
        "------\n",
        "\n",
        "Download and prepare the two subsets of `AN 4`"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "yV1rJcH6ApDo"
      },
      "outputs": [],
      "source": [
        "import wget\n",
        "import tarfile \n",
        "import subprocess \n",
        "import glob\n",
        "\n",
        "data_dir = \"datasets\"\n",
        "\n",
        "if not os.path.exists(data_dir):\n",
        "  os.makedirs(data_dir)\n",
        "\n",
        "# Download the dataset. This will take a few moments...\n",
        "print(\"******\")\n",
        "if not os.path.exists(data_dir + '/an4_sphere.tar.gz'):\n",
        "    an4_url = 'https://dldata-public.s3.us-east-2.amazonaws.com/an4_sphere.tar.gz'  # for the original source, please visit http://www.speech.cs.cmu.edu/databases/an4/an4_sphere.tar.gz \n",
        "    an4_path = wget.download(an4_url, data_dir)\n",
        "    print(f\"Dataset downloaded at: {an4_path}\")\n",
        "else:\n",
        "    print(\"Tarfile already exists.\")\n",
        "    an4_path = data_dir + '/an4_sphere.tar.gz'\n",
        "\n",
        "\n",
        "if not os.path.exists(data_dir + '/an4/'):\n",
        "    # Untar and convert .sph to .wav (using sox)\n",
        "    tar = tarfile.open(an4_path)\n",
        "    tar.extractall(path=data_dir)\n",
        "\n",
        "    print(\"Converting .sph to .wav...\")\n",
        "    sph_list = glob.glob(data_dir + '/an4/**/*.sph', recursive=True)\n",
        "    for sph_path in sph_list:\n",
        "        wav_path = sph_path[:-4] + '.wav'\n",
        "        cmd = [\"sox\", sph_path, wav_path]\n",
        "        subprocess.run(cmd)\n",
        "\n",
        "print(\"Finished conversion.\\n******\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "# --- Building Manifest Files --- #\n",
        "import json\n",
        "import librosa\n",
        "\n",
        "# Function to build a manifest\n",
        "def build_manifest(transcripts_path, manifest_path, wav_path):\n",
        "    with open(transcripts_path, 'r') as fin:\n",
        "        with open(manifest_path, 'w') as fout:\n",
        "            for line in fin:\n",
        "                # Lines look like this:\n",
        "                # <s> transcript </s> (fileID)\n",
        "                transcript = line[: line.find('(')-1].lower()\n",
        "                transcript = transcript.replace('<s>', '').replace('</s>', '')\n",
        "                transcript = transcript.strip()\n",
        "\n",
        "                file_id = line[line.find('(')+1 : -2]  # e.g. \"cen4-fash-b\"\n",
        "                audio_path = os.path.join(\n",
        "                    data_dir, wav_path,\n",
        "                    file_id[file_id.find('-')+1 : file_id.rfind('-')],\n",
        "                    file_id + '.wav')\n",
        "\n",
        "                duration = librosa.core.get_duration(filename=audio_path)\n",
        "\n",
        "                # Write the metadata to the manifest\n",
        "                metadata = {\n",
        "                    \"audio_filepath\": audio_path,\n",
        "                    \"duration\": duration,\n",
        "                    \"text\": transcript\n",
        "                }\n",
        "                json.dump(metadata, fout)\n",
        "                fout.write('\\n')\n",
        "\n",
        "# Building Manifests\n",
        "print(\"******\")\n",
        "train_transcripts = os.path.join(data_dir, 'an4/etc/an4_train.transcription')\n",
        "train_manifest = os.path.join(data_dir, 'an4/train_manifest.json')\n",
        "if not os.path.isfile(train_manifest):\n",
        "    build_manifest(train_transcripts, train_manifest, 'an4/wav/an4_clstk')\n",
        "    print(\"Training manifest created.\")\n",
        "\n",
        "test_transcripts = os.path.join(data_dir, 'an4/etc/an4_test.transcription')\n",
        "test_manifest = os.path.join(data_dir, 'an4/test_manifest.json')\n",
        "if not os.path.isfile(test_manifest):\n",
        "    build_manifest(test_transcripts, test_manifest, 'an4/wav/an4test_clstk')\n",
        "    print(\"Test manifest created.\")\n",
        "print(\"***Done***\") \n",
        "# Manifest filepaths\n",
        "TRAIN_MANIFEST = train_manifest\n",
        "TEST_MANIFEST = test_manifest"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2sSycX_VJsCI"
      },
      "source": [
        "## Preparing the tokenizer\n",
        "\n",
        "Now that we have a dataset ready, we need to decide whether to use a character-based model or a sub-word-based model. For completeness' sake, we will use a tokenizer based model so that we can leverage a modern encoder architecture like ContextNet or Conformer-T."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "i2hD4LkoJvrx"
      },
      "outputs": [],
      "source": [
        "if not os.path.exists(\"scripts/process_asr_text_tokenizer.py\"):\n",
        "  !wget -P scripts/ https://raw.githubusercontent.com/NVIDIA/NeMo/$BRANCH/scripts/tokenizers/process_asr_text_tokenizer.py"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dQdeiafdKE4A"
      },
      "source": [
        "-----\n",
        "\n",
        "Since the dataset is tiny, we can use a small SentencePiece based tokenizer. We always delete the tokenizer directory so any changes to the manifest files are always replicated in the tokenizer."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "i6Jzpt6UJvuI"
      },
      "outputs": [],
      "source": [
        "VOCAB_SIZE = 32  # can be any value above 29\n",
        "TOKENIZER_TYPE = \"spe\"  # can be wpe or spe\n",
        "SPE_TYPE = \"unigram\"  # can be bpe or unigram\n",
        "\n",
        "# ------------------------------------------------------------------- #\n",
        "!rm -r tokenizers/\n",
        "\n",
        "if not os.path.exists(\"tokenizers\"):\n",
        "  os.makedirs(\"tokenizers\")\n",
        "\n",
        "!python scripts/process_asr_text_tokenizer.py \\\n",
        "   --manifest=$TRAIN_MANIFEST \\\n",
        "   --data_root=\"tokenizers\" \\\n",
        "   --tokenizer=$TOKENIZER_TYPE \\\n",
        "   --spe_type=$SPE_TYPE \\\n",
        "   --no_lower_case \\\n",
        "   --log \\\n",
        "   --vocab_size=$VOCAB_SIZE"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "JHDZswN6LIBJ"
      },
      "outputs": [],
      "source": [
        "# Tokenizer path\n",
        "if TOKENIZER_TYPE == 'spe':\n",
        "  TOKENIZER = os.path.join(\"tokenizers\", f\"tokenizer_spe_{SPE_TYPE}_v{VOCAB_SIZE}\")\n",
        "  TOKENIZER_TYPE_CFG = \"bpe\"\n",
        "else:\n",
        "  TOKENIZER = os.path.join(\"tokenizers\", f\"tokenizer_wpe_v{VOCAB_SIZE}\")\n",
        "  TOKENIZER_TYPE_CFG = \"wpe\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "j8tx2m2w6C87"
      },
      "source": [
        "# Preparing a Transducer Model\n",
        "\n",
        "Now that we have the dataset and tokenizer prepared, let us begin by setting up the config of the Transducer model! In this tutorial, we will build a slightly modified ContextNet architecture (which is obtained from the paper [ContextNet: Improving Convolutional Neural Networks for Automatic Speech Recognition with Global Context](https://arxiv.org/abs/2005.03191)).\n",
        "\n",
        "We can note that many of the steps here are identical to the setup of a CTC model!\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pQETBCSML0Us"
      },
      "source": [
        "## Prepare the config\n",
        "\n",
        "For a dataset such as AN4, we do not need such a deep model. In fact, the depth of this model will cause much slower convergence on a small dataset, which would require far too long to train on Colab.\n",
        "\n",
        "In order to speed up training for this demo, we will take only the first five blocks of ContextNet, and discard the rest - and we can do this directly from the config.\n",
        "\n",
        "**Note**: On any realistic dataset (say Librispeech) this step would hurt the model's accuracy significantly. It is being done only to reduce the time spent waiting for training to finish on Colab."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "6a_vedo0Lyo8"
      },
      "outputs": [],
      "source": [
        "from omegaconf import OmegaConf, open_dict\n",
        "\n",
        "config = OmegaConf.load(\"../../examples/asr/conf/contextnet_rnnt/contextnet_rnnt.yaml\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "N-t7fRl6GS3A"
      },
      "source": [
        "-----\n",
        "\n",
        "Here, we will slice off the first five blocks from the Jasper block (used to build ContextNet). Setting the config with this subset will create a stride 2x model with just five blocks.\n",
        "\n",
        "We will also explicitly state that the last block dimension must be obtained from `model.model_defaults.enc_hidden` inside the config. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "B9nY5JQaIhKz"
      },
      "outputs": [],
      "source": [
        "config.model.encoder.jasper = config.model.encoder.jasper[:5]\n",
        "config.model.encoder.jasper[-1].filters = '${model.model_defaults.enc_hidden}'"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "d0tyOaqNLq-4"
      },
      "source": [
        "-------\n",
        "\n",
        "Next, set up the data loaders of the config for the ContextNet model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "mJbqMEkjMTfM"
      },
      "outputs": [],
      "source": [
        "# print out the train and validation configs to know what needs to be changed\n",
        "print(OmegaConf.to_yaml(config.model.train_ds))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fyYY6hU5Meyk"
      },
      "source": [
        "-------\n",
        "\n",
        "We can note that the config here is nearly identical to the CTC ASR model configs! So let us take the same steps here to update the configs."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "5QNYNItcMeOC"
      },
      "outputs": [],
      "source": [
        "config.model.train_ds.manifest_filepath = TRAIN_MANIFEST\n",
        "config.model.validation_ds.manifest_filepath = TEST_MANIFEST\n",
        "config.model.test_ds.manifest_filepath = TEST_MANIFEST"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MfW6KQ0gM6QQ"
      },
      "source": [
        "------\n",
        "\n",
        "Next, we need to setup the tokenizer section of the config"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "CNlRNlVCNAjr"
      },
      "outputs": [],
      "source": [
        "print(OmegaConf.to_yaml(config.model.tokenizer))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "MzmCNJZ7NAng"
      },
      "outputs": [],
      "source": [
        "config.model.tokenizer.dir = TOKENIZER\n",
        "config.model.tokenizer.type = TOKENIZER_TYPE_CFG"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "j1gVDybDq5vN"
      },
      "source": [
        "------\n",
        "Now, we can update the optimization and augmentation for this dataset in order to converge to some reasonable score within a short training run."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "cUkqv_bQraQe"
      },
      "outputs": [],
      "source": [
        "print(OmegaConf.to_yaml(config.model.optim))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "mEyK1fqEiyxo"
      },
      "outputs": [],
      "source": [
        "# Finally, let's remove logging of samples and the warmup since the dataset is small (similar to CTC models)\n",
        "config.model.log_prediction = False\n",
        "config.model.optim.sched.warmup_steps = None"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lwrvBFCcZO1q"
      },
      "source": [
        "------\n",
        "\n",
        "Next, we remove the spec augment that is provided by default for ContextNet. While additional augmentation would surely help training, it would require longer training to see significant benefits."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "i4udFKMwDbRm"
      },
      "outputs": [],
      "source": [
        "print(OmegaConf.to_yaml(config.model.spec_augment))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "OLUbRvoDDedd"
      },
      "outputs": [],
      "source": [
        "config.model.spec_augment.freq_masks = 0\n",
        "config.model.spec_augment.time_masks = 0"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0plKYG1zNKQK"
      },
      "source": [
        "------\n",
        "\n",
        "... We are now almost done! Most of the updates to a Transducer config are nearly the same as any CTC model."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gn8WzknQ6C-t"
      },
      "source": [
        "## Fused Batch during training and evaluation\n",
        "\n",
        "We discussed in the previous tutorial (Intro-to-Transducers) the significant memory cost of the Transducer Joint calculation during training. We also discussed that NeMo provides a simple yet effective method to nearly sidestep this limitation. We can now dive deeper into understanding what precisely NeMo's Transducer framework will do to alleviate this memory consumption issue.\n",
        "\n",
        "The following sub-cells are **voluntary** and valuable for understanding the cause, effect, and resolution of memory issues in Transducer models. The content can be skipped if one is familiar with the topic, and it is only required to use the `fused batch step`."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "V5sMoFHmVvhg"
      },
      "source": [
        "## Transducer Memory reduction with Fused Batch step\n",
        "\n",
        "The following few cells explain why memory is an issue when training Transducer models and how NeMo tackles the issue with its Fused Batch step.\n",
        "\n",
        "The material can be read for a thorough understanding, otherwise, it can be skipped."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kdPilgiIOncw"
      },
      "source": [
        "### Diving deeper into the memory costs of Transducer Joint\n",
        "-------\n",
        "\n",
        "One of the significant limitations of Transducers is the exorbitant memory cost of computing the Joint module. The Joint module is comprised of two steps. \n",
        "\n",
        "1) Projecting the Acoustic and Transcription feature dimensions to some standard hidden dimension (specified by `model.model_defaults.joint_hidden`)\n",
        "\n",
        "2) Projecting this intermediate hidden dimension to the final vocabulary space to obtain the transcription.\n",
        "\n",
        "Take the following example.\n",
        "\n",
        "**BS**=32 ; **T** (after **2x** stride) = 800, **U** (with character encoding) = 400-450 tokens, Vocabulary size  **V** = 28 (26 alphabet chars, space and apostrophe). Let the hidden dimension of the Joint model be 640 (Most Google Transducer papers use hidden dimension of 640).\n",
        "\n",
        "$ Memory \\, (Hidden, \\, gb) = 32 \\times 800 \\times 450 \\times 640 \\times 4 = 29.49 $ gigabytes (4 bytes per float). \n",
        "\n",
        "$ Memory \\, (Joint, \\, gb) = 32 \\times 800 \\times 450 \\times 28 \\times 4 = 1.290 $ gigabytes (4 bytes per float)\n",
        "\n",
        "-----\n",
        "\n",
        "**NOTE**: This is just for the forward pass! We need to double this memory to store gradients! This much memory is also just for the Joint model **alone**. Far more memory is required for the Prediction model as well as the large Acoustic model itself and its gradients!\n",
        "\n",
        "Even with mixed precision, that's $\\sim 30$ GB of GPU RAM for just 1 part of the network + its gradients.\n",
        "\n",
        "---------"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZxVlEP9eQim8"
      },
      "source": [
        "### Simple methods to reduce memory consumption\n",
        "\n",
        "------\n",
        "\n",
        "The easiest way to reduce memory consumption is to perform more downsampling in the acoustic model and use sub-word tokenization of the text to reduce the length of the target sequence.\n",
        "\n",
        "**BS**=32 ; **T** (after **8x** stride) = 200, **U** (with sub-word encoding) = 100-180 tokens, Vocabulary size  **V** = 1024.\n",
        "\n",
        "$ Memory \\, (Hidden, \\, gb) = 32 \\times 200 \\times 150 \\times 640 \\times 4 = 2.45 $ gigabytes (4 bytes per float).\n",
        "\n",
        "$ Memory \\, (Joint, \\, gb) = 32 \\times 200 \\times 150 \\times 1024 \\times 4 = 3.93 $ gigabytes (4 bytes per float)\n",
        "\n",
        "-----\n",
        "\n",
        "Using Automatic Mixed Precision, we expend just around 6-7 GB of GPU RAM on the Joint + its gradient.\n",
        "\n",
        "The above memory cost is much more tractable - but we generally want larger and larger acoustic models. It is consistently the easiest way to improve transcription accuracy. So that means on a limited 32 GB GPU, we have to partition 7 GB just for the Joint and remaining memory allocated between Transcription + Acoustic Models.\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7uPZyTENRwv9"
      },
      "source": [
        "### Fused Transcription-Joint-Loss-WER (also called Batch Splitting)\n",
        "----------\n",
        "\n",
        "The fundamental problem is that the joint tensor grows in size when `[T x U]` grows in size. This growth in memory cost is due to many reasons - either by model construction (downsampling) or the choice of dataset preprocessing (character tokenization vs. sub-word tokenization).\n",
        "\n",
        "Another dimension that NeMo can control is **batch**. Due to how we batch our samples, small and large samples all get clumped together into a single batch. So even though the individual samples are not all as long as the maximum length of T and U in that batch, when a batch of such samples is constructed, it will consume a significant amount of memory for the sake of compute efficiency.\n",
        "\n",
        "So as is always the case - **trade-off compute speed for memory savings**.\n",
        "\n",
        "------\n",
        "\n",
        "The fused operation goes as follows : \n",
        "\n",
        "1) Forward the entire acoustic model in a single pass. (Use global batch size here for acoustic model - found in `model.*_ds.batch_size`)\n",
        "\n",
        "2) Split the Acoustic Model's logits by `fused_batch_size` and loop over these sub-batches.\n",
        "\n",
        "3) Construct a sub-batch of same `fused_batch_size` for the Prediction model. Now the target sequence length is $U_{sub-batch} < U$. \n",
        "\n",
        "4) Feed this $U_{sub-batch}$ into the Joint model, along with a sub-batch from the Acoustic model (with $T_{sub-batch} < T$). Remember, we only have to slice off a part of the acoustic model here since we have the full batch of samples $(B, T, D)$ from the acoustic model.\n",
        "\n",
        "5) Performing steps (3) and (4) yields $T_{sub-batch}$ and $U_{sub-batch}$. Perform sub-batch joint step - costing an intermediate $(B, T_{sub-batch}, U_{sub-batch}, V)$ in memory.\n",
        "\n",
        "6) Compute loss on sub-batch and preserve in a list to be later concatenated. \n",
        "\n",
        "7) Compute sub-batch metrics (such as Character / Word Error Rate) using the above Joint tensor and sub-batch of ground truth labels. Preserve the scores to be averaged across the entire batch later.\n",
        "\n",
        "8) Delete the sub-batch joint matrix  $(B, T_{sub-batch}, U_{sub-batch}, V)$. Only gradients from .backward() are preserved now in the computation graph.\n",
        "\n",
        "9) Repeat steps (3) - (8) until all sub-batches are consumed.\n",
        "\n",
        "10) Cleanup step. Compute full batch WER and log. Concatenate loss list and pass to PTL to compute the equivalent of the original (full batch) Joint step. Delete ancillary objects necessary for sub-batching. \n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0T3vHdkeWo5S"
      },
      "source": [
        "## Setting up Fused Batch step in a Transducer Config\n",
        "\n",
        "After all that discussion above, let us look at how to enable that entire pipeline in NeMo.\n",
        "\n",
        "As we can note below, it takes precisely two changes in the config to enable the fused batch step:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Unv8-GvOWhad"
      },
      "outputs": [],
      "source": [
        "print(OmegaConf.to_yaml(config.model.joint))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "VsunP99saF5c"
      },
      "outputs": [],
      "source": [
        "# Two lines to enable the fused batch step\n",
        "config.model.joint.fuse_loss_wer = True\n",
        "config.model.joint.fused_batch_size = 16  # this can be any value (preferably less than model.*_ds.batch_size)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "yT1vJH9OkS0u"
      },
      "outputs": [],
      "source": [
        "# We will also reduce the hidden dimension of the joint and the prediction networks to preserve some memory\n",
        "config.model.model_defaults.pred_hidden = 64\n",
        "config.model.model_defaults.joint_hidden = 64"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IPb1gS3OZprs"
      },
      "source": [
        "--------\n",
        "\n",
        "Finally, since the dataset is tiny, we do not need an enormous model (the default is roughly 40 M parameters!)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "MywZQ9ADZpDW"
      },
      "outputs": [],
      "source": [
        "# Use just 128 filters across the model to speed up training and reduce parameter count\n",
        "config.model.model_defaults.filters = 128"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zUgThKbxatyg"
      },
      "source": [
        "## Initialize a Transducer ASR Model\n",
        "\n",
        "Finally, let us create a Transducer model, which is as easy as changing a line of import if you already have a script to create CTC models. We will use a small model since the dataset is just 5 hours of speech."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SH4ZfXHOdGhX"
      },
      "source": [
        "------\n",
        "\n",
        "Setup a Pytorch Lightning Trainer:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Fmf0iSY-a6LC"
      },
      "outputs": [],
      "source": [
        "import torch\n",
        "from pytorch_lightning import Trainer\n",
        "\n",
        "if torch.cuda.is_available():\n",
        "  accelerator = 'gpu'\n",
        "else:\n",
        "  accelerator = 'gpu'\n",
        "\n",
        "EPOCHS = 50\n",
        "\n",
        "# Initialize a Trainer for the Transducer model\n",
        "trainer = Trainer(devices=1, accelerator=accelerator, max_epochs=EPOCHS,\n",
        "                  enable_checkpointing=False, logger=False,\n",
        "                  log_every_n_steps=5, check_val_every_n_epoch=10)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "jqVt4TEncEqv"
      },
      "outputs": [],
      "source": [
        "# Import the Transducer Model\n",
        "import nemo.collections.asr as nemo_asr"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "RheLsmA1cRz0"
      },
      "outputs": [],
      "source": [
        "# Build the model\n",
        "model = nemo_asr.models.EncDecRNNTBPEModel(cfg=config.model, trainer=trainer)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "9eqYnTwqnRtI"
      },
      "outputs": [],
      "source": [
        "model.summarize();"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wZC2JM_8cqzR"
      },
      "source": [
        "------\n",
        "\n",
        "We now have a Transducer model ready to be trained!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1NkfrA2l6DBF"
      },
      "source": [
        "## (Optional) Partially loading pre-trained weights from another model\n",
        "\n",
        "An interesting point to note about Transducer models - the Acoustic model config (and therefore the Acoustic model itself) can be shared between CTC and Transducer models.\n",
        "\n",
        "This means that we can initialize the weights of a Transducer's Acoustic model with weights from a pre-trained CTC encoder model.\n",
        "\n",
        "------\n",
        "\n",
        "**Note**: This step is optional and not necessary at all to train a Transducer model. Below, we show the steps that we would take if we wanted to do this, however as the loaded model has different kernel sizes compared to the current model, the checkpoint cannot be loaded."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "OAjMrbK-dtIv"
      },
      "outputs": [],
      "source": [
        "# Load a small CTC model\n",
        "# ctc_model = nemo_asr.models.EncDecCTCModelBPE.from_pretrained(\"stt_en_citrinet_256\", map_location='cpu')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "urK_ZtVKeEpR"
      },
      "source": [
        "------\n",
        "\n",
        "Then load the state dict of the CTC model's encoder into the Transducer model's encoder."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Ugz8Y8eieLMK"
      },
      "outputs": [],
      "source": [
        "# <<< NOTE: This is only for demonstration ! >>>\n",
        "# Below cell will fail because the two model's have incompatible kernel sizes in their Conv layers.\n",
        "\n",
        "# <<< NOTE: Below cell is only shown to illustrate the method >>>\n",
        "# model.encoder.load_state_dict(ctc_model.encoder.state_dict(), strict=True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0VV1HNVD6DDc"
      },
      "source": [
        "# Training on AN4\n",
        "\n",
        "Now that the model is ready, we can finally train it!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "OS_U6uO-fnW9"
      },
      "outputs": [],
      "source": [
        "# Prepare NeMo's Experiment manager to handle checkpoint saving and logging for us\n",
        "from nemo.utils import exp_manager\n",
        "\n",
        "# Environment variable generally used for multi-node multi-gpu training.\n",
        "# In notebook environments, this flag is unnecessary and can cause logs of multiple training runs to overwrite each other.\n",
        "os.environ.pop('NEMO_EXPM_VERSION', None)\n",
        "\n",
        "exp_config = exp_manager.ExpManagerConfig(\n",
        "    exp_dir=f'experiments/',\n",
        "    name=f\"Transducer-Model\",\n",
        "    checkpoint_callback_params=exp_manager.CallbackParams(\n",
        "        monitor=\"val_wer\",\n",
        "        mode=\"min\",\n",
        "        always_save_nemo=True,\n",
        "        save_best_model=True,\n",
        "    ),\n",
        ")\n",
        "\n",
        "exp_config = OmegaConf.structured(exp_config)\n",
        "\n",
        "logdir = exp_manager.exp_manager(trainer, exp_config)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "SDleCL0Zf7lU"
      },
      "outputs": [],
      "source": [
        "try:\n",
        "  from google import colab\n",
        "  COLAB_ENV = True\n",
        "except (ImportError, ModuleNotFoundError):\n",
        "  COLAB_ENV = False\n",
        "\n",
        "# Load the TensorBoard notebook extension\n",
        "if COLAB_ENV:\n",
        "  %load_ext tensorboard\n",
        "  %tensorboard --logdir /content/experiments/Transducer-Model/\n",
        "else:\n",
        "  print(\"To use TensorBoard, please use this notebook in a Google Colab environment.\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "hzqAsK23uYHG"
      },
      "outputs": [],
      "source": [
        "# Release resources prior to training\n",
        "import gc\n",
        "gc.collect()\n",
        "\n",
        "if accelerator == 'gpu':\n",
        "  torch.cuda.empty_cache()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "background_save": true
        },
        "id": "A4neHTnSgaDb"
      },
      "outputs": [],
      "source": [
        "# Train the model\n",
        "trainer.fit(model)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "k0MgsBt_NyyT"
      },
      "source": [
        "-------\n",
        "\n",
        "Lets check what the final performance on the test set."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "background_save": true
        },
        "id": "iEjuXo4BNyOi"
      },
      "outputs": [],
      "source": [
        "trainer.test(model)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aY6S2SdeN9Be"
      },
      "source": [
        "------\n",
        "\n",
        "The model should obtain some score between 10-12% WER after 50 epochs of training. Quite a good score for just 50 epochs of training a tiny model! Note that these are greedy scores, yet they are pretty strong for such a short training run.\n",
        "\n",
        "We can further improve these scores by using the internal Prediction network to calculate beam scores."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eQUK6b266DF0"
      },
      "source": [
        "# Changing the Decoding Strategy\n",
        "\n",
        "During training, for the sake of efficiency, we were using the `greedy_batch` decoding strategy. However, we might want to perform inference with another method - say, beam search.\n",
        "\n",
        "NeMo allows changing the decoding strategy easily after the model has been trained."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "background_save": true
        },
        "id": "RJtrJKt0gY0i"
      },
      "outputs": [],
      "source": [
        "import copy\n",
        "\n",
        "decoding_config = copy.deepcopy(config.model.decoding)\n",
        "print(OmegaConf.to_yaml(decoding_config))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "background_save": true
        },
        "id": "MNL5L1KthC1E"
      },
      "outputs": [],
      "source": [
        "# Update the config for the decoding strategy\n",
        "decoding_config.strategy = \"alsd\"  # Options are `greedy`, `greedy_batch`, `beam`, `tsd` and `alsd`\n",
        "decoding_config.beam.beam_size = 4  # Increase beam size for better scores, but it will take much longer for transcription !"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "background_save": true
        },
        "id": "xQgQRDnlhC7M"
      },
      "outputs": [],
      "source": [
        "# Finally update the model's decoding strategy !\n",
        "model.change_decoding_strategy(decoding_config)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "h7jrCMJvh8vE"
      },
      "outputs": [],
      "source": [
        "trainer.test(model)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rPRFIeHsO8l7"
      },
      "source": [
        "------\n",
        "\n",
        "Here, we improved our scores significantly by using the `Alignment-Length Synchronous Decoding` beam search. Feel free to try the other algorithms and compare the speed-accuracy tradeoff!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dpUoqG_G6DII"
      },
      "source": [
        "# (Extra) Extracting Transducer Model Alignments \n",
        "\n",
        "Transducers are unique in the sense that for each timestep $t \\le T$, they can emit multiple target tokens $u_t$. During training, this is represented as the $T \\times U$ joint that maps to the vocabulary $V$. \n",
        "\n",
        "During inference, there is no need to compute the full joint $T \\times U$. Instead, after the model predicts the `Transducer Blank` token at the current timestep $t$ while predicting the target token $u_t$, the model will move onto the next acoustic timestep $t + 1$. As such, we can obtain the diagonal alignment of the Transducer model per sample relatively simply.\n",
        "\n",
        "------\n",
        "\n",
        "**Note**: While alignments can be calculated for both greedy and beam search - it is non-trivial to incorporate this alignment information for beam decoding. Therefore NeMo only supports extracting alignments during greedy decoding."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jjoWXknzP7En"
      },
      "source": [
        "-----\n",
        "\n",
        "Restore model to greedy decoding for alignment calculation"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "background_save": true
        },
        "id": "lgpsXSEeQAp4"
      },
      "outputs": [],
      "source": [
        "decoding_config.strategy = \"greedy_batch\"\n",
        "\n",
        "# Special flag which is generally disabled\n",
        "# Instruct Greedy Decoders to preserve alignment information during autoregressive decoding\n",
        "with open_dict(decoding_config):\n",
        "  decoding_config.preserve_alignments = True\n",
        "  decoding_config.fused_batch_size = -1  # temporarily stop fused batch during inference.\n",
        "\n",
        "model.change_decoding_strategy(decoding_config)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yUD44kBtdftZ"
      },
      "source": [
        "-------\n",
        "\n",
        "Set up a test data loader that we will use to obtain the alignments for a single batch. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "background_save": true
        },
        "id": "vrhldu5WPkvg"
      },
      "outputs": [],
      "source": [
        "test_dl = model.test_dataloader()\n",
        "test_dl = iter(test_dl)\n",
        "batch = next(test_dl)\n",
        "\n",
        "device = torch.device('cuda' if accelerator == 'gpu' else 'cpu')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "background_save": true
        },
        "id": "fAagUT_DPQhF"
      },
      "outputs": [],
      "source": [
        "def rnnt_alignments(model, batch):\n",
        "  model = model.to(device)\n",
        "  encoded, encoded_len = model.forward(\n",
        "                        input_signal=batch[0].to(device), input_signal_length=batch[1].to(device)\n",
        "                    )\n",
        "  \n",
        "  current_hypotheses = model.decoding.rnnt_decoder_predictions_tensor(\n",
        "                        encoded, encoded_len, return_hypotheses=True\n",
        "                    )\n",
        "  \n",
        "  del encoded, encoded_len\n",
        "  \n",
        "  # current hypothesis is a tuple of \n",
        "  # 1) best hypothesis \n",
        "  # 2) Sorted list of hypothesis (if using beam search); None otherwise\n",
        "  return current_hypotheses"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "background_save": true
        },
        "id": "OuSrv8lZPhwY"
      },
      "outputs": [],
      "source": [
        "# Get a batch of hypotheses, as well as a batch of all obtained hypotheses (if beam search is used)\n",
        "hypotheses, all_hypotheses = rnnt_alignments(model, batch)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "P4LknY78d3ZZ"
      },
      "source": [
        "------\n",
        "\n",
        "Select a sample ID from within the batch to observe the alignment information contained in the Hypothesis."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "background_save": true
        },
        "id": "arE7af_DPhyy"
      },
      "outputs": [],
      "source": [
        "# Select the sample ID from within the batch\n",
        "SAMPLE_ID = 0"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "S6YcE27WPh1G"
      },
      "outputs": [],
      "source": [
        "# Obtain the hypothesis for this sample, as well as some ground truth information about this sample\n",
        "hypothesis = hypotheses[SAMPLE_ID]\n",
        "original_sample_len = batch[1][SAMPLE_ID]\n",
        "ground_truth = batch[2][SAMPLE_ID]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "oEg_73h9Qe8t"
      },
      "outputs": [],
      "source": [
        "# The Hypothesis object contains a lot of useful information regarding the decoding step.\n",
        "print(hypothesis)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VlZRFF-kQU2d"
      },
      "source": [
        "-------\n",
        "\n",
        "Now, decode the hypothesis and compare it against the ground truth text. Note - this decoded hypothesis is at *sub-word* level for this model. Therefore sub-word tokens such as `_` may be seen here."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ALuefA4XPh5O"
      },
      "outputs": [],
      "source": [
        "decoded_text = hypothesis.text\n",
        "decoded_hypothesis = model.decoding.decode_ids_to_tokens(hypothesis.y_sequence.cpu().numpy().tolist())\n",
        "decoded_ground_truth = model.decoding.tokenizer.ids_to_text(ground_truth.cpu().numpy().tolist())\n",
        "\n",
        "print(\"Decoded ground truth :\", decoded_ground_truth)\n",
        "print(\"Decoded hypothesis   :\", decoded_text)\n",
        "print(\"Decoded hyp tokens   :\", decoded_hypothesis)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "J1djRcGrSeAQ"
      },
      "source": [
        "---------\n",
        "\n",
        "Next we print out the 2-d alignment grid of the RNNT model:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "vl1BF52iSjEq"
      },
      "outputs": [],
      "source": [
        "alignments = hypothesis.alignments\n",
        "\n",
        "# These two values should normally always match\n",
        "print(\"Length of alignments (T): \", len(alignments))\n",
        "print(\"Length of padded acoustic model after striding : \", int(hypothesis.length))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xkv_x8NAfpX3"
      },
      "source": [
        "------\n",
        "\n",
        "Finally, let us calculate the alignment grid. We will de-tokenize the sub-word token if it is a valid index in the vocabulary and use `''` as a placeholder for the `Transducer Blank` token.\n",
        "\n",
        "Note that each `timestep` here is (roughly) $timestep * total\\_stride\\_of\\_model * preprocessor.window\\_stride$ seconds timestamp. \n",
        "\n",
        "**Note**: You can modify the value of `config.model.loss.warprnnt_numba_kwargs.fastemit_lambda` prior to training and see an impact on final alignment latency!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Xt5nDL55SdRL"
      },
      "outputs": [],
      "source": [
        "# Compute the alignment grid\n",
        "for ti in range(len(alignments)):\n",
        "  t_u = []\n",
        "  for uj in range(len(alignments[ti])):\n",
        "    logprob, token = alignments[ti][uj]\n",
        "    token = token.to('cpu').numpy().tolist()\n",
        "    decoded_token = model.decoding.decode_ids_to_tokens([token])[0] if token != model.decoding.blank_id else ''  # token at index len(vocab) == RNNT blank token\n",
        "    t_u.append(decoded_token)\n",
        "  \n",
        "  print(f\"Tokens at timestep {ti} = {t_u}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ScgUTa-16DKu"
      },
      "source": [
        "# Takeaways\n",
        "------\n",
        "\n",
        "We covered significant ground in this tutorial, but here are a few key takeaways -\n",
        "\n",
        "1) **Any CTC config can be easily converted to a Transducer config** by copy-pasting the default Transducer config components.\n",
        "\n",
        "-------\n",
        "\n",
        "2) **Dataset processing for CTC and Transducer models are the same!** If it works for CTC it works exactly the same way for Transducers. This also applies to Character-based models vs. Sub-word tokenization-based models.\n",
        "\n",
        "-------\n",
        "\n",
        "3) **Fused Batch during training and evaluation significantly reduces the need for large GPU memory**, and we can simply reduce the `fused_batch_size` to process samples at a larger acoustic (global) batch size (the maximum batch size that can be passed through the acoustic model, disregarding the Prediction and Joint models).\n",
        "\n",
        "-------\n",
        "\n",
        "4) Once trained, **transducer models implicitly can be used for beam search**! The prediction network acts as an implicit language model and can easily be used for this beam search by switching the decoding configuration.\n",
        "\n",
        "-------\n",
        "\n",
        "5) **Alignments can be easily extracted from Transducers** by adding a special flag to the Decoding setup. After this, it is quite simple to extract the 2-d alignment grid of the transducer for samples or a batch of samples. It must be noted that this alignment information does require slightly more memory, and increases decoding time by a small amount as well.\n",
        "\n",
        "-------\n",
        "\n",
        "This is just the start of our journey into Transducers. There are several improvements coming out every few months for these models, and we encourage contributions to improve the transducer framework in NeMo!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gFvHVBIi4pRr"
      },
      "source": [
        "# Next Steps\n",
        "\n",
        "Head on over to the `examples/asr` directory in the NeMo repository in order to find the training scripts for Transducer based models - `sppech_to_text_rnnt.py` (for Character decoding based Transducer models) and `speech_to_text_rnnt_bpe.py` (for Sub-word decoding based Transducer models).\n",
        "\n",
        "You will find that following many of the steps from CTC models, and simply modifying the config to include the Transducer components, we can train character or sub-word based transducer models with the same flexibility as we can train CTC models !"
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "collapsed_sections": [
        "V5sMoFHmVvhg"
      ],
      "name": "ASR-with-Transducers.ipynb",
      "provenance": [],
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}