en-kam/jw300-baseline/starter_notebook.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Igc5itf-xMGj"
   },
   "source": [
    "# Masakhane - Machine Translation for African Languages (Using JoeyNMT)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "x4fXCKCf36IK"
   },
   "source": [
    "## Note before beginning:\n",
    "### - The idea is that you should be able to make minimal changes to this in order to get SOME result for your own translation corpus. \n",
    "\n",
    "### - The tl;dr: Go to the **\"TODO\"** comments which will tell you what to update to get up and running\n",
    "\n",
    "### - If you actually want to have a clue what you're doing, read the text and peek at the links\n",
    "\n",
    "### - With 100 epochs, it should take around 7 hours to run in Google Colab\n",
    "\n",
    "### - Once you've gotten a result for your language, please attach and email your notebook that generated it to masakhanetranslation@gmail.com\n",
    "\n",
    "### - If you care enough and get a chance, doing a brief background on your language would be amazing. See examples in  [(Martinus, 2019)](https://arxiv.org/abs/1906.05685)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "l929HimrxS0a"
   },
   "source": [
    "## Retrieve your data & make a parallel corpus\n",
    "\n",
    "If you are wanting to use the JW300 data referenced on the Masakhane website or in our GitHub repo, you can use `opus-tools` to convert the data into a convenient format. `opus_read` from that package provides a convenient tool for reading the native aligned XML files and to convert them to TMX format. The tool can also be used to fetch relevant files from OPUS on the fly and to filter the data as necessary. [Read the documentation](https://pypi.org/project/opustools-pkg/) for more details.\n",
    "\n",
    "Once you have your corpus files in TMX format (an xml structure which will include the sentences in your target language and your source language in a single file), we recommend reading them into a pandas dataframe. Thankfully, Jade wrote a silly `tmx2dataframe` package which converts your tmx file to a pandas dataframe. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "oGRmDELn7Az0"
   },
   "outputs": [],
   "source": [
    "# from google.colab import drive\n",
    "# drive.mount('/content/drive')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Cn3tgQLzUxwn"
   },
   "outputs": [],
   "source": [
    "# TODO: Set your source and target languages. Keep in mind, these traditionally use language codes as found here:\n",
    "# These will also become the suffix's of all vocab and corpus files used throughout\n",
    "import os\n",
    "source_language = \"en\"\n",
    "target_language = \"kam\" \n",
    "lc = False  # If True, lowercase the data.\n",
    "seed = 42  # Random seed for shuffling.\n",
    "tag = \"baseline\" # Give a unique name to your folder - this is to ensure you don't rewrite any models you've already submitted\n",
    "\n",
    "os.environ[\"src\"] = source_language # Sets them in bash as well, since we often use bash scripts\n",
    "os.environ[\"tgt\"] = target_language\n",
    "os.environ[\"tag\"] = tag\n",
    "\n",
    "# This will save it to a folder in our gdrive instead!\n",
    "# !mkdir -p \"/content/drive/My Drive/masakhane/$src-$tgt-$tag\"\n",
    "# os.environ[\"gdrive_path\"] = \"/content/drive/My Drive/masakhane/%s-%s-%s\" % (source_language, target_language, tag)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "kBSgJHEw7Nvx"
   },
   "outputs": [],
   "source": [
    "# !echo $gdrive_path"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "gA75Fs9ys8Y9"
   },
   "outputs": [],
   "source": [
    "# Install opus-tools\n",
    "#! pip install opustools-pkg"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Uncomment cell below if notebook is being run for the first time and you need to download the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "xq-tDZVks7ZD"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Alignment file /proj/nlpl/data/OPUS/JW300/latest/xml/en-kam.xml.gz not found. The following files are available for downloading:\n",
      "\n",
      " 548 KB https://object.pouta.csc.fi/OPUS-JW300/v1/xml/en-kam.xml.gz\n",
      " 263 MB https://object.pouta.csc.fi/OPUS-JW300/v1/xml/en.zip\n",
      "   6 MB https://object.pouta.csc.fi/OPUS-JW300/v1/xml/kam.zip\n",
      "\n",
      " 269 MB Total size\n",
      "./JW300_latest_xml_en-kam.xml.gz ... 100% of 548 KB\n",
      "./JW300_latest_xml_en.zip ... 100% of 263 MB\n",
      "./JW300_latest_xml_kam.zip ... 100% of 6 MB\n"
     ]
    }
   ],
   "source": [
    "# Downloading our corpus\n",
    "#! opus_read -d JW300 -s $src -t $tgt -wm moses -w jw300.$src jw300.$tgt -q\n",
    "\n",
    "# extract the corpus file\n",
    "#! gunzip JW300_latest_xml_$src-$tgt.xml.gz"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "n48GDRnP8y2G"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2020-02-18 07:58:38--  https://raw.githubusercontent.com/juliakreutzer/masakhane/master/jw300_utils/test/test.en-any.en\n",
      "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.192.133, 151.101.128.133, 151.101.64.133, ...\n",
      "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.192.133|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 277791 (271K) [text/plain]\n",
      "Saving to: ‘test.en-any.en’\n",
      "\n",
      "test.en-any.en      100%[===================>] 271.28K  --.-KB/s    in 0.05s   \n",
      "\n",
      "2020-02-18 07:58:39 (5.15 MB/s) - ‘test.en-any.en’ saved [277791/277791]\n",
      "\n",
      "--2020-02-18 07:58:39--  https://raw.githubusercontent.com/juliakreutzer/masakhane/master/jw300_utils/test/test.en-kam.en\n",
      "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.192.133, 151.101.128.133, 151.101.64.133, ...\n",
      "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.192.133|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 197066 (192K) [text/plain]\n",
      "Saving to: ‘test.en-kam.en’\n",
      "\n",
      "test.en-kam.en      100%[===================>] 192.45K  --.-KB/s    in 0.04s   \n",
      "\n",
      "2020-02-18 07:58:39 (4.70 MB/s) - ‘test.en-kam.en’ saved [197066/197066]\n",
      "\n",
      "--2020-02-18 07:58:39--  https://raw.githubusercontent.com/juliakreutzer/masakhane/master/jw300_utils/test/test.en-kam.kam\n",
      "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.192.133, 151.101.128.133, 151.101.64.133, ...\n",
      "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.192.133|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 231348 (226K) [text/plain]\n",
      "Saving to: ‘test.en-kam.kam’\n",
      "\n",
      "test.en-kam.kam     100%[===================>] 225.93K  --.-KB/s    in 0.04s   \n",
      "\n",
      "2020-02-18 07:58:40 (5.31 MB/s) - ‘test.en-kam.kam’ saved [231348/231348]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Download the global test set.\n",
    "! wget https://raw.githubusercontent.com/juliakreutzer/masakhane/master/jw300_utils/test/test.en-any.en\n",
    "  \n",
    "# And the specific test set for this language pair.\n",
    "os.environ[\"trg\"] = target_language \n",
    "os.environ[\"src\"] = source_language \n",
    "\n",
    "! wget https://raw.githubusercontent.com/juliakreutzer/masakhane/master/jw300_utils/test/test.en-$trg.en \n",
    "! mv test.en-$trg.en test.en\n",
    "! wget https://raw.githubusercontent.com/juliakreutzer/masakhane/master/jw300_utils/test/test.en-$trg.$trg \n",
    "! mv test.en-$trg.$trg test.$trg"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "NqDG-CI28y2L"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loaded 3571 global test sentences to filter from the training/dev data.\n"
     ]
    }
   ],
   "source": [
    "# Read the test data to filter from train and dev splits.\n",
    "# Store english portion in set for quick filtering checks.\n",
    "en_test_sents = set()\n",
    "filter_test_sents = \"test.en-any.en\"\n",
    "j = 0\n",
    "with open(filter_test_sents) as f:\n",
    "  for line in f:\n",
    "    en_test_sents.add(line.strip())\n",
    "    j += 1\n",
    "print('Loaded {} global test sentences to filter from the training/dev data.'.format(j))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "3CNdwLBCfSIl"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loaded data and skipped 3688/58312 lines since contained in test set.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>source_sentence</th>\n",
       "      <th>target_sentence</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td></td>\n",
       "      <td>Kĩla Ĩkaseti Yĩtumbĩthaw’a : 45,944,000 KWA IT...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td></td>\n",
       "      <td>MATUKŨ 3 - 9 , MWEI WA 3 , 2014 | ĨTHANGŨ YA 7...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td></td>\n",
       "      <td>Mũthaithae Yeova , Mũsumbĩ wa Tene na Tene</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  source_sentence                                    target_sentence\n",
       "0                  Kĩla Ĩkaseti Yĩtumbĩthaw’a : 45,944,000 KWA IT...\n",
       "1                  MATUKŨ 3 - 9 , MWEI WA 3 , 2014 | ĨTHANGŨ YA 7...\n",
       "2                         Mũthaithae Yeova , Mũsumbĩ wa Tene na Tene"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "# TMX file to dataframe\n",
    "source_file = 'jw300.' + source_language\n",
    "target_file = 'jw300.' + target_language\n",
    "\n",
    "source = []\n",
    "target = []\n",
    "skip_lines = []  # Collect the line numbers of the source portion to skip the same lines for the target portion.\n",
    "with open(source_file) as f:\n",
    "    for i, line in enumerate(f):\n",
    "        # Skip sentences that are contained in the test set.\n",
    "        if line.strip() not in en_test_sents:\n",
    "            source.append(line.strip())\n",
    "        else:\n",
    "            skip_lines.append(i)             \n",
    "with open(target_file) as f:\n",
    "    for j, line in enumerate(f):\n",
    "        # Only add to corpus if corresponding source was not skipped.\n",
    "        if j not in skip_lines:\n",
    "            target.append(line.strip())\n",
    "    \n",
    "print('Loaded data and skipped {}/{} lines since contained in test set.'.format(len(skip_lines), i))\n",
    "    \n",
    "df = pd.DataFrame(zip(source, target), columns=['source_sentence', 'target_sentence'])\n",
    "df.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "YkuK3B4p2AkN"
   },
   "source": [
    "## Pre-processing and export\n",
    "\n",
    "It is generally a good idea to remove duplicate translations and conflicting translations from the corpus. In practice, these public corpora include some number of these that need to be cleaned.\n",
    "\n",
    "In addition we will split our data into dev/test/train and export to the filesystem."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "M_2ouEOH1_1q"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.6/dist-packages/pandas/core/generic.py:6786: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame\n",
      "\n",
      "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  self._update_inplace(new_data)\n",
      "/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:8: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame\n",
      "\n",
      "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  \n",
      "/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:9: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame\n",
      "\n",
      "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  if __name__ == '__main__':\n",
      "/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:12: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame\n",
      "\n",
      "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  if sys.path[0] == '':\n",
      "/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:13: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame\n",
      "\n",
      "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  del sys.path[0]\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "# drop duplicate translations\n",
    "df_pp = df.drop_duplicates()\n",
    "\n",
    "#drop empty lines (alp)\n",
    "df_pp['source_sentence'].replace('', np.nan, inplace=True)\n",
    "df_pp['target_sentence'].replace('', np.nan, inplace=True)\n",
    "df_pp.dropna(subset=['source_sentence'], inplace=True)\n",
    "df_pp.dropna(subset=['target_sentence'], inplace=True)\n",
    "\n",
    "# drop conflicting translations\n",
    "df_pp.drop_duplicates(subset='source_sentence', inplace=True)\n",
    "df_pp.drop_duplicates(subset='target_sentence', inplace=True)\n",
    "\n",
    "# Shuffle the data to remove bias in dev set selection.\n",
    "df_pp = df_pp.sample(frac=1, random_state=seed).reset_index(drop=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "hxxBOCA-xXhy"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "==> train.en <==\n",
      "What can cause us to ask : “ How long ” ?\n",
      "If so , it would not be hard to understand why . There was a lot of history in this house .\n",
      "He exercised his authority by driving the man and woman out of the garden of Eden , and to prevent their return , he assigned cherubs to stand guard at the entrance .\n",
      "Then , referring to Jehovah as ‘ sending the snow , scattering the frost , and hurling down the hailstones , ’ the psalmist asks : “ Who can withstand his cold ? ”\n",
      "If you are a single sister with a heartfelt desire to share in a ministry that is deeply satisfying , we are sure you will benefit from their comments .\n",
      "Because the Bible tells us : “ As for the heavens , they belong to Jehovah , but the earth he has given to the sons of men . ”\n",
      "So be diligent in your study of God’s Word , meditate deeply on its meaning , and look for ways to express your faith at congregation meetings .\n",
      "Yet , it would be contrary to God’s direction to date someone who is not dedicated to him and faithful to his standards . ​ — 8 / 15 , p .\n",
      "Did the physical universe have a beginning ?\n",
      "Likely , he performed many other miracles .\n",
      "\n",
      "==> train.kam <==\n",
      "Kyo nĩ kyaũ kĩtonya kũtuma twĩkũlya : “ Ngaĩa ĩvinda yĩana ata ? ”\n",
      "Ethĩwa nĩwĩmakĩe , ti vinya kũmanya nĩkĩ ũilyĩ ũu .\n",
      "Oonanisye e na ũkũmũ ĩla walũngilye mũndũũme na mũndũ mũka mũũndanĩ wa Eteni , na aia akeluvi masiĩĩe nzĩa ya kũlika mũũndanĩ nĩ kana andũ asu maikasyoke . ( Mwa .\n",
      "Asyokete akasya kana Yeova ‘ nũnenganae ĩa , nũnyaĩĩkasya ĩmwe , na nĩwĩkasya manzĩ make momĩtw’e nĩ mbevo me tũlungu , ’ na ĩndĩ akakũlya atĩĩ : “ Nũũ ũtonya kũũngama mbee wa mbevo yake ? ”\n",
      "Ethĩwa wĩ mwĩĩtu - a - asa ũte mũtwae na nĩwĩthĩawa na mea ma kũtanĩa ũtavany’a waku mũnango , tũi na nzika kana ndeto syoo niũkũtethya .\n",
      "Nũndũ Mbivilia yaitye atĩĩ : “ Matu nĩ matu ma Yeova ; ĩndĩ nthĩ nũnengete ana ma andũ . ”\n",
      "Kwoou ĩmanyĩasye Ndeto ya Ngai na kĩthito , vindĩĩasya kĩla ũkwĩmanyĩsya , na ĩla wĩ maũmbanonĩ , umasya maelesyo mekwonany’a wĩ na mũĩkĩĩo mũlũmu .\n",
      "( 1 Ako . 15 : 33 ) — 15 / 8 , ĩth .\n",
      "Matũ na nthĩ syĩ mwambĩĩo ?\n",
      "Nĩvatonyeka ũkethĩa eekie syama ingĩ mbingĩ eka ila syĩ Mbivilianĩ .\n",
      "==> dev.en <==\n",
      "Gideon wondered how it would be possible for him to “ save Israel out of Midian’s hand . ”\n",
      "And then she would try to return to Hosea .\n",
      "The Bible itself says : “ The green grass dries up , the blossom withers , but the word of our God endures forever . ” ​ — Isaiah 40 : 8 .\n",
      "It includes links to lists of Witnesses currently imprisoned for their faith .\n",
      "( January 1 , 2014 ) .\n",
      "In vision , the apostle John heard Jehovah’s servants in heaven say : “ You are worthy , Jehovah our God , to receive the glory and the honor and the power , because you created all things , and because of your will they came into existence and were created . ”\n",
      "Although David made a serious mistake in breaking God’s law , he revealed what was truly in his heart by sincerely repenting and throwing himself on God’s mercy . ​ — Psalm 51 .\n",
      "Jehovah sees to it that you are enriched beyond measure . ”\n",
      "By choosing to count my blessings each day , I can see what a privilege it is to be known and loved by our protective heavenly Father .\n",
      "Is it proper for a brother today to have a beard ?\n",
      "\n",
      "==> dev.kam <==\n",
      "Mũlaĩka ũsu nĩwaneenie vandũ va Mũmbi na amũĩkĩĩthya Ngiteoni kana Yeova aĩ vamwe nake .\n",
      "Na ĩndĩ kyamina kũsembany’a na endwa makyo Yeova aĩtye kĩkatata kũmũsyokea Osea .\n",
      "Mbivilia yaĩtye atĩĩ : “ Nyeki nĩyũmaa , na ĩlaa nĩyĩvovaa ; ĩndĩ ndeto ya Ngai waitũ ĩkekala tene na tene . ” — Isaia 40 : 8 .\n",
      "7 : 12 ) Walika Kĩsesenĩ kya jw.org no wone masyĩtwa ma Ngũsĩ ila syovetwe .\n",
      "( 01 / 01 / 2014 ) .\n",
      "Woninĩ ũla mũtũmwa Yoana wooniw’e nĩweewie athũkũmi ma Yeova ĩtunĩ mayasya : “ Nĩwaĩle we , Mwĩaĩi waitũ na Ngai waitũ , kwosa ndaĩa na nguma na vinya : nĩkwĩthĩwa nĩwoombie syĩndũ syonthe , na kwondũ wa kwenda kwaku syaĩ kw’o , na ĩngĩ syoombwa . ” ( Ũvu .\n",
      "O na kau Ndaviti nĩweekie naĩ ngito kwa kũtũla mwĩao wa Ngai , nĩwoonanisye kĩla kyaĩ ngoonĩ yake ĩla weetĩkĩlile mavĩtyo make , na eetya Ngai ũekeo . — Savuli 51 .\n",
      "Yeova akekalaa ayĩkĩĩthya kana nĩwakũathima mũno kũthonoka o na ũndũ ũtonya kũsũanĩa . ”\n",
      "Kũtalaa kĩla mũthenya moathimo ala nĩkwatĩte nĩkũndetheeasya kwona ũndũ ne ũndũ wa mwanya kwĩthĩwa nendetwe na ngĩsũvĩwa nĩ Ĩthe witũ wa ĩtunĩ .\n",
      "Ve ũthũku ũmũnthĩ mwana - a - asa akaĩthya kĩng’ee ?\n"
     ]
    }
   ],
   "source": [
    "# This section does the split between train/dev for the parallel corpora then saves them as separate files\n",
    "# We use 1000 dev test and the given test set.\n",
    "import csv\n",
    "\n",
    "# Do the split between dev/train and create parallel corpora\n",
    "num_dev_patterns = 1000\n",
    "\n",
    "# Optional: lower case the corpora - this will make it easier to generalize, but without proper casing.\n",
    "if lc:  # Julia: making lowercasing optional\n",
    "    df_pp[\"source_sentence\"] = df_pp[\"source_sentence\"].str.lower()\n",
    "    df_pp[\"target_sentence\"] = df_pp[\"target_sentence\"].str.lower()\n",
    "\n",
    "# Julia: test sets are already generated\n",
    "dev = df_pp.tail(num_dev_patterns) # Herman: Error in original\n",
    "stripped = df_pp.drop(df_pp.tail(num_dev_patterns).index)\n",
    "\n",
    "with open(\"train.\"+source_language, \"w\") as src_file, open(\"train.\"+target_language, \"w\") as trg_file:\n",
    "  for index, row in stripped.iterrows():\n",
    "    src_file.write(row[\"source_sentence\"]+\"\\n\")\n",
    "    trg_file.write(row[\"target_sentence\"]+\"\\n\")\n",
    "    \n",
    "with open(\"dev.\"+source_language, \"w\") as src_file, open(\"dev.\"+target_language, \"w\") as trg_file:\n",
    "  for index, row in dev.iterrows():\n",
    "    src_file.write(row[\"source_sentence\"]+\"\\n\")\n",
    "    trg_file.write(row[\"target_sentence\"]+\"\\n\")\n",
    "\n",
    "#stripped[[\"source_sentence\"]].to_csv(\"train.\"+source_language, header=False, index=False)  # Herman: Added `header=False` everywhere\n",
    "#stripped[[\"target_sentence\"]].to_csv(\"train.\"+target_language, header=False, index=False)  # Julia: Problematic handling of quotation marks.\n",
    "\n",
    "#dev[[\"source_sentence\"]].to_csv(\"dev.\"+source_language, header=False, index=False)\n",
    "#dev[[\"target_sentence\"]].to_csv(\"dev.\"+target_language, header=False, index=False)\n",
    "\n",
    "\n",
    "# Doublecheck the format below. There should be no extra quotation marks or weird characters.\n",
    "! head train.*\n",
    "! head dev.*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "epeCydmCyS8X"
   },
   "source": [
    "\n",
    "\n",
    "---\n",
    "\n",
    "\n",
    "## Installation of JoeyNMT\n",
    "\n",
    "JoeyNMT is a simple, minimalist NMT package which is useful for learning and teaching. Check out the documentation for JoeyNMT [here](https://joeynmt.readthedocs.io)  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "iBRMm4kMxZ8L"
   },
   "outputs": [],
   "source": [
    "# Install JoeyNMT\n",
    "#! git clone https://github.com/joeynmt/joeynmt.git\n",
    "#! cd joeynmt; pip3 install ."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "AaE77Tcppex9"
   },
   "source": [
    "# Preprocessing the Data into Subword BPE Tokens\n",
    "\n",
    "- One of the most powerful improvements for agglutinative languages (a feature of most Bantu languages) is using BPE tokenization [ (Sennrich, 2015) ](https://arxiv.org/abs/1508.07909).\n",
    "\n",
    "- It was also shown that by optimizing the umber of BPE codes we significantly improve results for low-resourced languages [(Sennrich, 2019)](https://www.aclweb.org/anthology/P19-1021) [(Martinus, 2019)](https://arxiv.org/abs/1906.05685)\n",
    "\n",
    "- Below we have the scripts for doing BPE tokenization of our data. We use 4000 tokens as recommended by [(Sennrich, 2019)](https://www.aclweb.org/anthology/P19-1021). You do not need to change anything. Simply running the below will be suitable. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# One of the huge boosts in NMT performance was to use a different method of tokenizing. \n",
    "# Usually, NMT would tokenize by words. However, using a method called BPE gave amazing boosts to performance\n",
    "\n",
    "# Do subword NMT\n",
    "from os import path\n",
    "os.environ[\"src\"] = source_language # Sets them in bash as well, since we often use bash scripts\n",
    "os.environ[\"tgt\"] = target_language\n",
    "\n",
    "# Learn BPEs on the training data.\n",
    "os.environ[\"data_path\"] = path.join(\"../../joeynmt\", \"data\", source_language + target_language) # Herman! \n",
    "! subword-nmt learn-joint-bpe-and-vocab --input train.$src train.$tgt -s 4000 -o bpe.codes.4000 --write-vocabulary vocab.$src vocab.$tgt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Apply BPE splits to the development and test data.\n",
    "! subword-nmt apply-bpe -c bpe.codes.4000 --vocabulary vocab.$src < train.$src > train.bpe.$src\n",
    "! subword-nmt apply-bpe -c bpe.codes.4000 --vocabulary vocab.$tgt < train.$tgt > train.bpe.$tgt\n",
    "\n",
    "! subword-nmt apply-bpe -c bpe.codes.4000 --vocabulary vocab.$src < dev.$src > dev.bpe.$src\n",
    "! subword-nmt apply-bpe -c bpe.codes.4000 --vocabulary vocab.$tgt < dev.$tgt > dev.bpe.$tgt\n",
    "! subword-nmt apply-bpe -c bpe.codes.4000 --vocabulary vocab.$src < test.$src > test.bpe.$src\n",
    "! subword-nmt apply-bpe -c bpe.codes.4000 --vocabulary vocab.$tgt < test.$tgt > test.bpe.$tgt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "! sudo chmod 777 ../../joeynmt/data/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "bpe.codes.4000\tdev.en\t     test.bpe.kam    test.kam\t    train.en\r\n",
      "dev.bpe.en\tdev.kam      test.en\t     train.bpe.en   train.kam\r\n",
      "dev.bpe.kam\ttest.bpe.en  test.en-any.en  train.bpe.kam\r\n"
     ]
    }
   ],
   "source": [
    "# Create directory, move everyone we care about to the correct location\n",
    "! mkdir -p $data_path\n",
    "! cp train.* $data_path\n",
    "! cp test.* $data_path\n",
    "! cp dev.* $data_path\n",
    "! cp bpe.codes.4000 $data_path\n",
    "! ls $data_path"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Also move everything we care about to a mounted location in google drive (relevant if running in colab) at gdrive_path\n",
    "# ! cp train.* \"$gdrive_path\"\n",
    "# ! cp test.* \"$gdrive_path\"\n",
    "# ! cp dev.* \"$gdrive_path\"\n",
    "# ! cp bpe.codes.4000 \"$gdrive_path\"\n",
    "# ! ls \"$gdrive_path\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create that vocab using build_vocab\n",
    "! sudo chmod 777 ../../joeynmt/scripts/build_vocab.py\n",
    "! ../../joeynmt/scripts/build_vocab.py ../../joeynmt/data/$src$tgt/train.bpe.$src ../../joeynmt/data/$src$tgt/train.bpe.$tgt --output_path ../../joeynmt/data/$src$tgt/vocab.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "H-TyjtmXB1mL"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "BPE Kamba Sentences\n",
      "6 : 10 ) O na ethĩwa wĩana ata , no wĩ@@ ke mĩvango ya kĩ - veva .\n",
      "Ĩandĩko ya Nthimo 21 : 5 nthĩnĩ wa Mbivilia ya Kĩk@@ amba kya ĩvinda yĩĩ yaĩtye : “ M@@ ĩv@@ ango ya ala me kĩthito kwa w’o ĩ@@ mat@@ ongo@@ easya methĩwe na syĩndũ mbingĩ . ”\n",
      "W@@ eeka mĩvango tene , wa@@ mbĩ@@ ĩaa o kũendeea nesa tene .\n",
      "K@@ ũth@@ i y@@ un@@ iv@@ as@@ iti nĩ@@ kwate nd@@ ik@@ ili@@ i ya mĩao kw@@ esaa kũnenga wĩa wĩ mbesa , ĩndĩ nd@@ yaĩ nĩ@@ s@@ a kũkwata wĩa wa mas@@ aa man@@ ini . ”\n",
      "17 , 18 . ( a ) Yeova ende@@ aa amũ@@ ika kyaũ ?\n",
      "Combined BPE Vocab\n",
      "Isra@@\n",
      "sider\n",
      "ʺ\n",
      "Ũvu@@\n",
      "Revel@@\n",
      "pub@@\n",
      "ʼ\n",
      "ā@@\n",
      "espec@@\n",
      "▲\n"
     ]
    }
   ],
   "source": [
    "# Some output\n",
    "! echo \"BPE Kamba Sentences\"\n",
    "! tail -n 5 test.bpe.$tgt\n",
    "! echo \"Combined BPE Vocab\"\n",
    "! tail -n 10 ../../joeynmt/data/$src$tgt/vocab.txt  # Herman"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "IlMitUHR8Qy-"
   },
   "outputs": [],
   "source": [
    "# Also move everything we care about to a mounted location in google drive (relevant if running in colab) at gdrive_path\n",
    "#! cp train.* \"$gdrive_path\"\n",
    "#! cp test.* \"$gdrive_path\"\n",
    "#! cp dev.* \"$gdrive_path\"\n",
    "#! cp bpe.codes.4000 \"$gdrive_path\"\n",
    "#! ls \"$gdrive_path\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Ixmzi60WsUZ8"
   },
   "source": [
    "# Creating the JoeyNMT Config\n",
    "\n",
    "JoeyNMT requires a yaml config. We provide a template below. We've also set a number of defaults with it, that you may play with!\n",
    "\n",
    "- We used Transformer architecture \n",
    "- We set our dropout to reasonably high: 0.3 (recommended in  [(Sennrich, 2019)](https://www.aclweb.org/anthology/P19-1021))\n",
    "\n",
    "Things worth playing with:\n",
    "- The batch size (also recommended to change for low-resourced languages)\n",
    "- The number of epochs (we've set it at 30 just so it runs in about an hour, for testing purposes)\n",
    "- The decoder options (beam_size, alpha)\n",
    "- Evaluation metrics (BLEU versus Crhf4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "PIs1lY2hxMsl"
   },
   "outputs": [],
   "source": [
    "# This creates the config file for our JoeyNMT system. It might seem overwhelming so we've provided a couple of useful parameters you'll need to update\n",
    "# (You can of course play with all the parameters if you'd like!)\n",
    "\n",
    "name = '%s%s' % (source_language, target_language)\n",
    "# gdrive_path = os.environ[\"gdrive_path\"]\n",
    "\n",
    "# Create the config\n",
    "config = \"\"\"\n",
    "name: \"{name}_transformer\"\n",
    "\n",
    "data:\n",
    "    src: \"{source_language}\"\n",
    "    trg: \"{target_language}\"\n",
    "    train: \"data/{name}/train.bpe\"\n",
    "    dev:   \"data/{name}/dev.bpe\"\n",
    "    test:  \"data/{name}/test.bpe\"\n",
    "    level: \"bpe\"\n",
    "    lowercase: False\n",
    "    max_sent_length: 100\n",
    "    src_vocab: \"data/{name}/vocab.txt\"\n",
    "    trg_vocab: \"data/{name}/vocab.txt\"\n",
    "\n",
    "testing:\n",
    "    beam_size: 5\n",
    "    alpha: 1.0\n",
    "\n",
    "training:\n",
    "    load_model: \"models/{name}_transformer/1000.ckpt\" # if uncommented, load a pre-trained model from this checkpoint\n",
    "    random_seed: 42\n",
    "    optimizer: \"adam\"\n",
    "    normalization: \"tokens\"\n",
    "    adam_betas: [0.9, 0.999] \n",
    "    scheduling: \"plateau\"           # TODO: try switching from plateau to Noam scheduling\n",
    "    patience: 5                     # For plateau: decrease learning rate by decrease_factor if validation score has not improved for this many validation rounds.\n",
    "    learning_rate_factor: 0.5       # factor for Noam scheduler (used with Transformer)\n",
    "    learning_rate_warmup: 1000      # warmup steps for Noam scheduler (used with Transformer)\n",
    "    decrease_factor: 0.7\n",
    "    loss: \"crossentropy\"\n",
    "    learning_rate: 0.0003\n",
    "    learning_rate_min: 0.00000001\n",
    "    weight_decay: 0.0\n",
    "    label_smoothing: 0.1\n",
    "    batch_size: 4096\n",
    "    batch_type: \"token\"\n",
    "    eval_batch_size: 3600\n",
    "    eval_batch_type: \"token\"\n",
    "    batch_multiplier: 1\n",
    "    early_stopping_metric: \"ppl\"\n",
    "    epochs: 28                     # TODO: Decrease for when playing around and checking of working. Around 30 is sufficient to check if its working at all\n",
    "    validation_freq: 1000          # TODO: Set to at least once per epoch.\n",
    "    logging_freq: 100\n",
    "    eval_metric: \"bleu\"\n",
    "    model_dir: \"models/{name}_transformer2\"\n",
    "    overwrite: False               # TODO: Set to True if you want to overwrite possibly existing models. \n",
    "    shuffle: True\n",
    "    use_cuda: True\n",
    "    max_output_length: 100\n",
    "    print_valid_sents: [0, 1, 2, 3]\n",
    "    keep_last_ckpts: 3\n",
    "\n",
    "model:\n",
    "    initializer: \"xavier\"\n",
    "    bias_initializer: \"zeros\"\n",
    "    init_gain: 1.0\n",
    "    embed_initializer: \"xavier\"\n",
    "    embed_init_gain: 1.0\n",
    "    tied_embeddings: True\n",
    "    tied_softmax: True\n",
    "    encoder:\n",
    "        type: \"transformer\"\n",
    "        num_layers: 6\n",
    "        num_heads: 4             # TODO: Increase to 8 for larger data.\n",
    "        embeddings:\n",
    "            embedding_dim: 256   # TODO: Increase to 512 for larger data.\n",
    "            scale: True\n",
    "            dropout: 0.2\n",
    "        # typically ff_size = 4 x hidden_size\n",
    "        hidden_size: 256         # TODO: Increase to 512 for larger data.\n",
    "        ff_size: 1024            # TODO: Increase to 2048 for larger data.\n",
    "        dropout: 0.3\n",
    "    decoder:\n",
    "        type: \"transformer\"\n",
    "        num_layers: 6\n",
    "        num_heads: 4              # TODO: Increase to 8 for larger data.\n",
    "        embeddings:\n",
    "            embedding_dim: 256    # TODO: Increase to 512 for larger data.\n",
    "            scale: True\n",
    "            dropout: 0.2\n",
    "        # typically ff_size = 4 x hidden_size\n",
    "        hidden_size: 256         # TODO: Increase to 512 for larger data.\n",
    "        ff_size: 1024            # TODO: Increase to 2048 for larger data.\n",
    "        dropout: 0.3\n",
    "\"\"\".format(name=name, gdrive_path=\"n/a\", source_language=source_language, target_language=target_language)\n",
    "with open(\"../../joeynmt/configs/transformer_{name}.yaml\".format(name=name),'w') as f:\n",
    "    f.write(config)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "pIifxE3Qzuvs"
   },
   "source": [
    "# Train the Model\n",
    "\n",
    "This single line of joeynmt runs the training using the config we made above"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "total 136\r\n",
      "drwxr-xr-x 11 root root  4096 Oct 25 12:22 .\r\n",
      "drwxrwxrwx  9 root root  4096 Feb 18 07:57 ..\r\n",
      "drwxr-xr-x  8 root root  4096 Oct 24 15:14 .git\r\n",
      "-rw-r--r--  1 root root    49 Oct 24 15:14 .gitattributes\r\n",
      "drwxr-xr-x  3 root root  4096 Oct 24 15:14 .github\r\n",
      "-rw-r--r--  1 root root    71 Oct 24 15:14 .gitignore\r\n",
      "-rw-r--r--  1 root root 13514 Oct 24 15:14 .pylintrc\r\n",
      "-rw-r--r--  1 root root   159 Oct 24 15:14 .readthedocs.yml\r\n",
      "-rw-r--r--  1 root root   542 Oct 24 15:14 .travis.yml\r\n",
      "-rwxrw-rwx  1 root root  3354 Oct 24 15:14 CODE_OF_CONDUCT.md\r\n",
      "-rwxrw-rwx  1 root root  1071 Oct 24 15:14 LICENSE\r\n",
      "-rwxrw-rwx  1 root root 13286 Oct 24 15:14 README.md\r\n",
      "-rwxrw-rwx  1 root root  8229 Oct 24 15:14 benchmarks.md\r\n",
      "drwxrw-rwx  3 root root  4096 Feb 18 08:12 configs\r\n",
      "drwxrwxrwx  5 root root  4096 Feb 18 08:11 data\r\n",
      "drwxrw-rwx  4 root root  4096 Oct 24 15:14 docs\r\n",
      "-rwxrw-rwx  1 root root 14373 Oct 24 15:14 joey-small.png\r\n",
      "drwxrw-rwx  3 root root  4096 Oct 24 16:35 joeynmt\r\n",
      "drwxrwxrwx  5 root root  4096 Feb 18 08:12 models\r\n",
      "-rwxrw-rwx  1 root root   167 Oct 24 15:14 requirements.txt\r\n",
      "drwxrw-rwx  2 root root  4096 Oct 24 15:14 scripts\r\n",
      "-rwxrw-rwx  1 root root   810 Oct 24 15:14 setup.py\r\n",
      "drwxrw-rwx  4 root root  4096 Oct 24 15:14 test\r\n"
     ]
    }
   ],
   "source": [
    "! ls -la ../../joeynmt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "! sudo chmod 777 ../../joeynmt/models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "6ZBPFwT94WpI"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/home/espoir_mur_gmail_com/.local/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_qint8 = np.dtype([(\"qint8\", np.int8, 1)])\n",
      "/home/espoir_mur_gmail_com/.local/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_quint8 = np.dtype([(\"quint8\", np.uint8, 1)])\n",
      "/home/espoir_mur_gmail_com/.local/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_qint16 = np.dtype([(\"qint16\", np.int16, 1)])\n",
      "/home/espoir_mur_gmail_com/.local/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_quint16 = np.dtype([(\"quint16\", np.uint16, 1)])\n",
      "/home/espoir_mur_gmail_com/.local/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_qint32 = np.dtype([(\"qint32\", np.int32, 1)])\n",
      "/home/espoir_mur_gmail_com/.local/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  np_resource = np.dtype([(\"resource\", np.ubyte, 1)])\n",
      "2020-02-18 09:38:56,718 Hello! This is Joey-NMT.\n",
      "2020-02-18 09:38:56,724 Total params: 12132864\n",
      "2020-02-18 09:38:56,725 Trainable parameters: ['decoder.layer_norm.bias', 'decoder.layer_norm.weight', 'decoder.layers.0.dec_layer_norm.bias', 'decoder.layers.0.dec_layer_norm.weight', 'decoder.layers.0.feed_forward.layer_norm.bias', 'decoder.layers.0.feed_forward.layer_norm.weight', 'decoder.layers.0.feed_forward.pwff_layer.0.bias', 'decoder.layers.0.feed_forward.pwff_layer.0.weight', 'decoder.layers.0.feed_forward.pwff_layer.3.bias', 'decoder.layers.0.feed_forward.pwff_layer.3.weight', 'decoder.layers.0.src_trg_att.k_layer.bias', 'decoder.layers.0.src_trg_att.k_layer.weight', 'decoder.layers.0.src_trg_att.output_layer.bias', 'decoder.layers.0.src_trg_att.output_layer.weight', 'decoder.layers.0.src_trg_att.q_layer.bias', 'decoder.layers.0.src_trg_att.q_layer.weight', 'decoder.layers.0.src_trg_att.v_layer.bias', 'decoder.layers.0.src_trg_att.v_layer.weight', 'decoder.layers.0.trg_trg_att.k_layer.bias', 'decoder.layers.0.trg_trg_att.k_layer.weight', 'decoder.layers.0.trg_trg_att.output_layer.bias', 'decoder.layers.0.trg_trg_att.output_layer.weight', 'decoder.layers.0.trg_trg_att.q_layer.bias', 'decoder.layers.0.trg_trg_att.q_layer.weight', 'decoder.layers.0.trg_trg_att.v_layer.bias', 'decoder.layers.0.trg_trg_att.v_layer.weight', 'decoder.layers.0.x_layer_norm.bias', 'decoder.layers.0.x_layer_norm.weight', 'decoder.layers.1.dec_layer_norm.bias', 'decoder.layers.1.dec_layer_norm.weight', 'decoder.layers.1.feed_forward.layer_norm.bias', 'decoder.layers.1.feed_forward.layer_norm.weight', 'decoder.layers.1.feed_forward.pwff_layer.0.bias', 'decoder.layers.1.feed_forward.pwff_layer.0.weight', 'decoder.layers.1.feed_forward.pwff_layer.3.bias', 'decoder.layers.1.feed_forward.pwff_layer.3.weight', 'decoder.layers.1.src_trg_att.k_layer.bias', 'decoder.layers.1.src_trg_att.k_layer.weight', 'decoder.layers.1.src_trg_att.output_layer.bias', 'decoder.layers.1.src_trg_att.output_layer.weight', 'decoder.layers.1.src_trg_att.q_layer.bias', 'decoder.layers.1.src_trg_att.q_layer.weight', 'decoder.layers.1.src_trg_att.v_layer.bias', 'decoder.layers.1.src_trg_att.v_layer.weight', 'decoder.layers.1.trg_trg_att.k_layer.bias', 'decoder.layers.1.trg_trg_att.k_layer.weight', 'decoder.layers.1.trg_trg_att.output_layer.bias', 'decoder.layers.1.trg_trg_att.output_layer.weight', 'decoder.layers.1.trg_trg_att.q_layer.bias', 'decoder.layers.1.trg_trg_att.q_layer.weight', 'decoder.layers.1.trg_trg_att.v_layer.bias', 'decoder.layers.1.trg_trg_att.v_layer.weight', 'decoder.layers.1.x_layer_norm.bias', 'decoder.layers.1.x_layer_norm.weight', 'decoder.layers.2.dec_layer_norm.bias', 'decoder.layers.2.dec_layer_norm.weight', 'decoder.layers.2.feed_forward.layer_norm.bias', 'decoder.layers.2.feed_forward.layer_norm.weight', 'decoder.layers.2.feed_forward.pwff_layer.0.bias', 'decoder.layers.2.feed_forward.pwff_layer.0.weight', 'decoder.layers.2.feed_forward.pwff_layer.3.bias', 'decoder.layers.2.feed_forward.pwff_layer.3.weight', 'decoder.layers.2.src_trg_att.k_layer.bias', 'decoder.layers.2.src_trg_att.k_layer.weight', 'decoder.layers.2.src_trg_att.output_layer.bias', 'decoder.layers.2.src_trg_att.output_layer.weight', 'decoder.layers.2.src_trg_att.q_layer.bias', 'decoder.layers.2.src_trg_att.q_layer.weight', 'decoder.layers.2.src_trg_att.v_layer.bias', 'decoder.layers.2.src_trg_att.v_layer.weight', 'decoder.layers.2.trg_trg_att.k_layer.bias', 'decoder.layers.2.trg_trg_att.k_layer.weight', 'decoder.layers.2.trg_trg_att.output_layer.bias', 'decoder.layers.2.trg_trg_att.output_layer.weight', 'decoder.layers.2.trg_trg_att.q_layer.bias', 'decoder.layers.2.trg_trg_att.q_layer.weight', 'decoder.layers.2.trg_trg_att.v_layer.bias', 'decoder.layers.2.trg_trg_att.v_layer.weight', 'decoder.layers.2.x_layer_norm.bias', 'decoder.layers.2.x_layer_norm.weight', 'decoder.layers.3.dec_layer_norm.bias', 'decoder.layers.3.dec_layer_norm.weight', 'decoder.layers.3.feed_forward.layer_norm.bias', 'decoder.layers.3.feed_forward.layer_norm.weight', 'decoder.layers.3.feed_forward.pwff_layer.0.bias', 'decoder.layers.3.feed_forward.pwff_layer.0.weight', 'decoder.layers.3.feed_forward.pwff_layer.3.bias', 'decoder.layers.3.feed_forward.pwff_layer.3.weight', 'decoder.layers.3.src_trg_att.k_layer.bias', 'decoder.layers.3.src_trg_att.k_layer.weight', 'decoder.layers.3.src_trg_att.output_layer.bias', 'decoder.layers.3.src_trg_att.output_layer.weight', 'decoder.layers.3.src_trg_att.q_layer.bias', 'decoder.layers.3.src_trg_att.q_layer.weight', 'decoder.layers.3.src_trg_att.v_layer.bias', 'decoder.layers.3.src_trg_att.v_layer.weight', 'decoder.layers.3.trg_trg_att.k_layer.bias', 'decoder.layers.3.trg_trg_att.k_layer.weight', 'decoder.layers.3.trg_trg_att.output_layer.bias', 'decoder.layers.3.trg_trg_att.output_layer.weight', 'decoder.layers.3.trg_trg_att.q_layer.bias', 'decoder.layers.3.trg_trg_att.q_layer.weight', 'decoder.layers.3.trg_trg_att.v_layer.bias', 'decoder.layers.3.trg_trg_att.v_layer.weight', 'decoder.layers.3.x_layer_norm.bias', 'decoder.layers.3.x_layer_norm.weight', 'decoder.layers.4.dec_layer_norm.bias', 'decoder.layers.4.dec_layer_norm.weight', 'decoder.layers.4.feed_forward.layer_norm.bias', 'decoder.layers.4.feed_forward.layer_norm.weight', 'decoder.layers.4.feed_forward.pwff_layer.0.bias', 'decoder.layers.4.feed_forward.pwff_layer.0.weight', 'decoder.layers.4.feed_forward.pwff_layer.3.bias', 'decoder.layers.4.feed_forward.pwff_layer.3.weight', 'decoder.layers.4.src_trg_att.k_layer.bias', 'decoder.layers.4.src_trg_att.k_layer.weight', 'decoder.layers.4.src_trg_att.output_layer.bias', 'decoder.layers.4.src_trg_att.output_layer.weight', 'decoder.layers.4.src_trg_att.q_layer.bias', 'decoder.layers.4.src_trg_att.q_layer.weight', 'decoder.layers.4.src_trg_att.v_layer.bias', 'decoder.layers.4.src_trg_att.v_layer.weight', 'decoder.layers.4.trg_trg_att.k_layer.bias', 'decoder.layers.4.trg_trg_att.k_layer.weight', 'decoder.layers.4.trg_trg_att.output_layer.bias', 'decoder.layers.4.trg_trg_att.output_layer.weight', 'decoder.layers.4.trg_trg_att.q_layer.bias', 'decoder.layers.4.trg_trg_att.q_layer.weight', 'decoder.layers.4.trg_trg_att.v_layer.bias', 'decoder.layers.4.trg_trg_att.v_layer.weight', 'decoder.layers.4.x_layer_norm.bias', 'decoder.layers.4.x_layer_norm.weight', 'decoder.layers.5.dec_layer_norm.bias', 'decoder.layers.5.dec_layer_norm.weight', 'decoder.layers.5.feed_forward.layer_norm.bias', 'decoder.layers.5.feed_forward.layer_norm.weight', 'decoder.layers.5.feed_forward.pwff_layer.0.bias', 'decoder.layers.5.feed_forward.pwff_layer.0.weight', 'decoder.layers.5.feed_forward.pwff_layer.3.bias', 'decoder.layers.5.feed_forward.pwff_layer.3.weight', 'decoder.layers.5.src_trg_att.k_layer.bias', 'decoder.layers.5.src_trg_att.k_layer.weight', 'decoder.layers.5.src_trg_att.output_layer.bias', 'decoder.layers.5.src_trg_att.output_layer.weight', 'decoder.layers.5.src_trg_att.q_layer.bias', 'decoder.layers.5.src_trg_att.q_layer.weight', 'decoder.layers.5.src_trg_att.v_layer.bias', 'decoder.layers.5.src_trg_att.v_layer.weight', 'decoder.layers.5.trg_trg_att.k_layer.bias', 'decoder.layers.5.trg_trg_att.k_layer.weight', 'decoder.layers.5.trg_trg_att.output_layer.bias', 'decoder.layers.5.trg_trg_att.output_layer.weight', 'decoder.layers.5.trg_trg_att.q_layer.bias', 'decoder.layers.5.trg_trg_att.q_layer.weight', 'decoder.layers.5.trg_trg_att.v_layer.bias', 'decoder.layers.5.trg_trg_att.v_layer.weight', 'decoder.layers.5.x_layer_norm.bias', 'decoder.layers.5.x_layer_norm.weight', 'encoder.layer_norm.bias', 'encoder.layer_norm.weight', 'encoder.layers.0.feed_forward.layer_norm.bias', 'encoder.layers.0.feed_forward.layer_norm.weight', 'encoder.layers.0.feed_forward.pwff_layer.0.bias', 'encoder.layers.0.feed_forward.pwff_layer.0.weight', 'encoder.layers.0.feed_forward.pwff_layer.3.bias', 'encoder.layers.0.feed_forward.pwff_layer.3.weight', 'encoder.layers.0.layer_norm.bias', 'encoder.layers.0.layer_norm.weight', 'encoder.layers.0.src_src_att.k_layer.bias', 'encoder.layers.0.src_src_att.k_layer.weight', 'encoder.layers.0.src_src_att.output_layer.bias', 'encoder.layers.0.src_src_att.output_layer.weight', 'encoder.layers.0.src_src_att.q_layer.bias', 'encoder.layers.0.src_src_att.q_layer.weight', 'encoder.layers.0.src_src_att.v_layer.bias', 'encoder.layers.0.src_src_att.v_layer.weight', 'encoder.layers.1.feed_forward.layer_norm.bias', 'encoder.layers.1.feed_forward.layer_norm.weight', 'encoder.layers.1.feed_forward.pwff_layer.0.bias', 'encoder.layers.1.feed_forward.pwff_layer.0.weight', 'encoder.layers.1.feed_forward.pwff_layer.3.bias', 'encoder.layers.1.feed_forward.pwff_layer.3.weight', 'encoder.layers.1.layer_norm.bias', 'encoder.layers.1.layer_norm.weight', 'encoder.layers.1.src_src_att.k_layer.bias', 'encoder.layers.1.src_src_att.k_layer.weight', 'encoder.layers.1.src_src_att.output_layer.bias', 'encoder.layers.1.src_src_att.output_layer.weight', 'encoder.layers.1.src_src_att.q_layer.bias', 'encoder.layers.1.src_src_att.q_layer.weight', 'encoder.layers.1.src_src_att.v_layer.bias', 'encoder.layers.1.src_src_att.v_layer.weight', 'encoder.layers.2.feed_forward.layer_norm.bias', 'encoder.layers.2.feed_forward.layer_norm.weight', 'encoder.layers.2.feed_forward.pwff_layer.0.bias', 'encoder.layers.2.feed_forward.pwff_layer.0.weight', 'encoder.layers.2.feed_forward.pwff_layer.3.bias', 'encoder.layers.2.feed_forward.pwff_layer.3.weight', 'encoder.layers.2.layer_norm.bias', 'encoder.layers.2.layer_norm.weight', 'encoder.layers.2.src_src_att.k_layer.bias', 'encoder.layers.2.src_src_att.k_layer.weight', 'encoder.layers.2.src_src_att.output_layer.bias', 'encoder.layers.2.src_src_att.output_layer.weight', 'encoder.layers.2.src_src_att.q_layer.bias', 'encoder.layers.2.src_src_att.q_layer.weight', 'encoder.layers.2.src_src_att.v_layer.bias', 'encoder.layers.2.src_src_att.v_layer.weight', 'encoder.layers.3.feed_forward.layer_norm.bias', 'encoder.layers.3.feed_forward.layer_norm.weight', 'encoder.layers.3.feed_forward.pwff_layer.0.bias', 'encoder.layers.3.feed_forward.pwff_layer.0.weight', 'encoder.layers.3.feed_forward.pwff_layer.3.bias', 'encoder.layers.3.feed_forward.pwff_layer.3.weight', 'encoder.layers.3.layer_norm.bias', 'encoder.layers.3.layer_norm.weight', 'encoder.layers.3.src_src_att.k_layer.bias', 'encoder.layers.3.src_src_att.k_layer.weight', 'encoder.layers.3.src_src_att.output_layer.bias', 'encoder.layers.3.src_src_att.output_layer.weight', 'encoder.layers.3.src_src_att.q_layer.bias', 'encoder.layers.3.src_src_att.q_layer.weight', 'encoder.layers.3.src_src_att.v_layer.bias', 'encoder.layers.3.src_src_att.v_layer.weight', 'encoder.layers.4.feed_forward.layer_norm.bias', 'encoder.layers.4.feed_forward.layer_norm.weight', 'encoder.layers.4.feed_forward.pwff_layer.0.bias', 'encoder.layers.4.feed_forward.pwff_layer.0.weight', 'encoder.layers.4.feed_forward.pwff_layer.3.bias', 'encoder.layers.4.feed_forward.pwff_layer.3.weight', 'encoder.layers.4.layer_norm.bias', 'encoder.layers.4.layer_norm.weight', 'encoder.layers.4.src_src_att.k_layer.bias', 'encoder.layers.4.src_src_att.k_layer.weight', 'encoder.layers.4.src_src_att.output_layer.bias', 'encoder.layers.4.src_src_att.output_layer.weight', 'encoder.layers.4.src_src_att.q_layer.bias', 'encoder.layers.4.src_src_att.q_layer.weight', 'encoder.layers.4.src_src_att.v_layer.bias', 'encoder.layers.4.src_src_att.v_layer.weight', 'encoder.layers.5.feed_forward.layer_norm.bias', 'encoder.layers.5.feed_forward.layer_norm.weight', 'encoder.layers.5.feed_forward.pwff_layer.0.bias', 'encoder.layers.5.feed_forward.pwff_layer.0.weight', 'encoder.layers.5.feed_forward.pwff_layer.3.bias', 'encoder.layers.5.feed_forward.pwff_layer.3.weight', 'encoder.layers.5.layer_norm.bias', 'encoder.layers.5.layer_norm.weight', 'encoder.layers.5.src_src_att.k_layer.bias', 'encoder.layers.5.src_src_att.k_layer.weight', 'encoder.layers.5.src_src_att.output_layer.bias', 'encoder.layers.5.src_src_att.output_layer.weight', 'encoder.layers.5.src_src_att.q_layer.bias', 'encoder.layers.5.src_src_att.q_layer.weight', 'encoder.layers.5.src_src_att.v_layer.bias', 'encoder.layers.5.src_src_att.v_layer.weight', 'src_embed.lut.weight']\n",
      "2020-02-18 09:39:00,320 Loading model from models/enkam_transformer/1000.ckpt\n",
      "2020-02-18 09:39:00,819 cfg.name                           : enkam_transformer\n",
      "2020-02-18 09:39:00,819 cfg.data.src                       : en\n",
      "2020-02-18 09:39:00,819 cfg.data.trg                       : kam\n",
      "2020-02-18 09:39:00,819 cfg.data.train                     : data/enkam/train.bpe\n",
      "2020-02-18 09:39:00,819 cfg.data.dev                       : data/enkam/dev.bpe\n",
      "2020-02-18 09:39:00,819 cfg.data.test                      : data/enkam/test.bpe\n",
      "2020-02-18 09:39:00,819 cfg.data.level                     : bpe\n",
      "2020-02-18 09:39:00,819 cfg.data.lowercase                 : False\n",
      "2020-02-18 09:39:00,819 cfg.data.max_sent_length           : 100\n",
      "2020-02-18 09:39:00,820 cfg.data.src_vocab                 : data/enkam/vocab.txt\n",
      "2020-02-18 09:39:00,820 cfg.data.trg_vocab                 : data/enkam/vocab.txt\n",
      "2020-02-18 09:39:00,820 cfg.testing.beam_size              : 5\n",
      "2020-02-18 09:39:00,820 cfg.testing.alpha                  : 1.0\n",
      "2020-02-18 09:39:00,820 cfg.training.load_model            : models/enkam_transformer/1000.ckpt\n",
      "2020-02-18 09:39:00,820 cfg.training.random_seed           : 42\n",
      "2020-02-18 09:39:00,820 cfg.training.optimizer             : adam\n",
      "2020-02-18 09:39:00,820 cfg.training.normalization         : tokens\n",
      "2020-02-18 09:39:00,820 cfg.training.adam_betas            : [0.9, 0.999]\n",
      "2020-02-18 09:39:00,820 cfg.training.scheduling            : plateau\n",
      "2020-02-18 09:39:00,820 cfg.training.patience              : 5\n",
      "2020-02-18 09:39:00,820 cfg.training.learning_rate_factor  : 0.5\n",
      "2020-02-18 09:39:00,820 cfg.training.learning_rate_warmup  : 1000\n",
      "2020-02-18 09:39:00,821 cfg.training.decrease_factor       : 0.7\n",
      "2020-02-18 09:39:00,821 cfg.training.loss                  : crossentropy\n",
      "2020-02-18 09:39:00,821 cfg.training.learning_rate         : 0.0003\n",
      "2020-02-18 09:39:00,821 cfg.training.learning_rate_min     : 1e-08\n",
      "2020-02-18 09:39:00,821 cfg.training.weight_decay          : 0.0\n",
      "2020-02-18 09:39:00,821 cfg.training.label_smoothing       : 0.1\n",
      "2020-02-18 09:39:00,821 cfg.training.batch_size            : 4096\n",
      "2020-02-18 09:39:00,821 cfg.training.batch_type            : token\n",
      "2020-02-18 09:39:00,821 cfg.training.eval_batch_size       : 3600\n",
      "2020-02-18 09:39:00,821 cfg.training.eval_batch_type       : token\n",
      "2020-02-18 09:39:00,821 cfg.training.batch_multiplier      : 1\n",
      "2020-02-18 09:39:00,821 cfg.training.early_stopping_metric : ppl\n",
      "2020-02-18 09:39:00,821 cfg.training.epochs                : 28\n",
      "2020-02-18 09:39:00,821 cfg.training.validation_freq       : 1000\n",
      "2020-02-18 09:39:00,821 cfg.training.logging_freq          : 100\n",
      "2020-02-18 09:39:00,822 cfg.training.eval_metric           : bleu\n",
      "2020-02-18 09:39:00,822 cfg.training.model_dir             : models/enkam_transformer2\n",
      "2020-02-18 09:39:00,822 cfg.training.overwrite             : False\n",
      "2020-02-18 09:39:00,822 cfg.training.shuffle               : True\n",
      "2020-02-18 09:39:00,822 cfg.training.use_cuda              : True\n",
      "2020-02-18 09:39:00,822 cfg.training.max_output_length     : 100\n",
      "2020-02-18 09:39:00,822 cfg.training.print_valid_sents     : [0, 1, 2, 3]\n",
      "2020-02-18 09:39:00,822 cfg.training.keep_last_ckpts       : 3\n",
      "2020-02-18 09:39:00,822 cfg.model.initializer              : xavier\n",
      "2020-02-18 09:39:00,822 cfg.model.bias_initializer         : zeros\n",
      "2020-02-18 09:39:00,822 cfg.model.init_gain                : 1.0\n",
      "2020-02-18 09:39:00,822 cfg.model.embed_initializer        : xavier\n",
      "2020-02-18 09:39:00,822 cfg.model.embed_init_gain          : 1.0\n",
      "2020-02-18 09:39:00,822 cfg.model.tied_embeddings          : True\n",
      "2020-02-18 09:39:00,823 cfg.model.tied_softmax             : True\n",
      "2020-02-18 09:39:00,823 cfg.model.encoder.type             : transformer\n",
      "2020-02-18 09:39:00,823 cfg.model.encoder.num_layers       : 6\n",
      "2020-02-18 09:39:00,823 cfg.model.encoder.num_heads        : 4\n",
      "2020-02-18 09:39:00,823 cfg.model.encoder.embeddings.embedding_dim : 256\n",
      "2020-02-18 09:39:00,823 cfg.model.encoder.embeddings.scale : True\n",
      "2020-02-18 09:39:00,823 cfg.model.encoder.embeddings.dropout : 0.2\n",
      "2020-02-18 09:39:00,823 cfg.model.encoder.hidden_size      : 256\n",
      "2020-02-18 09:39:00,823 cfg.model.encoder.ff_size          : 1024\n",
      "2020-02-18 09:39:00,823 cfg.model.encoder.dropout          : 0.3\n",
      "2020-02-18 09:39:00,823 cfg.model.decoder.type             : transformer\n",
      "2020-02-18 09:39:00,823 cfg.model.decoder.num_layers       : 6\n",
      "2020-02-18 09:39:00,823 cfg.model.decoder.num_heads        : 4\n",
      "2020-02-18 09:39:00,823 cfg.model.decoder.embeddings.embedding_dim : 256\n",
      "2020-02-18 09:39:00,824 cfg.model.decoder.embeddings.scale : True\n",
      "2020-02-18 09:39:00,824 cfg.model.decoder.embeddings.dropout : 0.2\n",
      "2020-02-18 09:39:00,824 cfg.model.decoder.hidden_size      : 256\n",
      "2020-02-18 09:39:00,824 cfg.model.decoder.ff_size          : 1024\n",
      "2020-02-18 09:39:00,824 cfg.model.decoder.dropout          : 0.3\n",
      "2020-02-18 09:39:00,824 Data set sizes: \n",
      "\ttrain 48856,\n",
      "\tvalid 1000,\n",
      "\ttest 2588\n",
      "2020-02-18 09:39:00,824 First training example:\n",
      "\t[SRC] What can cause us to ask : “ How long ” ?\n",
      "\t[TRG] K@@ yo nĩ kyaũ kĩtonya kũtuma twĩ@@ kũlya : “ Ng@@ a@@ ĩa ĩvinda yĩ@@ ana ata ? ”\n",
      "2020-02-18 09:39:00,824 First 10 words (src): (0) <unk> (1) <pad> (2) <s> (3) </s> (4) . (5) , (6) na (7) the (8) to (9) wa\n",
      "2020-02-18 09:39:00,824 First 10 words (trg): (0) <unk> (1) <pad> (2) <s> (3) </s> (4) . (5) , (6) na (7) the (8) to (9) wa\n",
      "2020-02-18 09:39:00,824 Number of Src words (types): 4190\n",
      "2020-02-18 09:39:00,824 Number of Trg words (types): 4190\n",
      "2020-02-18 09:39:00,824 Model(\n",
      "\tencoder=TransformerEncoder(num_layers=6, num_heads=4),\n",
      "\tdecoder=TransformerDecoder(num_layers=6, num_heads=4),\n",
      "\tsrc_embed=Embeddings(embedding_dim=256, vocab_size=4190),\n",
      "\ttrg_embed=Embeddings(embedding_dim=256, vocab_size=4190))\n",
      "2020-02-18 09:39:00,829 EPOCH 1\n",
      "2020-02-18 09:39:53,213 Epoch   1 Step:     1100 Batch Loss:     3.554983 Tokens per Sec:     4025, Lr: 0.000300\n",
      "2020-02-18 09:40:45,650 Epoch   1 Step:     1200 Batch Loss:     3.999159 Tokens per Sec:     3976, Lr: 0.000300\n",
      "2020-02-18 09:41:38,978 Epoch   1 Step:     1300 Batch Loss:     3.724230 Tokens per Sec:     4111, Lr: 0.000300\n",
      "2020-02-18 09:42:31,344 Epoch   1 Step:     1400 Batch Loss:     3.929605 Tokens per Sec:     3975, Lr: 0.000300\n",
      "2020-02-18 09:43:25,552 Epoch   1 Step:     1500 Batch Loss:     3.141755 Tokens per Sec:     3913, Lr: 0.000300\n",
      "2020-02-18 09:44:19,747 Epoch   1 Step:     1600 Batch Loss:     3.874290 Tokens per Sec:     3884, Lr: 0.000300\n",
      "2020-02-18 09:44:20,340 Epoch   1: total training loss 2320.43\n",
      "2020-02-18 09:44:20,341 EPOCH 2\n",
      "2020-02-18 09:45:13,789 Epoch   2 Step:     1700 Batch Loss:     3.070514 Tokens per Sec:     3883, Lr: 0.000300\n",
      "2020-02-18 09:46:05,950 Epoch   2 Step:     1800 Batch Loss:     3.873306 Tokens per Sec:     3979, Lr: 0.000300\n",
      "2020-02-18 09:46:59,195 Epoch   2 Step:     1900 Batch Loss:     3.651958 Tokens per Sec:     4108, Lr: 0.000300\n",
      "2020-02-18 09:47:51,651 Epoch   2 Step:     2000 Batch Loss:     3.715076 Tokens per Sec:     3997, Lr: 0.000300\n",
      "2020-02-18 09:50:45,551 Hooray! New best validation result [ppl]!\n",
      "2020-02-18 09:50:45,551 Saving new checkpoint.\n",
      "2020-02-18 09:50:45,761 Example #0\n",
      "2020-02-18 09:50:45,761 \tSource:     Gideon wondered how it would be possible for him to “ save Israel out of Midian’s hand . ”\n",
      "2020-02-18 09:50:45,761 \tReference:  Mũlaĩka ũsu nĩwaneenie vandũ va Mũmbi na amũĩkĩĩthya Ngiteoni kana Yeova aĩ vamwe nake .\n",
      "2020-02-18 09:50:45,761 \tHypothesis: ( Meko 4 : 1 - 4 ) O na kau no nginya nginya nginya nginya nginya nginya nginya nginya nginya nginya nginya nginya nginya nginya nginya nginya nginya nginya nginya nginya nginya nginya nginya nginya nginya nginya nginya nginya nginya nginya nginya nginya nginya nginya nginya nginya nginya nginya nginya nginya nginya nginya nginya ethĩwe na mũĩkĩĩo . ”\n",
      "2020-02-18 09:50:45,761 Example #1\n",
      "2020-02-18 09:50:45,761 \tSource:     And then she would try to return to Hosea .\n",
      "2020-02-18 09:50:45,762 \tReference:  Na ĩndĩ kyamina kũsembany’a na endwa makyo Yeova aĩtye kĩkatata kũmũsyokea Osea .\n",
      "2020-02-18 09:50:45,762 \tHypothesis: Nĩwaendie kũlika na atheu .\n",
      "2020-02-18 09:50:45,762 Example #2\n",
      "2020-02-18 09:50:45,762 \tSource:     The Bible itself says : “ The green grass dries up , the blossom withers , but the word of our God endures forever . ” ​ — Isaiah 40 : 8 .\n",
      "2020-02-18 09:50:45,762 \tReference:  Mbivilia yaĩtye atĩĩ : “ Nyeki nĩyũmaa , na ĩlaa nĩyĩvovaa ; ĩndĩ ndeto ya Ngai waitũ ĩkekala tene na tene . ” — Isaia 40 : 8 .\n",
      "2020-02-18 09:50:45,762 \tHypothesis: Mbivilia yaĩtye atĩĩ : “ Ũtianĩo wa Ngai , na ũla wĩ na mũĩkĩĩo , na ũla wĩ na mũĩkĩĩo , na Ngai , na ũla wĩ na mũĩkĩĩo , na ũla wĩ na mũĩkĩĩo wake . ” — Soma Savuli 3 : 3 , 11 .\n",
      "2020-02-18 09:50:45,762 Example #3\n",
      "2020-02-18 09:50:45,762 \tSource:     It includes links to lists of Witnesses currently imprisoned for their faith .\n",
      "2020-02-18 09:50:45,762 \tReference:  7 : 12 ) Walika Kĩsesenĩ kya jw.org no wone masyĩtwa ma Ngũsĩ ila syovetwe .\n",
      "2020-02-18 09:50:45,762 \tHypothesis: O na kau andũ aingĩ ma mũsyĩ ala mawetetwe Mbivilianĩ mawetetwe Mbivilianĩ .\n",
      "2020-02-18 09:50:45,762 Validation result at epoch   2, step     2000: bleu:   2.32, loss: 87591.9062, ppl:  30.4744, duration: 174.1112s\n",
      "2020-02-18 09:51:38,340 Epoch   2 Step:     2100 Batch Loss:     3.641190 Tokens per Sec:     4024, Lr: 0.000300\n",
      "2020-02-18 09:52:30,668 Epoch   2 Step:     2200 Batch Loss:     3.641436 Tokens per Sec:     3968, Lr: 0.000300\n",
      "2020-02-18 09:52:32,812 Epoch   2: total training loss 2125.80\n",
      "2020-02-18 09:52:32,813 EPOCH 3\n",
      "2020-02-18 09:53:23,444 Epoch   3 Step:     2300 Batch Loss:     3.743196 Tokens per Sec:     4017, Lr: 0.000300\n",
      "2020-02-18 09:54:15,486 Epoch   3 Step:     2400 Batch Loss:     3.796329 Tokens per Sec:     4045, Lr: 0.000300\n",
      "2020-02-18 09:55:09,233 Epoch   3 Step:     2500 Batch Loss:     3.669096 Tokens per Sec:     3912, Lr: 0.000300\n",
      "2020-02-18 09:56:03,284 Epoch   3 Step:     2600 Batch Loss:     3.059393 Tokens per Sec:     3876, Lr: 0.000300\n",
      "2020-02-18 09:56:58,173 Epoch   3 Step:     2700 Batch Loss:     3.497191 Tokens per Sec:     3963, Lr: 0.000300\n",
      "2020-02-18 09:57:50,729 Epoch   3 Step:     2800 Batch Loss:     3.370025 Tokens per Sec:     3970, Lr: 0.000300\n",
      "2020-02-18 09:57:53,845 Epoch   3: total training loss 1992.60\n",
      "2020-02-18 09:57:53,845 EPOCH 4\n",
      "2020-02-18 09:58:42,943 Epoch   4 Step:     2900 Batch Loss:     3.394607 Tokens per Sec:     3985, Lr: 0.000300\n",
      "2020-02-18 09:59:35,960 Epoch   4 Step:     3000 Batch Loss:     2.585446 Tokens per Sec:     4072, Lr: 0.000300\n",
      "2020-02-18 10:02:29,883 Hooray! New best validation result [ppl]!\n",
      "2020-02-18 10:02:29,884 Saving new checkpoint.\n",
      "2020-02-18 10:02:30,087 Example #0\n",
      "2020-02-18 10:02:30,088 \tSource:     Gideon wondered how it would be possible for him to “ save Israel out of Midian’s hand . ”\n",
      "2020-02-18 10:02:30,088 \tReference:  Mũlaĩka ũsu nĩwaneenie vandũ va Mũmbi na amũĩkĩĩthya Ngiteoni kana Yeova aĩ vamwe nake .\n",
      "2020-02-18 10:02:30,088 \tHypothesis: Kũvindĩĩsya ũndũ ũsu wawetie kana mũndũ mũka ũsu nĩwamũtetheeisye “ aeni . ”\n",
      "2020-02-18 10:02:30,088 Example #1\n",
      "2020-02-18 10:02:30,088 \tSource:     And then she would try to return to Hosea .\n",
      "2020-02-18 10:02:30,088 \tReference:  Na ĩndĩ kyamina kũsembany’a na endwa makyo Yeova aĩtye kĩkatata kũmũsyokea Osea .\n",
      "2020-02-18 10:02:30,088 \tHypothesis: Ĩndĩ o na kau nĩwamũtetheeisye .\n",
      "2020-02-18 10:02:30,088 Example #2\n",
      "2020-02-18 10:02:30,089 \tSource:     The Bible itself says : “ The green grass dries up , the blossom withers , but the word of our God endures forever . ” ​ — Isaiah 40 : 8 .\n",
      "2020-02-18 10:02:30,089 \tReference:  Mbivilia yaĩtye atĩĩ : “ Nyeki nĩyũmaa , na ĩlaa nĩyĩvovaa ; ĩndĩ ndeto ya Ngai waitũ ĩkekala tene na tene . ” — Isaia 40 : 8 .\n",
      "2020-02-18 10:02:30,089 \tHypothesis: Mbivilia yaĩtye atĩĩ : “ Ĩĩ , ĩndĩ ti ũndũ wa vata mũno , ĩndĩ ti ũndũ wa kũseũvya Ngai , ĩndĩ ti ũndũ wa vata mũno . ” — Savuli 119 : 8 .\n",
      "2020-02-18 10:02:30,089 Example #3\n",
      "2020-02-18 10:02:30,089 \tSource:     It includes links to lists of Witnesses currently imprisoned for their faith .\n",
      "2020-02-18 10:02:30,089 \tReference:  7 : 12 ) Walika Kĩsesenĩ kya jw.org no wone masyĩtwa ma Ngũsĩ ila syovetwe .\n",
      "2020-02-18 10:02:30,089 \tHypothesis: Andũ aingĩ ma mũsyĩ nĩmambĩĩe kũalyũla Mbivilia na mayaĩ mawoni ma Yeova .\n",
      "2020-02-18 10:02:30,089 Validation result at epoch   4, step     3000: bleu:   4.00, loss: 79091.3984, ppl:  21.8738, duration: 174.1291s\n",
      "2020-02-18 10:03:22,385 Epoch   4 Step:     3100 Batch Loss:     3.145908 Tokens per Sec:     4000, Lr: 0.000300\n",
      "2020-02-18 10:04:15,226 Epoch   4 Step:     3200 Batch Loss:     2.867697 Tokens per Sec:     3995, Lr: 0.000300\n",
      "2020-02-18 10:05:07,641 Epoch   4 Step:     3300 Batch Loss:     3.066645 Tokens per Sec:     4039, Lr: 0.000300\n",
      "2020-02-18 10:05:59,200 Epoch   4 Step:     3400 Batch Loss:     3.253880 Tokens per Sec:     4047, Lr: 0.000300\n",
      "2020-02-18 10:06:04,288 Epoch   4: total training loss 1896.54\n",
      "2020-02-18 10:06:04,288 EPOCH 5\n",
      "2020-02-18 10:06:52,674 Epoch   5 Step:     3500 Batch Loss:     3.337849 Tokens per Sec:     3942, Lr: 0.000300\n",
      "2020-02-18 10:07:46,162 Epoch   5 Step:     3600 Batch Loss:     3.141412 Tokens per Sec:     3878, Lr: 0.000300\n",
      "2020-02-18 10:08:40,635 Epoch   5 Step:     3700 Batch Loss:     3.043408 Tokens per Sec:     3949, Lr: 0.000300\n",
      "2020-02-18 10:09:34,360 Epoch   5 Step:     3800 Batch Loss:     3.153746 Tokens per Sec:     3925, Lr: 0.000300\n",
      "2020-02-18 10:10:26,588 Epoch   5 Step:     3900 Batch Loss:     3.129083 Tokens per Sec:     4122, Lr: 0.000300\n",
      "2020-02-18 10:11:18,386 Epoch   5 Step:     4000 Batch Loss:     2.835610 Tokens per Sec:     3948, Lr: 0.000300\n",
      "2020-02-18 10:14:12,195 Hooray! New best validation result [ppl]!\n",
      "2020-02-18 10:14:12,195 Saving new checkpoint.\n",
      "2020-02-18 10:14:12,402 Example #0\n",
      "2020-02-18 10:14:12,403 \tSource:     Gideon wondered how it would be possible for him to “ save Israel out of Midian’s hand . ”\n",
      "2020-02-18 10:14:12,403 \tReference:  Mũlaĩka ũsu nĩwaneenie vandũ va Mũmbi na amũĩkĩĩthya Ngiteoni kana Yeova aĩ vamwe nake .\n",
      "2020-02-18 10:14:12,403 \tHypothesis: ( Luka 5 : 1 - 5 ) Nĩ ũndũ wĩva ũtonya kũkũlya kana ‘ akw’ũ ’ na athembi .\n",
      "2020-02-18 10:14:12,403 Example #1\n",
      "2020-02-18 10:14:12,403 \tSource:     And then she would try to return to Hosea .\n",
      "2020-02-18 10:14:12,403 \tReference:  Na ĩndĩ kyamina kũsembany’a na endwa makyo Yeova aĩtye kĩkatata kũmũsyokea Osea .\n",
      "2020-02-18 10:14:12,403 \tHypothesis: Na ĩndĩ ndaema kũmũvoya .\n",
      "2020-02-18 10:14:12,403 Example #2\n",
      "2020-02-18 10:14:12,403 \tSource:     The Bible itself says : “ The green grass dries up , the blossom withers , but the word of our God endures forever . ” ​ — Isaiah 40 : 8 .\n",
      "2020-02-18 10:14:12,404 \tReference:  Mbivilia yaĩtye atĩĩ : “ Nyeki nĩyũmaa , na ĩlaa nĩyĩvovaa ; ĩndĩ ndeto ya Ngai waitũ ĩkekala tene na tene . ” — Isaia 40 : 8 .\n",
      "2020-02-18 10:14:12,404 \tHypothesis: Mbivilia yaĩtye atĩĩ : “ O na kau Mbivilia yaĩtye ũũ : “ Ngai ndakaathima , ĩndĩ ti andũ onthe , ĩndĩ ti ma Ngai . ” — Isaia 32 : 8 .\n",
      "2020-02-18 10:14:12,404 Example #3\n",
      "2020-02-18 10:14:12,404 \tSource:     It includes links to lists of Witnesses currently imprisoned for their faith .\n",
      "2020-02-18 10:14:12,404 \tReference:  7 : 12 ) Walika Kĩsesenĩ kya jw.org no wone masyĩtwa ma Ngũsĩ ila syovetwe .\n",
      "2020-02-18 10:14:12,404 \tHypothesis: Andũ aingĩ ma mũika nĩmakwatasya kana Ngũsĩ sya Yeova syatetheeisye mũno .\n",
      "2020-02-18 10:14:12,404 Validation result at epoch   5, step     4000: bleu:   5.92, loss: 74410.0625, ppl:  18.2228, duration: 174.0179s\n",
      "2020-02-18 10:14:19,551 Epoch   5: total training loss 1816.19\n",
      "2020-02-18 10:14:19,552 EPOCH 6\n",
      "2020-02-18 10:15:04,717 Epoch   6 Step:     4100 Batch Loss:     2.951361 Tokens per Sec:     3971, Lr: 0.000300\n",
      "2020-02-18 10:15:57,173 Epoch   6 Step:     4200 Batch Loss:     2.986690 Tokens per Sec:     4032, Lr: 0.000300\n",
      "2020-02-18 10:16:49,646 Epoch   6 Step:     4300 Batch Loss:     2.920062 Tokens per Sec:     3964, Lr: 0.000300\n",
      "2020-02-18 10:17:42,114 Epoch   6 Step:     4400 Batch Loss:     2.641401 Tokens per Sec:     4063, Lr: 0.000300\n",
      "2020-02-18 10:18:34,591 Epoch   6 Step:     4500 Batch Loss:     3.079638 Tokens per Sec:     4008, Lr: 0.000300\n",
      "2020-02-18 10:19:29,412 Epoch   6 Step:     4600 Batch Loss:     2.869104 Tokens per Sec:     3916, Lr: 0.000300\n",
      "2020-02-18 10:19:38,918 Epoch   6: total training loss 1755.78\n",
      "2020-02-18 10:19:38,918 EPOCH 7\n",
      "2020-02-18 10:20:23,125 Epoch   7 Step:     4700 Batch Loss:     3.424023 Tokens per Sec:     3896, Lr: 0.000300\n",
      "2020-02-18 10:21:17,427 Epoch   7 Step:     4800 Batch Loss:     2.631806 Tokens per Sec:     3910, Lr: 0.000300\n",
      "2020-02-18 10:22:09,376 Epoch   7 Step:     4900 Batch Loss:     3.188667 Tokens per Sec:     4029, Lr: 0.000300\n",
      "2020-02-18 10:23:01,891 Epoch   7 Step:     5000 Batch Loss:     3.217089 Tokens per Sec:     4038, Lr: 0.000300\n",
      "2020-02-18 10:25:55,627 Hooray! New best validation result [ppl]!\n",
      "2020-02-18 10:25:55,627 Saving new checkpoint.\n",
      "2020-02-18 10:25:55,851 Example #0\n",
      "2020-02-18 10:25:55,852 \tSource:     Gideon wondered how it would be possible for him to “ save Israel out of Midian’s hand . ”\n",
      "2020-02-18 10:25:55,852 \tReference:  Mũlaĩka ũsu nĩwaneenie vandũ va Mũmbi na amũĩkĩĩthya Ngiteoni kana Yeova aĩ vamwe nake .\n",
      "2020-02-18 10:25:55,852 \tHypothesis: Kũsũanĩa ũndũ ũsu no nginya wambĩĩe kũweta ĩũlũ wa nthĩ ya Kanaani .\n",
      "2020-02-18 10:25:55,852 Example #1\n",
      "2020-02-18 10:25:55,852 \tSource:     And then she would try to return to Hosea .\n",
      "2020-02-18 10:25:55,852 \tReference:  Na ĩndĩ kyamina kũsembany’a na endwa makyo Yeova aĩtye kĩkatata kũmũsyokea Osea .\n",
      "2020-02-18 10:25:55,852 \tHypothesis: Na ĩndĩ ambĩĩa kũmũvoya Yeova .\n",
      "2020-02-18 10:25:55,852 Example #2\n",
      "2020-02-18 10:25:55,852 \tSource:     The Bible itself says : “ The green grass dries up , the blossom withers , but the word of our God endures forever . ” ​ — Isaiah 40 : 8 .\n",
      "2020-02-18 10:25:55,853 \tReference:  Mbivilia yaĩtye atĩĩ : “ Nyeki nĩyũmaa , na ĩlaa nĩyĩvovaa ; ĩndĩ ndeto ya Ngai waitũ ĩkekala tene na tene . ” — Isaia 40 : 8 .\n",
      "2020-02-18 10:25:55,853 \tHypothesis: Mbivilia yaĩtye atĩĩ : “ Mũikemawa nĩ Ngai , ĩndĩ ti ũndũ wa vata mũno , ĩndĩ ti kwasya kana Ngai e na naĩ , ĩndĩ ti o na wĩva . ” — Isaia 40 : 8 .\n",
      "2020-02-18 10:25:55,853 Example #3\n",
      "2020-02-18 10:25:55,853 \tSource:     It includes links to lists of Witnesses currently imprisoned for their faith .\n",
      "2020-02-18 10:25:55,853 \tReference:  7 : 12 ) Walika Kĩsesenĩ kya jw.org no wone masyĩtwa ma Ngũsĩ ila syovetwe .\n",
      "2020-02-18 10:25:55,853 \tHypothesis: Andũ aingĩ ma nthĩ mayaĩ na nzika kana Ngũsĩ sya Yeova nĩsyoonie mũĩkĩĩo woo .\n",
      "2020-02-18 10:25:55,853 Validation result at epoch   7, step     5000: bleu:   7.58, loss: 70467.1953, ppl:  15.6249, duration: 173.9610s\n",
      "2020-02-18 10:26:48,355 Epoch   7 Step:     5100 Batch Loss:     2.118326 Tokens per Sec:     4033, Lr: 0.000300\n",
      "2020-02-18 10:27:40,455 Epoch   7 Step:     5200 Batch Loss:     2.921861 Tokens per Sec:     3959, Lr: 0.000300\n",
      "2020-02-18 10:27:52,922 Epoch   7: total training loss 1707.03\n",
      "2020-02-18 10:27:52,922 EPOCH 8\n",
      "2020-02-18 10:28:33,097 Epoch   8 Step:     5300 Batch Loss:     2.861975 Tokens per Sec:     4111, Lr: 0.000300\n",
      "2020-02-18 10:29:25,336 Epoch   8 Step:     5400 Batch Loss:     2.623770 Tokens per Sec:     4028, Lr: 0.000300\n",
      "2020-02-18 10:30:17,422 Epoch   8 Step:     5500 Batch Loss:     2.083694 Tokens per Sec:     3999, Lr: 0.000300\n",
      "2020-02-18 10:31:11,536 Epoch   8 Step:     5600 Batch Loss:     2.747229 Tokens per Sec:     3952, Lr: 0.000300\n",
      "2020-02-18 10:32:05,390 Epoch   8 Step:     5700 Batch Loss:     2.604842 Tokens per Sec:     3880, Lr: 0.000300\n",
      "2020-02-18 10:32:59,251 Epoch   8 Step:     5800 Batch Loss:     2.628845 Tokens per Sec:     3852, Lr: 0.000300\n",
      "2020-02-18 10:33:14,038 Epoch   8: total training loss 1644.40\n",
      "2020-02-18 10:33:14,038 EPOCH 9\n",
      "2020-02-18 10:33:52,259 Epoch   9 Step:     5900 Batch Loss:     2.690629 Tokens per Sec:     4038, Lr: 0.000300\n",
      "2020-02-18 10:34:45,016 Epoch   9 Step:     6000 Batch Loss:     2.631066 Tokens per Sec:     4057, Lr: 0.000300\n",
      "2020-02-18 10:37:39,145 Hooray! New best validation result [ppl]!\n",
      "2020-02-18 10:37:39,145 Saving new checkpoint.\n",
      "2020-02-18 10:37:39,372 Example #0\n",
      "2020-02-18 10:37:39,373 \tSource:     Gideon wondered how it would be possible for him to “ save Israel out of Midian’s hand . ”\n",
      "2020-02-18 10:37:39,373 \tReference:  Mũlaĩka ũsu nĩwaneenie vandũ va Mũmbi na amũĩkĩĩthya Ngiteoni kana Yeova aĩ vamwe nake .\n",
      "2020-02-18 10:37:39,373 \tHypothesis: Kũsũanĩa ũndũ ũsu no kwĩthĩwa wamwĩtye Yeova ‘ amũtavya mbaĩ ya Isilaeli . ’\n",
      "2020-02-18 10:37:39,373 Example #1\n",
      "2020-02-18 10:37:39,373 \tSource:     And then she would try to return to Hosea .\n",
      "2020-02-18 10:37:39,374 \tReference:  Na ĩndĩ kyamina kũsembany’a na endwa makyo Yeova aĩtye kĩkatata kũmũsyokea Osea .\n",
      "2020-02-18 10:37:39,374 \tHypothesis: Na ĩndĩ atindĩaa kũmũvoya Yeova amũtavya .\n",
      "2020-02-18 10:37:39,374 Example #2\n",
      "2020-02-18 10:37:39,374 \tSource:     The Bible itself says : “ The green grass dries up , the blossom withers , but the word of our God endures forever . ” ​ — Isaiah 40 : 8 .\n",
      "2020-02-18 10:37:39,374 \tReference:  Mbivilia yaĩtye atĩĩ : “ Nyeki nĩyũmaa , na ĩlaa nĩyĩvovaa ; ĩndĩ ndeto ya Ngai waitũ ĩkekala tene na tene . ” — Isaia 40 : 8 .\n",
      "2020-02-18 10:37:39,374 \tHypothesis: Mbivilia yaĩtye atĩĩ : “ Ĩsũvĩei , ĩndĩ ndeto sya Ngai , ĩndĩ nĩ ndeto sya andũ , ĩndĩ ti sya Ngai ; ĩndĩ ti ũndũ ũtangaa . ” — Isaia 40 : 8 .\n",
      "2020-02-18 10:37:39,374 Example #3\n",
      "2020-02-18 10:37:39,374 \tSource:     It includes links to lists of Witnesses currently imprisoned for their faith .\n",
      "2020-02-18 10:37:39,374 \tReference:  7 : 12 ) Walika Kĩsesenĩ kya jw.org no wone masyĩtwa ma Ngũsĩ ila syovetwe .\n",
      "2020-02-18 10:37:39,374 \tHypothesis: Andũ aingĩ ma ndĩni ila syoombiwe nĩ Ngũsĩ sya Yeova ila syoombiwe .\n",
      "2020-02-18 10:37:39,375 Validation result at epoch   9, step     6000: bleu:   8.54, loss: 67483.0547, ppl:  13.9079, duration: 174.3574s\n",
      "2020-02-18 10:38:30,788 Epoch   9 Step:     6100 Batch Loss:     2.630455 Tokens per Sec:     3985, Lr: 0.000300\n",
      "2020-02-18 10:39:22,961 Epoch   9 Step:     6200 Batch Loss:     2.681404 Tokens per Sec:     4081, Lr: 0.000300\n",
      "2020-02-18 10:40:15,332 Epoch   9 Step:     6300 Batch Loss:     2.177304 Tokens per Sec:     4028, Lr: 0.000300\n",
      "2020-02-18 10:41:07,574 Epoch   9 Step:     6400 Batch Loss:     2.930943 Tokens per Sec:     4005, Lr: 0.000300\n",
      "2020-02-18 10:41:23,910 Epoch   9: total training loss 1609.61\n",
      "2020-02-18 10:41:23,910 EPOCH 10\n",
      "2020-02-18 10:41:59,683 Epoch  10 Step:     6500 Batch Loss:     2.611111 Tokens per Sec:     4088, Lr: 0.000300\n",
      "2020-02-18 10:42:53,187 Epoch  10 Step:     6600 Batch Loss:     2.940239 Tokens per Sec:     3993, Lr: 0.000300\n",
      "2020-02-18 10:43:46,991 Epoch  10 Step:     6700 Batch Loss:     2.327575 Tokens per Sec:     3838, Lr: 0.000300\n",
      "2020-02-18 10:44:40,984 Epoch  10 Step:     6800 Batch Loss:     2.491139 Tokens per Sec:     3835, Lr: 0.000300\n",
      "2020-02-18 10:45:35,284 Epoch  10 Step:     6900 Batch Loss:     2.732940 Tokens per Sec:     3990, Lr: 0.000300\n",
      "2020-02-18 10:46:26,785 Epoch  10 Step:     7000 Batch Loss:     2.411959 Tokens per Sec:     4075, Lr: 0.000300\n",
      "2020-02-18 10:49:20,709 Hooray! New best validation result [ppl]!\n",
      "2020-02-18 10:49:20,709 Saving new checkpoint.\n",
      "2020-02-18 10:49:20,930 Example #0\n",
      "2020-02-18 10:49:20,930 \tSource:     Gideon wondered how it would be possible for him to “ save Israel out of Midian’s hand . ”\n",
      "2020-02-18 10:49:20,930 \tReference:  Mũlaĩka ũsu nĩwaneenie vandũ va Mũmbi na amũĩkĩĩthya Ngiteoni kana Yeova aĩ vamwe nake .\n",
      "2020-02-18 10:49:20,930 \tHypothesis: Kũsũanĩa ũndũ ũsu no kwĩthĩwa wamũtetheeisye mũno nũndũ wa “ Isilaeli ma Isilaeli . ”\n",
      "2020-02-18 10:49:20,931 Example #1\n",
      "2020-02-18 10:49:20,931 \tSource:     And then she would try to return to Hosea .\n",
      "2020-02-18 10:49:20,931 \tReference:  Na ĩndĩ kyamina kũsembany’a na endwa makyo Yeova aĩtye kĩkatata kũmũsyokea Osea .\n",
      "2020-02-18 10:49:20,931 \tHypothesis: Na ĩndĩ atanamba kũka , nĩwaĩle kũmũvoya Ngai .\n",
      "2020-02-18 10:49:20,931 Example #2\n",
      "2020-02-18 10:49:20,931 \tSource:     The Bible itself says : “ The green grass dries up , the blossom withers , but the word of our God endures forever . ” ​ — Isaiah 40 : 8 .\n",
      "2020-02-18 10:49:20,931 \tReference:  Mbivilia yaĩtye atĩĩ : “ Nyeki nĩyũmaa , na ĩlaa nĩyĩvovaa ; ĩndĩ ndeto ya Ngai waitũ ĩkekala tene na tene . ” — Isaia 40 : 8 .\n",
      "2020-02-18 10:49:20,931 \tHypothesis: Mbivilia yaĩtye atĩĩ : “ Ĩĩ , na ndeto sya Ngai , ĩndĩ ndeto sya andũ onthe , ĩndĩ ndeto sya Ngai ikavetanga , ĩndĩ ti lasima . ” — Isaia 40 : 8 .\n",
      "2020-02-18 10:49:20,931 Example #3\n",
      "2020-02-18 10:49:20,932 \tSource:     It includes links to lists of Witnesses currently imprisoned for their faith .\n",
      "2020-02-18 10:49:20,932 \tReference:  7 : 12 ) Walika Kĩsesenĩ kya jw.org no wone masyĩtwa ma Ngũsĩ ila syovetwe .\n",
      "2020-02-18 10:49:20,932 \tHypothesis: Andũ amwe ma ndĩni ila syĩthĩawa na mũĩkĩĩo mũlũmu wa Ngũsĩ sya Yeova .\n",
      "2020-02-18 10:49:20,932 Validation result at epoch  10, step     7000: bleu:   8.82, loss: 65314.2305, ppl:  12.7796, duration: 174.1459s\n",
      "2020-02-18 10:49:38,839 Epoch  10: total training loss 1564.13\n",
      "2020-02-18 10:49:38,839 EPOCH 11\n",
      "2020-02-18 10:50:13,137 Epoch  11 Step:     7100 Batch Loss:     2.448247 Tokens per Sec:     3967, Lr: 0.000300\n",
      "2020-02-18 10:51:05,240 Epoch  11 Step:     7200 Batch Loss:     2.586154 Tokens per Sec:     4043, Lr: 0.000300\n",
      "2020-02-18 10:51:57,569 Epoch  11 Step:     7300 Batch Loss:     2.728182 Tokens per Sec:     3981, Lr: 0.000300\n",
      "2020-02-18 10:52:49,876 Epoch  11 Step:     7400 Batch Loss:     2.446147 Tokens per Sec:     4085, Lr: 0.000300\n",
      "2020-02-18 10:53:42,618 Epoch  11 Step:     7500 Batch Loss:     2.623372 Tokens per Sec:     3979, Lr: 0.000300\n",
      "2020-02-18 10:54:35,076 Epoch  11 Step:     7600 Batch Loss:     2.971872 Tokens per Sec:     4076, Lr: 0.000300\n",
      "2020-02-18 10:54:55,652 Epoch  11: total training loss 1540.82\n",
      "2020-02-18 10:54:55,652 EPOCH 12\n",
      "2020-02-18 10:55:29,379 Epoch  12 Step:     7700 Batch Loss:     2.695367 Tokens per Sec:     3920, Lr: 0.000300\n",
      "2020-02-18 10:56:23,617 Epoch  12 Step:     7800 Batch Loss:     2.228133 Tokens per Sec:     3887, Lr: 0.000300\n",
      "2020-02-18 10:57:17,713 Epoch  12 Step:     7900 Batch Loss:     2.218940 Tokens per Sec:     3851, Lr: 0.000300\n",
      "2020-02-18 10:58:09,477 Epoch  12 Step:     8000 Batch Loss:     2.496810 Tokens per Sec:     4021, Lr: 0.000300\n",
      "2020-02-18 11:01:03,677 Hooray! New best validation result [ppl]!\n",
      "2020-02-18 11:01:03,678 Saving new checkpoint.\n",
      "2020-02-18 11:01:03,917 Example #0\n",
      "2020-02-18 11:01:03,917 \tSource:     Gideon wondered how it would be possible for him to “ save Israel out of Midian’s hand . ”\n",
      "2020-02-18 11:01:03,917 \tReference:  Mũlaĩka ũsu nĩwaneenie vandũ va Mũmbi na amũĩkĩĩthya Ngiteoni kana Yeova aĩ vamwe nake .\n",
      "2020-02-18 11:01:03,917 \tHypothesis: Kũsũanĩa ũndũ Sala weewie ĩla wamwĩie “ Isilaeli ma Isilaeli ” ma Isilaeli .\n",
      "2020-02-18 11:01:03,917 Example #1\n",
      "2020-02-18 11:01:03,917 \tSource:     And then she would try to return to Hosea .\n",
      "2020-02-18 11:01:03,917 \tReference:  Na ĩndĩ kyamina kũsembany’a na endwa makyo Yeova aĩtye kĩkatata kũmũsyokea Osea .\n",
      "2020-02-18 11:01:03,918 \tHypothesis: Na ĩndĩ atanamba kũmũvoya Ngai amũtavisye Austria .\n",
      "2020-02-18 11:01:03,918 Example #2\n",
      "2020-02-18 11:01:03,918 \tSource:     The Bible itself says : “ The green grass dries up , the blossom withers , but the word of our God endures forever . ” ​ — Isaiah 40 : 8 .\n",
      "2020-02-18 11:01:03,918 \tReference:  Mbivilia yaĩtye atĩĩ : “ Nyeki nĩyũmaa , na ĩlaa nĩyĩvovaa ; ĩndĩ ndeto ya Ngai waitũ ĩkekala tene na tene . ” — Isaia 40 : 8 .\n",
      "2020-02-18 11:01:03,918 \tHypothesis: Mbivilia yaĩtye atĩĩ : “ Mũikolwe nĩ nthakame , ĩndĩ ndeto sya Ngai , ĩndĩ ndeto sya Ngai ikavetanga . ” — Isaia 40 : 8 .\n",
      "2020-02-18 11:01:03,918 Example #3\n",
      "2020-02-18 11:01:03,918 \tSource:     It includes links to lists of Witnesses currently imprisoned for their faith .\n",
      "2020-02-18 11:01:03,918 \tReference:  7 : 12 ) Walika Kĩsesenĩ kya jw.org no wone masyĩtwa ma Ngũsĩ ila syovetwe .\n",
      "2020-02-18 11:01:03,918 \tHypothesis: Andũ amwe ala maĩkĩĩaa kana Ngũsĩ sya Yeova nĩsyavĩndũiwe nĩ mũĩkĩĩo woo .\n",
      "2020-02-18 11:01:03,918 Validation result at epoch  12, step     8000: bleu:   9.83, loss: 63758.0625, ppl:  12.0269, duration: 174.4406s\n",
      "2020-02-18 11:01:56,582 Epoch  12 Step:     8100 Batch Loss:     1.935899 Tokens per Sec:     4053, Lr: 0.000300\n",
      "2020-02-18 11:02:49,124 Epoch  12 Step:     8200 Batch Loss:     2.444760 Tokens per Sec:     4011, Lr: 0.000300\n",
      "2020-02-18 11:03:11,376 Epoch  12: total training loss 1508.04\n",
      "2020-02-18 11:03:11,377 EPOCH 13\n",
      "2020-02-18 11:03:42,022 Epoch  13 Step:     8300 Batch Loss:     2.835359 Tokens per Sec:     4092, Lr: 0.000300\n",
      "2020-02-18 11:04:34,582 Epoch  13 Step:     8400 Batch Loss:     2.464274 Tokens per Sec:     3933, Lr: 0.000300\n",
      "2020-02-18 11:05:27,084 Epoch  13 Step:     8500 Batch Loss:     2.941682 Tokens per Sec:     3992, Lr: 0.000300\n",
      "2020-02-18 11:06:18,665 Epoch  13 Step:     8600 Batch Loss:     2.388735 Tokens per Sec:     3941, Lr: 0.000300\n",
      "2020-02-18 11:07:12,854 Epoch  13 Step:     8700 Batch Loss:     2.270217 Tokens per Sec:     4018, Lr: 0.000300\n",
      "2020-02-18 11:08:06,702 Epoch  13 Step:     8800 Batch Loss:     2.455169 Tokens per Sec:     3846, Lr: 0.000300\n",
      "2020-02-18 11:08:32,717 Epoch  13: total training loss 1485.66\n",
      "2020-02-18 11:08:32,718 EPOCH 14\n",
      "2020-02-18 11:09:00,567 Epoch  14 Step:     8900 Batch Loss:     2.364492 Tokens per Sec:     3780, Lr: 0.000300\n",
      "2020-02-18 11:09:53,104 Epoch  14 Step:     9000 Batch Loss:     2.629810 Tokens per Sec:     4015, Lr: 0.000300\n",
      "2020-02-18 11:12:47,044 Hooray! New best validation result [ppl]!\n",
      "2020-02-18 11:12:47,045 Saving new checkpoint.\n",
      "2020-02-18 11:12:47,266 Example #0\n",
      "2020-02-18 11:12:47,267 \tSource:     Gideon wondered how it would be possible for him to “ save Israel out of Midian’s hand . ”\n",
      "2020-02-18 11:12:47,267 \tReference:  Mũlaĩka ũsu nĩwaneenie vandũ va Mũmbi na amũĩkĩĩthya Ngiteoni kana Yeova aĩ vamwe nake .\n",
      "2020-02-18 11:12:47,267 \tHypothesis: Kũsũanĩa ũndũ andũ mamwĩkwatĩtye , “ Ngoliathu ” no aũtetheesye Aisilaeli .\n",
      "2020-02-18 11:12:47,267 Example #1\n",
      "2020-02-18 11:12:47,267 \tSource:     And then she would try to return to Hosea .\n",
      "2020-02-18 11:12:47,267 \tReference:  Na ĩndĩ kyamina kũsembany’a na endwa makyo Yeova aĩtye kĩkatata kũmũsyokea Osea .\n",
      "2020-02-18 11:12:47,268 \tHypothesis: Na ĩndĩ atanamba kũmũvoya Ngai akũnie Austria .\n",
      "2020-02-18 11:12:47,268 Example #2\n",
      "2020-02-18 11:12:47,268 \tSource:     The Bible itself says : “ The green grass dries up , the blossom withers , but the word of our God endures forever . ” ​ — Isaiah 40 : 8 .\n",
      "2020-02-18 11:12:47,268 \tReference:  Mbivilia yaĩtye atĩĩ : “ Nyeki nĩyũmaa , na ĩlaa nĩyĩvovaa ; ĩndĩ ndeto ya Ngai waitũ ĩkekala tene na tene . ” — Isaia 40 : 8 .\n",
      "2020-02-18 11:12:47,268 \tHypothesis: Mbivilia yaĩtye atĩĩ : “ Na [ Ngai ] nĩwookie , ĩndĩ nĩ kana tũkwate ndeto sya Ngai , ĩndĩ nĩ kana tũkwate thayũ ũtathela . ” — Isaia 40 : 8 .\n",
      "2020-02-18 11:12:47,269 Example #3\n",
      "2020-02-18 11:12:47,269 \tSource:     It includes links to lists of Witnesses currently imprisoned for their faith .\n",
      "2020-02-18 11:12:47,269 \tReference:  7 : 12 ) Walika Kĩsesenĩ kya jw.org no wone masyĩtwa ma Ngũsĩ ila syovetwe .\n",
      "2020-02-18 11:12:47,269 \tHypothesis: Andũ amwe maĩkĩĩaa kana Ngũsĩ sya Yeova nĩsyavĩndũiwe nĩ mũĩkĩĩo woo .\n",
      "2020-02-18 11:12:47,269 Validation result at epoch  14, step     9000: bleu:   9.73, loss: 62237.8672, ppl:  11.3345, duration: 174.1636s\n",
      "2020-02-18 11:13:39,296 Epoch  14 Step:     9100 Batch Loss:     2.585392 Tokens per Sec:     3981, Lr: 0.000300\n",
      "2020-02-18 11:14:31,769 Epoch  14 Step:     9200 Batch Loss:     2.421518 Tokens per Sec:     4040, Lr: 0.000300\n",
      "2020-02-18 11:15:24,354 Epoch  14 Step:     9300 Batch Loss:     2.619276 Tokens per Sec:     4010, Lr: 0.000300\n",
      "2020-02-18 11:16:16,970 Epoch  14 Step:     9400 Batch Loss:     2.445053 Tokens per Sec:     4060, Lr: 0.000300\n",
      "2020-02-18 11:16:44,487 Epoch  14: total training loss 1456.33\n",
      "2020-02-18 11:16:44,487 EPOCH 15\n",
      "2020-02-18 11:17:09,692 Epoch  15 Step:     9500 Batch Loss:     2.360329 Tokens per Sec:     4033, Lr: 0.000300\n",
      "2020-02-18 11:18:02,119 Epoch  15 Step:     9600 Batch Loss:     2.077299 Tokens per Sec:     4056, Lr: 0.000300\n",
      "2020-02-18 11:18:54,596 Epoch  15 Step:     9700 Batch Loss:     2.569376 Tokens per Sec:     3920, Lr: 0.000300\n",
      "2020-02-18 11:19:49,118 Epoch  15 Step:     9800 Batch Loss:     2.516905 Tokens per Sec:     3893, Lr: 0.000300\n",
      "2020-02-18 11:20:43,676 Epoch  15 Step:     9900 Batch Loss:     2.531918 Tokens per Sec:     3902, Lr: 0.000300\n",
      "2020-02-18 11:21:36,716 Epoch  15 Step:    10000 Batch Loss:     2.200826 Tokens per Sec:     3868, Lr: 0.000300\n",
      "2020-02-18 11:24:30,065 Hooray! New best validation result [ppl]!\n",
      "2020-02-18 11:24:30,065 Saving new checkpoint.\n",
      "2020-02-18 11:24:30,297 Example #0\n",
      "2020-02-18 11:24:30,297 \tSource:     Gideon wondered how it would be possible for him to “ save Israel out of Midian’s hand . ”\n",
      "2020-02-18 11:24:30,297 \tReference:  Mũlaĩka ũsu nĩwaneenie vandũ va Mũmbi na amũĩkĩĩthya Ngiteoni kana Yeova aĩ vamwe nake .\n",
      "2020-02-18 11:24:30,297 \tHypothesis: Keka nue wĩthĩwa wamanyie ũndũ ũtonya kwĩka , na kwoou no kwĩthĩwa aseng’ie mũno nũndũ wa ũu “ Isilaeli ma Isilaeli ma Isilaeli . ”\n",
      "2020-02-18 11:24:30,297 Example #1\n",
      "2020-02-18 11:24:30,298 \tSource:     And then she would try to return to Hosea .\n",
      "2020-02-18 11:24:30,298 \tReference:  Na ĩndĩ kyamina kũsembany’a na endwa makyo Yeova aĩtye kĩkatata kũmũsyokea Osea .\n",
      "2020-02-18 11:24:30,298 \tHypothesis: Na ĩndĩ atanamba kũmũvoya Ngai akũmina Akalatia .\n",
      "2020-02-18 11:24:30,298 Example #2\n",
      "2020-02-18 11:24:30,298 \tSource:     The Bible itself says : “ The green grass dries up , the blossom withers , but the word of our God endures forever . ” ​ — Isaiah 40 : 8 .\n",
      "2020-02-18 11:24:30,298 \tReference:  Mbivilia yaĩtye atĩĩ : “ Nyeki nĩyũmaa , na ĩlaa nĩyĩvovaa ; ĩndĩ ndeto ya Ngai waitũ ĩkekala tene na tene . ” — Isaia 40 : 8 .\n",
      "2020-02-18 11:24:30,298 \tHypothesis: Mbivilia yaĩtye atĩĩ : “ Na ĩndĩ ndeto sya Ngai , ĩndĩ ndeto ya Ngai , ĩndĩ ndeto ya Ngai , ĩndĩ ndeto ya Ngai , ĩndĩ ndeto sitũ syoombiwe tene na tene . ” — Isaia 40 : 8 .\n",
      "2020-02-18 11:24:30,298 Example #3\n",
      "2020-02-18 11:24:30,298 \tSource:     It includes links to lists of Witnesses currently imprisoned for their faith .\n",
      "2020-02-18 11:24:30,299 \tReference:  7 : 12 ) Walika Kĩsesenĩ kya jw.org no wone masyĩtwa ma Ngũsĩ ila syovetwe .\n",
      "2020-02-18 11:24:30,299 \tHypothesis: Amwe ma andũ ma mbaĩ ila masyaanĩw’e nĩ Ngũsĩ sya Yeova ila syathyũlũlũkĩte .\n",
      "2020-02-18 11:24:30,299 Validation result at epoch  15, step    10000: bleu:  10.81, loss: 61609.9844, ppl:  11.0602, duration: 173.5824s\n",
      "2020-02-18 11:25:00,639 Epoch  15: total training loss 1440.54\n",
      "2020-02-18 11:25:00,639 EPOCH 16\n",
      "2020-02-18 11:25:23,052 Epoch  16 Step:    10100 Batch Loss:     2.338991 Tokens per Sec:     4017, Lr: 0.000300\n",
      "2020-02-18 11:26:15,214 Epoch  16 Step:    10200 Batch Loss:     2.420434 Tokens per Sec:     4030, Lr: 0.000300\n",
      "2020-02-18 11:27:07,598 Epoch  16 Step:    10300 Batch Loss:     2.320063 Tokens per Sec:     4081, Lr: 0.000300\n",
      "2020-02-18 11:27:59,020 Epoch  16 Step:    10400 Batch Loss:     2.539362 Tokens per Sec:     4006, Lr: 0.000300\n",
      "2020-02-18 11:28:51,779 Epoch  16 Step:    10500 Batch Loss:     2.219864 Tokens per Sec:     4106, Lr: 0.000300\n",
      "2020-02-18 11:29:44,011 Epoch  16 Step:    10600 Batch Loss:     2.384245 Tokens per Sec:     3959, Lr: 0.000300\n",
      "2020-02-18 11:30:16,606 Epoch  16: total training loss 1410.80\n",
      "2020-02-18 11:30:16,606 EPOCH 17\n",
      "2020-02-18 11:30:36,434 Epoch  17 Step:    10700 Batch Loss:     2.082400 Tokens per Sec:     4094, Lr: 0.000300\n",
      "2020-02-18 11:31:30,914 Epoch  17 Step:    10800 Batch Loss:     2.323931 Tokens per Sec:     3928, Lr: 0.000300\n",
      "2020-02-18 11:32:25,319 Epoch  17 Step:    10900 Batch Loss:     2.383948 Tokens per Sec:     3943, Lr: 0.000300\n",
      "2020-02-18 11:33:19,754 Epoch  17 Step:    11000 Batch Loss:     2.964631 Tokens per Sec:     3902, Lr: 0.000300\n",
      "2020-02-18 11:36:13,433 Example #0\n",
      "2020-02-18 11:36:13,433 \tSource:     Gideon wondered how it would be possible for him to “ save Israel out of Midian’s hand . ”\n",
      "2020-02-18 11:36:13,433 \tReference:  Mũlaĩka ũsu nĩwaneenie vandũ va Mũmbi na amũĩkĩĩthya Ngiteoni kana Yeova aĩ vamwe nake .\n",
      "2020-02-18 11:36:13,433 \tHypothesis: Kwaekai twone ũndũ Ngiteoni wamũtetheeisye Aisilaeli ‘ mamwona ’ ma Isilaeli .\n",
      "2020-02-18 11:36:13,433 Example #1\n",
      "2020-02-18 11:36:13,434 \tSource:     And then she would try to return to Hosea .\n",
      "2020-02-18 11:36:13,434 \tReference:  Na ĩndĩ kyamina kũsembany’a na endwa makyo Yeova aĩtye kĩkatata kũmũsyokea Osea .\n",
      "2020-02-18 11:36:13,434 \tHypothesis: Na ĩndĩ nĩwaumisye ũkany’o wa Akali .\n",
      "2020-02-18 11:36:13,434 Example #2\n",
      "2020-02-18 11:36:13,434 \tSource:     The Bible itself says : “ The green grass dries up , the blossom withers , but the word of our God endures forever . ” ​ — Isaiah 40 : 8 .\n",
      "2020-02-18 11:36:13,434 \tReference:  Mbivilia yaĩtye atĩĩ : “ Nyeki nĩyũmaa , na ĩlaa nĩyĩvovaa ; ĩndĩ ndeto ya Ngai waitũ ĩkekala tene na tene . ” — Isaia 40 : 8 .\n",
      "2020-02-18 11:36:13,434 \tHypothesis: Mbivilia yaĩtye atĩĩ : “ Na [ Ngai ] nĩwaleile na ngoo ya kwenda kwa Ngai . ” — Isaia 40 : 8 .\n",
      "2020-02-18 11:36:13,434 Example #3\n",
      "2020-02-18 11:36:13,435 \tSource:     It includes links to lists of Witnesses currently imprisoned for their faith .\n",
      "2020-02-18 11:36:13,435 \tReference:  7 : 12 ) Walika Kĩsesenĩ kya jw.org no wone masyĩtwa ma Ngũsĩ ila syovetwe .\n",
      "2020-02-18 11:36:13,435 \tHypothesis: Andũ amwe nĩmaandĩkie ĩũlũ wa Ngũsĩ sya Yeova nĩmaumisye mũĩkĩĩo woo .\n",
      "2020-02-18 11:36:13,435 Validation result at epoch  17, step    11000: bleu:   9.78, loss: 61798.1211, ppl:  11.1417, duration: 173.6799s\n",
      "2020-02-18 11:37:05,826 Epoch  17 Step:    11100 Batch Loss:     2.598932 Tokens per Sec:     3994, Lr: 0.000300\n",
      "2020-02-18 11:37:57,760 Epoch  17 Step:    11200 Batch Loss:     2.438352 Tokens per Sec:     3946, Lr: 0.000300\n",
      "2020-02-18 11:38:31,744 Epoch  17: total training loss 1392.46\n",
      "2020-02-18 11:38:31,744 EPOCH 18\n",
      "2020-02-18 11:38:50,331 Epoch  18 Step:    11300 Batch Loss:     2.301619 Tokens per Sec:     4135, Lr: 0.000300\n",
      "2020-02-18 11:39:43,165 Epoch  18 Step:    11400 Batch Loss:     1.953793 Tokens per Sec:     3954, Lr: 0.000300\n",
      "2020-02-18 11:40:35,521 Epoch  18 Step:    11500 Batch Loss:     2.618205 Tokens per Sec:     3988, Lr: 0.000300\n",
      "2020-02-18 11:41:28,014 Epoch  18 Step:    11600 Batch Loss:     1.617588 Tokens per Sec:     3985, Lr: 0.000300\n",
      "2020-02-18 11:42:20,794 Epoch  18 Step:    11700 Batch Loss:     2.464936 Tokens per Sec:     4116, Lr: 0.000300\n",
      "2020-02-18 11:43:14,045 Epoch  18 Step:    11800 Batch Loss:     2.339690 Tokens per Sec:     3914, Lr: 0.000300\n",
      "2020-02-18 11:43:50,407 Epoch  18: total training loss 1365.49\n",
      "2020-02-18 11:43:50,407 EPOCH 19\n",
      "2020-02-18 11:44:08,608 Epoch  19 Step:    11900 Batch Loss:     2.465928 Tokens per Sec:     3840, Lr: 0.000300\n",
      "2020-02-18 11:45:02,807 Epoch  19 Step:    12000 Batch Loss:     2.126937 Tokens per Sec:     3829, Lr: 0.000300\n",
      "2020-02-18 11:47:56,851 Hooray! New best validation result [ppl]!\n",
      "2020-02-18 11:47:56,852 Saving new checkpoint.\n",
      "2020-02-18 11:47:57,075 Example #0\n",
      "2020-02-18 11:47:57,076 \tSource:     Gideon wondered how it would be possible for him to “ save Israel out of Midian’s hand . ”\n",
      "2020-02-18 11:47:57,076 \tReference:  Mũlaĩka ũsu nĩwaneenie vandũ va Mũmbi na amũĩkĩĩthya Ngiteoni kana Yeova aĩ vamwe nake .\n",
      "2020-02-18 11:47:57,076 \tHypothesis: Kũsũanĩa ũndũ Sala weewie ĩla woonie ũndũ ũtonya ‘ kũtia Isilaeli ma Isilaeli . ’\n",
      "2020-02-18 11:47:57,076 Example #1\n",
      "2020-02-18 11:47:57,076 \tSource:     And then she would try to return to Hosea .\n",
      "2020-02-18 11:47:57,077 \tReference:  Na ĩndĩ kyamina kũsembany’a na endwa makyo Yeova aĩtye kĩkatata kũmũsyokea Osea .\n",
      "2020-02-18 11:47:57,077 \tHypothesis: Na ĩndĩ atata kũsyokea Hutter .\n",
      "2020-02-18 11:47:57,077 Example #2\n",
      "2020-02-18 11:47:57,077 \tSource:     The Bible itself says : “ The green grass dries up , the blossom withers , but the word of our God endures forever . ” ​ — Isaiah 40 : 8 .\n",
      "2020-02-18 11:47:57,077 \tReference:  Mbivilia yaĩtye atĩĩ : “ Nyeki nĩyũmaa , na ĩlaa nĩyĩvovaa ; ĩndĩ ndeto ya Ngai waitũ ĩkekala tene na tene . ” — Isaia 40 : 8 .\n",
      "2020-02-18 11:47:57,077 \tHypothesis: Mbivilia yaĩtye atĩĩ : “ Kwona , na kũya , na kũya , ĩndĩ kũya , na kũya , ĩndĩ ndeto ya Ngai , ĩndĩ thayũ ũtathela tene na tene . ” — Isaia 40 : 8 .\n",
      "2020-02-18 11:47:57,077 Example #3\n",
      "2020-02-18 11:47:57,078 \tSource:     It includes links to lists of Witnesses currently imprisoned for their faith .\n",
      "2020-02-18 11:47:57,078 \tReference:  7 : 12 ) Walika Kĩsesenĩ kya jw.org no wone masyĩtwa ma Ngũsĩ ila syovetwe .\n",
      "2020-02-18 11:47:57,078 \tHypothesis: Andũ amwe ma saenzi ala maandĩkie ĩũlũ wa Ngũsĩ sya Yeova nĩmambĩĩe kũlika mũĩkĩĩo woo .\n",
      "2020-02-18 11:47:57,078 Validation result at epoch  19, step    12000: bleu:  11.68, loss: 59236.7617, ppl:  10.0823, duration: 174.2707s\n",
      "2020-02-18 11:48:49,893 Epoch  19 Step:    12100 Batch Loss:     1.898798 Tokens per Sec:     4026, Lr: 0.000300\n",
      "2020-02-18 11:49:41,893 Epoch  19 Step:    12200 Batch Loss:     2.277779 Tokens per Sec:     4007, Lr: 0.000300\n",
      "2020-02-18 11:50:34,376 Epoch  19 Step:    12300 Batch Loss:     2.367811 Tokens per Sec:     4069, Lr: 0.000300\n",
      "2020-02-18 11:51:27,580 Epoch  19 Step:    12400 Batch Loss:     2.299616 Tokens per Sec:     4051, Lr: 0.000300\n",
      "2020-02-18 11:52:04,041 Epoch  19: total training loss 1349.84\n",
      "2020-02-18 11:52:04,042 EPOCH 20\n",
      "2020-02-18 11:52:19,956 Epoch  20 Step:    12500 Batch Loss:     1.189358 Tokens per Sec:     3970, Lr: 0.000300\n",
      "2020-02-18 11:53:12,587 Epoch  20 Step:    12600 Batch Loss:     2.306059 Tokens per Sec:     4044, Lr: 0.000300\n",
      "2020-02-18 11:54:05,543 Epoch  20 Step:    12700 Batch Loss:     2.216141 Tokens per Sec:     4044, Lr: 0.000300\n",
      "2020-02-18 11:54:58,332 Epoch  20 Step:    12800 Batch Loss:     2.317630 Tokens per Sec:     3947, Lr: 0.000300\n",
      "2020-02-18 11:55:52,986 Epoch  20 Step:    12900 Batch Loss:     2.310398 Tokens per Sec:     3934, Lr: 0.000300\n",
      "2020-02-18 11:56:47,466 Epoch  20 Step:    13000 Batch Loss:     2.270726 Tokens per Sec:     3944, Lr: 0.000300\n",
      "2020-02-18 11:59:41,922 Hooray! New best validation result [ppl]!\n",
      "2020-02-18 11:59:41,922 Saving new checkpoint.\n",
      "2020-02-18 11:59:42,144 Example #0\n",
      "2020-02-18 11:59:42,145 \tSource:     Gideon wondered how it would be possible for him to “ save Israel out of Midian’s hand . ”\n",
      "2020-02-18 11:59:42,145 \tReference:  Mũlaĩka ũsu nĩwaneenie vandũ va Mũmbi na amũĩkĩĩthya Ngiteoni kana Yeova aĩ vamwe nake .\n",
      "2020-02-18 11:59:42,145 \tHypothesis: Kũsũanĩa ũndũ Ngiteoni wamũtetheeisye Aisilaeli ‘ mamwona ’ wa Methike .\n",
      "2020-02-18 11:59:42,145 Example #1\n",
      "2020-02-18 11:59:42,145 \tSource:     And then she would try to return to Hosea .\n",
      "2020-02-18 11:59:42,146 \tReference:  Na ĩndĩ kyamina kũsembany’a na endwa makyo Yeova aĩtye kĩkatata kũmũsyokea Osea .\n",
      "2020-02-18 11:59:42,146 \tHypothesis: Na ĩndĩ atataa ũndũ ũtonya akasyoka Austria .\n",
      "2020-02-18 11:59:42,146 Example #2\n",
      "2020-02-18 11:59:42,146 \tSource:     The Bible itself says : “ The green grass dries up , the blossom withers , but the word of our God endures forever . ” ​ — Isaiah 40 : 8 .\n",
      "2020-02-18 11:59:42,146 \tReference:  Mbivilia yaĩtye atĩĩ : “ Nyeki nĩyũmaa , na ĩlaa nĩyĩvovaa ; ĩndĩ ndeto ya Ngai waitũ ĩkekala tene na tene . ” — Isaia 40 : 8 .\n",
      "2020-02-18 11:59:42,146 \tHypothesis: Mbivilia yaĩtye atĩĩ : “ Na ĩndĩ ndeto sya Ngai nĩsyavingũie , ĩndĩ ndeto sitũ syoombiwe tene na tene . ” — Isaia 40 : 8 .\n",
      "2020-02-18 11:59:42,146 Example #3\n",
      "2020-02-18 11:59:42,147 \tSource:     It includes links to lists of Witnesses currently imprisoned for their faith .\n",
      "2020-02-18 11:59:42,147 \tReference:  7 : 12 ) Walika Kĩsesenĩ kya jw.org no wone masyĩtwa ma Ngũsĩ ila syovetwe .\n",
      "2020-02-18 11:59:42,147 \tHypothesis: Amwe moo nĩmakwatawa nĩ andũ ma mbaĩ ila syavĩndũiwe nĩ andũ ma mbaĩ ila syalikile mũĩkĩĩo woo .\n",
      "2020-02-18 11:59:42,147 Validation result at epoch  20, step    13000: bleu:  12.00, loss: 58031.4453, ppl:   9.6192, duration: 174.6802s\n",
      "2020-02-18 12:00:17,989 Epoch  20: total training loss 1326.64\n",
      "2020-02-18 12:00:17,989 EPOCH 21\n",
      "2020-02-18 12:00:33,992 Epoch  21 Step:    13100 Batch Loss:     1.804206 Tokens per Sec:     4020, Lr: 0.000300\n",
      "2020-02-18 12:01:26,011 Epoch  21 Step:    13200 Batch Loss:     2.533078 Tokens per Sec:     4024, Lr: 0.000300\n",
      "2020-02-18 12:02:18,154 Epoch  21 Step:    13300 Batch Loss:     2.113488 Tokens per Sec:     4016, Lr: 0.000300\n",
      "2020-02-18 12:03:10,456 Epoch  21 Step:    13400 Batch Loss:     2.388974 Tokens per Sec:     4022, Lr: 0.000300\n",
      "2020-02-18 12:04:03,423 Epoch  21 Step:    13500 Batch Loss:     2.476287 Tokens per Sec:     4051, Lr: 0.000300\n",
      "2020-02-18 12:04:55,721 Epoch  21 Step:    13600 Batch Loss:     2.331734 Tokens per Sec:     4069, Lr: 0.000300\n",
      "2020-02-18 12:05:32,797 Epoch  21: total training loss 1320.69\n",
      "2020-02-18 12:05:32,797 EPOCH 22\n",
      "2020-02-18 12:05:48,563 Epoch  22 Step:    13700 Batch Loss:     1.824705 Tokens per Sec:     4216, Lr: 0.000300\n",
      "2020-02-18 12:06:41,448 Epoch  22 Step:    13800 Batch Loss:     2.019697 Tokens per Sec:     4065, Lr: 0.000300\n",
      "2020-02-18 12:07:35,250 Epoch  22 Step:    13900 Batch Loss:     1.992615 Tokens per Sec:     3898, Lr: 0.000300\n",
      "2020-02-18 12:08:29,883 Epoch  22 Step:    14000 Batch Loss:     1.867369 Tokens per Sec:     3938, Lr: 0.000300\n",
      "2020-02-18 12:11:25,081 Hooray! New best validation result [ppl]!\n",
      "2020-02-18 12:11:25,081 Saving new checkpoint.\n",
      "2020-02-18 12:11:25,296 Example #0\n",
      "2020-02-18 12:11:25,296 \tSource:     Gideon wondered how it would be possible for him to “ save Israel out of Midian’s hand . ”\n",
      "2020-02-18 12:11:25,296 \tReference:  Mũlaĩka ũsu nĩwaneenie vandũ va Mũmbi na amũĩkĩĩthya Ngiteoni kana Yeova aĩ vamwe nake .\n",
      "2020-02-18 12:11:25,296 \tHypothesis: Kyalo kĩu kyatumie Ngiteoni enda ‘ kũmũtavya mbaĩ ya Isilaeli . ’\n",
      "2020-02-18 12:11:25,297 Example #1\n",
      "2020-02-18 12:11:25,297 \tSource:     And then she would try to return to Hosea .\n",
      "2020-02-18 12:11:25,297 \tReference:  Na ĩndĩ kyamina kũsembany’a na endwa makyo Yeova aĩtye kĩkatata kũmũsyokea Osea .\n",
      "2020-02-18 12:11:25,297 \tHypothesis: Na ĩndĩ amina kũsyokea Akali .\n",
      "2020-02-18 12:11:25,297 Example #2\n",
      "2020-02-18 12:11:25,297 \tSource:     The Bible itself says : “ The green grass dries up , the blossom withers , but the word of our God endures forever . ” ​ — Isaiah 40 : 8 .\n",
      "2020-02-18 12:11:25,297 \tReference:  Mbivilia yaĩtye atĩĩ : “ Nyeki nĩyũmaa , na ĩlaa nĩyĩvovaa ; ĩndĩ ndeto ya Ngai waitũ ĩkekala tene na tene . ” — Isaia 40 : 8 .\n",
      "2020-02-18 12:11:25,297 \tHypothesis: Mbivilia yaĩtye atĩĩ : “ Na ĩndĩ ndeto ya Ngai , na ndeto ya Ngai , ĩndĩ ndeto sitũ ikavetanga mosũanĩo maitũ tene na tene . ” — Isaia 40 : 8 .\n",
      "2020-02-18 12:11:25,297 Example #3\n",
      "2020-02-18 12:11:25,298 \tSource:     It includes links to lists of Witnesses currently imprisoned for their faith .\n",
      "2020-02-18 12:11:25,298 \tReference:  7 : 12 ) Walika Kĩsesenĩ kya jw.org no wone masyĩtwa ma Ngũsĩ ila syovetwe .\n",
      "2020-02-18 12:11:25,298 \tHypothesis: Amwe ma ala maumĩĩasya Ngũsĩ sya Yeova nĩmambĩĩe kũolwa nĩ mũĩkĩĩo woo .\n",
      "2020-02-18 12:11:25,298 Validation result at epoch  22, step    14000: bleu:  12.77, loss: 57422.1758, ppl:   9.3932, duration: 175.4141s\n",
      "2020-02-18 12:12:18,055 Epoch  22 Step:    14100 Batch Loss:     2.198955 Tokens per Sec:     4051, Lr: 0.000300\n",
      "2020-02-18 12:13:09,976 Epoch  22 Step:    14200 Batch Loss:     2.221932 Tokens per Sec:     3962, Lr: 0.000300\n",
      "2020-02-18 12:13:46,391 Epoch  22: total training loss 1291.14\n",
      "2020-02-18 12:13:46,392 EPOCH 23\n",
      "2020-02-18 12:14:02,693 Epoch  23 Step:    14300 Batch Loss:     2.291621 Tokens per Sec:     3955, Lr: 0.000300\n",
      "2020-02-18 12:14:55,256 Epoch  23 Step:    14400 Batch Loss:     2.317471 Tokens per Sec:     4040, Lr: 0.000300\n",
      "2020-02-18 12:15:47,684 Epoch  23 Step:    14500 Batch Loss:     2.278880 Tokens per Sec:     4011, Lr: 0.000300\n",
      "2020-02-18 12:16:40,561 Epoch  23 Step:    14600 Batch Loss:     2.398612 Tokens per Sec:     4046, Lr: 0.000300\n",
      "2020-02-18 12:17:32,488 Epoch  23 Step:    14700 Batch Loss:     2.133232 Tokens per Sec:     3970, Lr: 0.000300\n",
      "2020-02-18 12:18:24,824 Epoch  23 Step:    14800 Batch Loss:     1.840600 Tokens per Sec:     4030, Lr: 0.000300\n",
      "2020-02-18 12:19:03,754 Epoch  23: total training loss 1290.07\n",
      "2020-02-18 12:19:03,755 EPOCH 24\n",
      "2020-02-18 12:19:18,910 Epoch  24 Step:    14900 Batch Loss:     2.193061 Tokens per Sec:     3906, Lr: 0.000300\n",
      "2020-02-18 12:20:13,619 Epoch  24 Step:    15000 Batch Loss:     2.240753 Tokens per Sec:     3895, Lr: 0.000300\n",
      "2020-02-18 12:23:09,163 Hooray! New best validation result [ppl]!\n",
      "2020-02-18 12:23:09,163 Saving new checkpoint.\n",
      "2020-02-18 12:23:09,383 Example #0\n",
      "2020-02-18 12:23:09,383 \tSource:     Gideon wondered how it would be possible for him to “ save Israel out of Midian’s hand . ”\n",
      "2020-02-18 12:23:09,383 \tReference:  Mũlaĩka ũsu nĩwaneenie vandũ va Mũmbi na amũĩkĩĩthya Ngiteoni kana Yeova aĩ vamwe nake .\n",
      "2020-02-18 12:23:09,383 \tHypothesis: Kũsũanĩa ũndũ ũtonya kũmũtavya ũndũ ũtonya ‘ kũkĩlya kw’oko kwa Isilaeli . ’\n",
      "2020-02-18 12:23:09,384 Example #1\n",
      "2020-02-18 12:23:09,384 \tSource:     And then she would try to return to Hosea .\n",
      "2020-02-18 12:23:09,384 \tReference:  Na ĩndĩ kyamina kũsembany’a na endwa makyo Yeova aĩtye kĩkatata kũmũsyokea Osea .\n",
      "2020-02-18 12:23:09,384 \tHypothesis: Na ĩndĩ atata kũsyokea Hutter .\n",
      "2020-02-18 12:23:09,384 Example #2\n",
      "2020-02-18 12:23:09,384 \tSource:     The Bible itself says : “ The green grass dries up , the blossom withers , but the word of our God endures forever . ” ​ — Isaiah 40 : 8 .\n",
      "2020-02-18 12:23:09,384 \tReference:  Mbivilia yaĩtye atĩĩ : “ Nyeki nĩyũmaa , na ĩlaa nĩyĩvovaa ; ĩndĩ ndeto ya Ngai waitũ ĩkekala tene na tene . ” — Isaia 40 : 8 .\n",
      "2020-02-18 12:23:09,384 \tHypothesis: Mbivilia yaĩtye atĩĩ : “ Na Ngai nĩwalũngile , na akĩĩa methoi maitũ , ĩndĩ ndeto ya Ngai ĩ thayũ tene na tene . ” — Isaia 40 : 8 .\n",
      "2020-02-18 12:23:09,384 Example #3\n",
      "2020-02-18 12:23:09,385 \tSource:     It includes links to lists of Witnesses currently imprisoned for their faith .\n",
      "2020-02-18 12:23:09,385 \tReference:  7 : 12 ) Walika Kĩsesenĩ kya jw.org no wone masyĩtwa ma Ngũsĩ ila syovetwe .\n",
      "2020-02-18 12:23:09,385 \tHypothesis: Amwe ma Ngũsĩ sya Yeova nĩmakwatawa nĩ mathĩna ala maumĩĩwe nĩ andũ ma mbaĩ ingĩ .\n",
      "2020-02-18 12:23:09,385 Validation result at epoch  24, step    15000: bleu:  12.62, loss: 56712.7461, ppl:   9.1369, duration: 175.7656s\n",
      "2020-02-18 12:24:01,625 Epoch  24 Step:    15100 Batch Loss:     1.952370 Tokens per Sec:     3934, Lr: 0.000300\n",
      "2020-02-18 12:24:53,894 Epoch  24 Step:    15200 Batch Loss:     2.189302 Tokens per Sec:     4008, Lr: 0.000300\n",
      "2020-02-18 12:25:46,597 Epoch  24 Step:    15300 Batch Loss:     2.070488 Tokens per Sec:     4053, Lr: 0.000300\n",
      "2020-02-18 12:26:39,473 Epoch  24 Step:    15400 Batch Loss:     2.148633 Tokens per Sec:     4039, Lr: 0.000300\n",
      "2020-02-18 12:27:18,890 Epoch  24: total training loss 1275.34\n",
      "2020-02-18 12:27:18,891 EPOCH 25\n",
      "2020-02-18 12:27:32,000 Epoch  25 Step:    15500 Batch Loss:     2.387338 Tokens per Sec:     4021, Lr: 0.000300\n",
      "2020-02-18 12:28:24,732 Epoch  25 Step:    15600 Batch Loss:     1.479596 Tokens per Sec:     4001, Lr: 0.000300\n",
      "2020-02-18 12:29:17,276 Epoch  25 Step:    15700 Batch Loss:     2.347648 Tokens per Sec:     3991, Lr: 0.000300\n",
      "2020-02-18 12:30:09,522 Epoch  25 Step:    15800 Batch Loss:     2.236275 Tokens per Sec:     3918, Lr: 0.000300\n",
      "2020-02-18 12:31:02,945 Epoch  25 Step:    15900 Batch Loss:     2.285147 Tokens per Sec:     3993, Lr: 0.000300\n",
      "2020-02-18 12:31:57,822 Epoch  25 Step:    16000 Batch Loss:     2.373275 Tokens per Sec:     4002, Lr: 0.000300\n",
      "2020-02-18 12:34:54,172 Hooray! New best validation result [ppl]!\n",
      "2020-02-18 12:34:54,172 Saving new checkpoint.\n",
      "2020-02-18 12:34:54,397 Example #0\n",
      "2020-02-18 12:34:54,398 \tSource:     Gideon wondered how it would be possible for him to “ save Israel out of Midian’s hand . ”\n",
      "2020-02-18 12:34:54,398 \tReference:  Mũlaĩka ũsu nĩwaneenie vandũ va Mũmbi na amũĩkĩĩthya Ngiteoni kana Yeova aĩ vamwe nake .\n",
      "2020-02-18 12:34:54,398 \tHypothesis: Ngiteoni eekũlasya ũndũ ũtonya ‘ kũtia Isilaeli . ’\n",
      "2020-02-18 12:34:54,398 Example #1\n",
      "2020-02-18 12:34:54,398 \tSource:     And then she would try to return to Hosea .\n",
      "2020-02-18 12:34:54,398 \tReference:  Na ĩndĩ kyamina kũsembany’a na endwa makyo Yeova aĩtye kĩkatata kũmũsyokea Osea .\n",
      "2020-02-18 12:34:54,398 \tHypothesis: Na ĩndĩ atataa ũndũ ũtonya aendeee na kũthamĩĩa Angelika .\n",
      "2020-02-18 12:34:54,398 Example #2\n",
      "2020-02-18 12:34:54,399 \tSource:     The Bible itself says : “ The green grass dries up , the blossom withers , but the word of our God endures forever . ” ​ — Isaiah 40 : 8 .\n",
      "2020-02-18 12:34:54,399 \tReference:  Mbivilia yaĩtye atĩĩ : “ Nyeki nĩyũmaa , na ĩlaa nĩyĩvovaa ; ĩndĩ ndeto ya Ngai waitũ ĩkekala tene na tene . ” — Isaia 40 : 8 .\n",
      "2020-02-18 12:34:54,399 \tHypothesis: Mbivilia yaĩtye atĩĩ : “ Na ĩtomo ya kĩthekanĩ , na ndeto ya Ngai , ĩndĩ ndeto sitũ syoombiwe tene na tene . ” — Isaia 40 : 8 .\n",
      "2020-02-18 12:34:54,399 Example #3\n",
      "2020-02-18 12:34:54,399 \tSource:     It includes links to lists of Witnesses currently imprisoned for their faith .\n",
      "2020-02-18 12:34:54,399 \tReference:  7 : 12 ) Walika Kĩsesenĩ kya jw.org no wone masyĩtwa ma Ngũsĩ ila syovetwe .\n",
      "2020-02-18 12:34:54,399 \tHypothesis: Amwe ma Ngũsĩ sya Yeova nĩmoovetwe nũndũ wa mũĩkĩĩo woo .\n",
      "2020-02-18 12:34:54,399 Validation result at epoch  25, step    16000: bleu:  13.26, loss: 55899.9883, ppl:   8.8517, duration: 176.5761s\n",
      "2020-02-18 12:35:35,100 Epoch  25: total training loss 1259.74\n",
      "2020-02-18 12:35:35,100 EPOCH 26\n",
      "2020-02-18 12:35:46,843 Epoch  26 Step:    16100 Batch Loss:     2.531349 Tokens per Sec:     3936, Lr: 0.000300\n",
      "2020-02-18 12:36:38,918 Epoch  26 Step:    16200 Batch Loss:     2.077816 Tokens per Sec:     3993, Lr: 0.000300\n",
      "2020-02-18 12:37:31,727 Epoch  26 Step:    16300 Batch Loss:     2.233574 Tokens per Sec:     4057, Lr: 0.000300\n",
      "2020-02-18 12:38:24,004 Epoch  26 Step:    16400 Batch Loss:     2.120835 Tokens per Sec:     4038, Lr: 0.000300\n",
      "2020-02-18 12:39:16,607 Epoch  26 Step:    16500 Batch Loss:     2.227366 Tokens per Sec:     4039, Lr: 0.000300\n",
      "2020-02-18 12:40:08,924 Epoch  26 Step:    16600 Batch Loss:     2.035489 Tokens per Sec:     4005, Lr: 0.000300\n",
      "2020-02-18 12:40:51,625 Epoch  26: total training loss 1253.41\n",
      "2020-02-18 12:40:51,626 EPOCH 27\n",
      "2020-02-18 12:41:01,892 Epoch  27 Step:    16700 Batch Loss:     2.173133 Tokens per Sec:     4219, Lr: 0.000300\n",
      "2020-02-18 12:41:54,053 Epoch  27 Step:    16800 Batch Loss:     2.505317 Tokens per Sec:     3932, Lr: 0.000300\n",
      "2020-02-18 12:42:46,713 Epoch  27 Step:    16900 Batch Loss:     2.225759 Tokens per Sec:     4014, Lr: 0.000300\n",
      "2020-02-18 12:43:40,957 Epoch  27 Step:    17000 Batch Loss:     2.229167 Tokens per Sec:     3863, Lr: 0.000300\n",
      "2020-02-18 12:46:37,859 Hooray! New best validation result [ppl]!\n",
      "2020-02-18 12:46:37,860 Saving new checkpoint.\n",
      "2020-02-18 12:46:38,088 Example #0\n",
      "2020-02-18 12:46:38,089 \tSource:     Gideon wondered how it would be possible for him to “ save Israel out of Midian’s hand . ”\n",
      "2020-02-18 12:46:38,089 \tReference:  Mũlaĩka ũsu nĩwaneenie vandũ va Mũmbi na amũĩkĩĩthya Ngiteoni kana Yeova aĩ vamwe nake .\n",
      "2020-02-18 12:46:38,089 \tHypothesis: Kamwana kau aĩ atonya kwĩkũlya - ĩ , “ Isilaeli ma Isilaeli mamwona . ”\n",
      "2020-02-18 12:46:38,089 Example #1\n",
      "2020-02-18 12:46:38,089 \tSource:     And then she would try to return to Hosea .\n",
      "2020-02-18 12:46:38,089 \tReference:  Na ĩndĩ kyamina kũsembany’a na endwa makyo Yeova aĩtye kĩkatata kũmũsyokea Osea .\n",
      "2020-02-18 12:46:38,089 \tHypothesis: Na ĩndĩ nĩwasyokie Avakuki ambĩĩa kũsyokea Hosua .\n",
      "2020-02-18 12:46:38,089 Example #2\n",
      "2020-02-18 12:46:38,090 \tSource:     The Bible itself says : “ The green grass dries up , the blossom withers , but the word of our God endures forever . ” ​ — Isaiah 40 : 8 .\n",
      "2020-02-18 12:46:38,090 \tReference:  Mbivilia yaĩtye atĩĩ : “ Nyeki nĩyũmaa , na ĩlaa nĩyĩvovaa ; ĩndĩ ndeto ya Ngai waitũ ĩkekala tene na tene . ” — Isaia 40 : 8 .\n",
      "2020-02-18 12:46:38,090 \tHypothesis: Mbivilia yaĩtye atĩĩ : “ Na mũisyo ũla mũnene , na ndeto ya Ngai , ĩndĩ ndeto ya Ngai yĩkalĩte tene na tene . ” — Isaia 40 : 8 .\n",
      "2020-02-18 12:46:38,090 Example #3\n",
      "2020-02-18 12:46:38,090 \tSource:     It includes links to lists of Witnesses currently imprisoned for their faith .\n",
      "2020-02-18 12:46:38,090 \tReference:  7 : 12 ) Walika Kĩsesenĩ kya jw.org no wone masyĩtwa ma Ngũsĩ ila syovetwe .\n",
      "2020-02-18 12:46:38,090 \tHypothesis: Amwe ma Ngũsĩ sya Yeova nĩmakwatawa nĩ mũĩkĩĩo woo .\n",
      "2020-02-18 12:46:38,090 Validation result at epoch  27, step    17000: bleu:  13.88, loss: 55588.4023, ppl:   8.7448, duration: 177.1328s\n",
      "2020-02-18 12:47:30,337 Epoch  27 Step:    17100 Batch Loss:     1.714485 Tokens per Sec:     4040, Lr: 0.000300\n",
      "2020-02-18 12:48:23,243 Epoch  27 Step:    17200 Batch Loss:     2.245825 Tokens per Sec:     4060, Lr: 0.000300\n",
      "2020-02-18 12:49:07,145 Epoch  27: total training loss 1233.83\n",
      "2020-02-18 12:49:07,145 EPOCH 28\n",
      "2020-02-18 12:49:15,454 Epoch  28 Step:    17300 Batch Loss:     1.847424 Tokens per Sec:     4011, Lr: 0.000300\n",
      "2020-02-18 12:50:06,472 Epoch  28 Step:    17400 Batch Loss:     1.910769 Tokens per Sec:     4098, Lr: 0.000300\n",
      "2020-02-18 12:50:58,780 Epoch  28 Step:    17500 Batch Loss:     1.850940 Tokens per Sec:     4009, Lr: 0.000300\n",
      "2020-02-18 12:51:51,479 Epoch  28 Step:    17600 Batch Loss:     2.364138 Tokens per Sec:     4093, Lr: 0.000300\n",
      "2020-02-18 12:52:43,614 Epoch  28 Step:    17700 Batch Loss:     1.836850 Tokens per Sec:     3943, Lr: 0.000300\n",
      "2020-02-18 12:53:36,114 Epoch  28 Step:    17800 Batch Loss:     2.066530 Tokens per Sec:     4031, Lr: 0.000300\n",
      "2020-02-18 12:54:22,666 Epoch  28: total training loss 1225.48\n",
      "2020-02-18 12:54:22,667 Training ended after  28 epochs.\n",
      "2020-02-18 12:54:22,667 Best validation result at step    17000:   8.74 ppl.\n",
      "2020-02-18 12:56:12,760  dev bleu:  14.83 [Beam search decoding with beam size = 5 and alpha = 1.0]\n",
      "2020-02-18 12:56:12,761 Translations saved to: models/enkam_transformer2/00017000.hyps.dev\n",
      "2020-02-18 12:59:34,356 test bleu:  24.96 [Beam search decoding with beam size = 5 and alpha = 1.0]\n",
      "2020-02-18 12:59:34,358 Translations saved to: models/enkam_transformer2/00017000.hyps.test\n"
     ]
    }
   ],
   "source": [
    "# Train the model\n",
    "# You can press Ctrl-C to stop. And then run the next cell to save your checkpoints! \n",
    "! cd ../../joeynmt; python3 -m joeynmt train configs/transformer_$src$tgt.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "collapsed": true,
    "id": "MBoDS09JM807"
   },
   "outputs": [],
   "source": [
    "# Copy the created models from the notebook storage to google drive for persistant storage \n",
    "!cp -r joeynmt/models/${src}${tgt}_transformer/* \"$gdrive_path/models/${src}${tgt}_transformer/\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "n94wlrCjVc17"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Steps: 2000\tLoss: 87591.90625\tPPL: 30.47441\tbleu: 2.31865\tLR: 0.00030000\t*\r\n",
      "Steps: 3000\tLoss: 79091.39844\tPPL: 21.87380\tbleu: 3.99645\tLR: 0.00030000\t*\r\n",
      "Steps: 4000\tLoss: 74410.06250\tPPL: 18.22282\tbleu: 5.92044\tLR: 0.00030000\t*\r\n",
      "Steps: 5000\tLoss: 70467.19531\tPPL: 15.62491\tbleu: 7.58122\tLR: 0.00030000\t*\r\n",
      "Steps: 6000\tLoss: 67483.05469\tPPL: 13.90791\tbleu: 8.53780\tLR: 0.00030000\t*\r\n",
      "Steps: 7000\tLoss: 65314.23047\tPPL: 12.77965\tbleu: 8.82416\tLR: 0.00030000\t*\r\n",
      "Steps: 8000\tLoss: 63758.06250\tPPL: 12.02694\tbleu: 9.82514\tLR: 0.00030000\t*\r\n",
      "Steps: 9000\tLoss: 62237.86719\tPPL: 11.33446\tbleu: 9.72829\tLR: 0.00030000\t*\r\n",
      "Steps: 10000\tLoss: 61609.98438\tPPL: 11.06021\tbleu: 10.81460\tLR: 0.00030000\t*\r\n",
      "Steps: 11000\tLoss: 61798.12109\tPPL: 11.14168\tbleu: 9.78319\tLR: 0.00030000\t\r\n",
      "Steps: 12000\tLoss: 59236.76172\tPPL: 10.08225\tbleu: 11.68094\tLR: 0.00030000\t*\r\n",
      "Steps: 13000\tLoss: 58031.44531\tPPL: 9.61917\tbleu: 11.99887\tLR: 0.00030000\t*\r\n",
      "Steps: 14000\tLoss: 57422.17578\tPPL: 9.39325\tbleu: 12.77454\tLR: 0.00030000\t*\r\n",
      "Steps: 15000\tLoss: 56712.74609\tPPL: 9.13686\tbleu: 12.61578\tLR: 0.00030000\t*\r\n",
      "Steps: 16000\tLoss: 55899.98828\tPPL: 8.85172\tbleu: 13.26186\tLR: 0.00030000\t*\r\n",
      "Steps: 17000\tLoss: 55588.40234\tPPL: 8.74478\tbleu: 13.87658\tLR: 0.00030000\t*\r\n"
     ]
    }
   ],
   "source": [
    "# Output our validation accuracy\n",
    "! cat \"../../joeynmt/models/${src}${tgt}_transformer2/validations.txt\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "66WhRE9lIhoD"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/home/espoir_mur_gmail_com/.local/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_qint8 = np.dtype([(\"qint8\", np.int8, 1)])\n",
      "/home/espoir_mur_gmail_com/.local/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_quint8 = np.dtype([(\"quint8\", np.uint8, 1)])\n",
      "/home/espoir_mur_gmail_com/.local/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_qint16 = np.dtype([(\"qint16\", np.int16, 1)])\n",
      "/home/espoir_mur_gmail_com/.local/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_quint16 = np.dtype([(\"quint16\", np.uint16, 1)])\n",
      "/home/espoir_mur_gmail_com/.local/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_qint32 = np.dtype([(\"qint32\", np.int32, 1)])\n",
      "/home/espoir_mur_gmail_com/.local/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  np_resource = np.dtype([(\"resource\", np.ubyte, 1)])\n",
      "2020-02-18 13:20:29,508 -  dev bleu:  14.83 [Beam search decoding with beam size = 5 and alpha = 1.0]\n",
      "2020-02-18 13:23:55,195 - test bleu:  24.96 [Beam search decoding with beam size = 5 and alpha = 1.0]\n"
     ]
    }
   ],
   "source": [
    "# Test our model\n",
    "! cd ../../joeynmt; python3 -m joeynmt test \"models/${src}${tgt}_transformer2/config.yaml\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": []
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "starter_notebook.ipynb",
   "provenance": [],
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}