{ "nbformat": 4, "nbformat_minor": 0, "metadata": { "accelerator": "GPU", "colab": { "name": "entn.ipynb", "provenance": [], "collapsed_sections": [], "toc_visible": true }, "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.5.6" } }, "cells": [ { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "Igc5itf-xMGj" }, "source": [ "# Masakhane - Machine Translation for African Languages (Using JoeyNMT)" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "x4fXCKCf36IK" }, "source": [ "## Note before beginning:\n", "### - The idea is that you should be able to make minimal changes to this in order to get SOME result for your own translation corpus. \n", "\n", "### - The tl;dr: Go to the **\"TODO\"** comments which will tell you what to update to get up and running\n", "\n", "### - If you actually want to have a clue what you're doing, read the text and peek at the links\n", "\n", "### - With 100 epochs, it should take around 7 hours to run in Google Colab\n", "\n", "### - Once you've gotten a result for your language, please attach and email your notebook that generated it to masakhanetranslation@gmail.com\n", "\n", "### - If you care enough and get a chance, doing a brief background on your language would be amazing. See examples in [(Martinus, 2019)](https://arxiv.org/abs/1906.05685)" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "l929HimrxS0a" }, "source": [ "## Retrieve your data & make a parallel corpus\n", "\n", "If you are wanting to use the JW300 data referenced on the Masakhane website or in our GitHub repo, you can use `opus-tools` to convert the data into a convenient format. `opus_read` from that package provides a convenient tool for reading the native aligned XML files and to convert them to TMX format. The tool can also be used to fetch relevant files from OPUS on the fly and to filter the data as necessary. [Read the documentation](https://pypi.org/project/opustools-pkg/) for more details.\n", "\n", "Once you have your corpus files in TMX format (an xml structure which will include the sentences in your target language and your source language in a single file), we recommend reading them into a pandas dataframe. Thankfully, Jade wrote a silly `tmx2dataframe` package which converts your tmx file to a pandas dataframe. " ] }, { "cell_type": "code", "metadata": { "colab_type": "code", "id": "oGRmDELn7Az0", "outputId": "430ae782-ad31-4a62-887a-95be3c1331fa", "colab": { "base_uri": "https://localhost:8080/", "height": 121 } }, "source": [ "from google.colab import drive\n", "drive.mount('/content/drive')" ], "execution_count": 1, "outputs": [ { "output_type": "stream", "text": [ "Go to this URL in a browser: https://accounts.google.com/o/oauth2/auth?client_id=947318989803-6bn6qk8qdgf4n4g3pfee6491hc0brc4i.apps.googleusercontent.com&redirect_uri=urn%3aietf%3awg%3aoauth%3a2.0%3aoob&response_type=code&scope=email%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdocs.test%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdrive%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdrive.photos.readonly%20https%3a%2f%2fwww.googleapis.com%2fauth%2fpeopleapi.readonly\n", "\n", "Enter your authorization code:\n", "··········\n", "Mounted at /content/drive\n" ], "name": "stdout" } ] }, { "cell_type": "code", "metadata": { "colab_type": "code", "id": "Cn3tgQLzUxwn", "colab": {} }, "source": [ "# TODO: Set your source and target languages. Keep in mind, these traditionally use language codes as found here:\n", "# These will also become the suffix's of all vocab and corpus files used throughout\n", "import os\n", "source_language = \"en\"\n", "target_language = \"tn\" \n", "lc = False # If True, lowercase the data.\n", "seed = 42 # Random seed for shuffling.\n", "tag = \"baseline\" # Give a unique name to your folder - this is to ensure you don't rewrite any models you've already submitted\n", "\n", "os.environ[\"src\"] = source_language # Sets them in bash as well, since we often use bash scripts\n", "os.environ[\"tgt\"] = target_language\n", "os.environ[\"tag\"] = tag\n", "\n", "# This will save it to a folder in our gdrive instead!\n", "!mkdir -p \"/content/drive/My Drive/masakhane/$src-$tgt-$tag\"\n", "os.environ[\"gdrive_path\"] = \"/content/drive/My Drive/masakhane/%s-%s-%s\" % (source_language, target_language, tag)" ], "execution_count": 0, "outputs": [] }, { "cell_type": "code", "metadata": { "colab_type": "code", "id": "kBSgJHEw7Nvx", "outputId": "14ae3279-8980-4a31-b722-0c4b91aefdf4", "colab": { "base_uri": "https://localhost:8080/", "height": 34 } }, "source": [ "!echo $gdrive_path" ], "execution_count": 3, "outputs": [ { "output_type": "stream", "text": [ "/content/drive/My Drive/masakhane/en-tn-baseline\n" ], "name": "stdout" } ] }, { "cell_type": "code", "metadata": { "colab_type": "code", "id": "gA75Fs9ys8Y9", "outputId": "e400f929-f943-44e9-cd59-e5e166d52d4a", "colab": { "base_uri": "https://localhost:8080/", "height": 121 } }, "source": [ "# Install opus-tools\n", "! pip install opustools-pkg" ], "execution_count": 4, "outputs": [ { "output_type": "stream", "text": [ "Collecting opustools-pkg\n", "\u001b[?25l Downloading https://files.pythonhosted.org/packages/6c/9f/e829a0cceccc603450cd18e1ff80807b6237a88d9a8df2c0bb320796e900/opustools_pkg-0.0.52-py3-none-any.whl (80kB)\n", "\r\u001b[K |████ | 10kB 18.5MB/s eta 0:00:01\r\u001b[K |████████ | 20kB 4.5MB/s eta 0:00:01\r\u001b[K |████████████▏ | 30kB 6.4MB/s eta 0:00:01\r\u001b[K |████████████████▏ | 40kB 8.1MB/s eta 0:00:01\r\u001b[K |████████████████████▎ | 51kB 5.2MB/s eta 0:00:01\r\u001b[K |████████████████████████▎ | 61kB 6.1MB/s eta 0:00:01\r\u001b[K |████████████████████████████▎ | 71kB 6.9MB/s eta 0:00:01\r\u001b[K |████████████████████████████████| 81kB 4.7MB/s \n", "\u001b[?25hInstalling collected packages: opustools-pkg\n", "Successfully installed opustools-pkg-0.0.52\n" ], "name": "stdout" } ] }, { "cell_type": "code", "metadata": { "colab_type": "code", "id": "xq-tDZVks7ZD", "colab": {} }, "source": [ "# Downloading our corpus\n", "! opus_read -d JW300 -s $src -t $tgt -wm moses -w jw300.$src jw300.$tgt -q\n", "\n", "# extract the corpus file\n", "! gunzip JW300_latest_xml_$src-$tgt.xml.gz" ], "execution_count": 0, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "n48GDRnP8y2G", "colab_type": "code", "outputId": "a35e99d1-3d1b-4665-e651-06e304285581", "colab": { "base_uri": "https://localhost:8080/", "height": 571 } }, "source": [ "# Download the global test set.\n", "! wget https://raw.githubusercontent.com/juliakreutzer/masakhane/master/jw300_utils/test/test.en-any.en\n", " \n", "# And the specific test set for this language pair.\n", "os.environ[\"trg\"] = target_language \n", "os.environ[\"src\"] = source_language \n", "\n", "! wget https://raw.githubusercontent.com/juliakreutzer/masakhane/master/jw300_utils/test/test.en-$trg.en \n", "! mv test.en-$trg.en test.en\n", "! wget https://raw.githubusercontent.com/juliakreutzer/masakhane/master/jw300_utils/test/test.en-$trg.$trg \n", "! mv test.en-$trg.$trg test.$trg" ], "execution_count": 5, "outputs": [ { "output_type": "stream", "text": [ "--2019-11-27 08:01:30-- https://raw.githubusercontent.com/juliakreutzer/masakhane/master/jw300_utils/test/test.en-any.en\n", "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.0.133, 151.101.64.133, 151.101.128.133, ...\n", "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.0.133|:443... connected.\n", "HTTP request sent, awaiting response... 200 OK\n", "Length: 277791 (271K) [text/plain]\n", "Saving to: ‘test.en-any.en’\n", "\n", "\rtest.en-any.en 0%[ ] 0 --.-KB/s \rtest.en-any.en 100%[===================>] 271.28K --.-KB/s in 0.02s \n", "\n", "2019-11-27 08:01:30 (12.8 MB/s) - ‘test.en-any.en’ saved [277791/277791]\n", "\n", "--2019-11-27 08:01:33-- https://raw.githubusercontent.com/juliakreutzer/masakhane/master/jw300_utils/test/test.en-tn.en\n", "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.0.133, 151.101.64.133, 151.101.128.133, ...\n", "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.0.133|:443... connected.\n", "HTTP request sent, awaiting response... 200 OK\n", "Length: 206185 (201K) [text/plain]\n", "Saving to: ‘test.en-tn.en’\n", "\n", "test.en-tn.en 100%[===================>] 201.35K --.-KB/s in 0.02s \n", "\n", "2019-11-27 08:01:33 (12.0 MB/s) - ‘test.en-tn.en’ saved [206185/206185]\n", "\n", "--2019-11-27 08:01:38-- https://raw.githubusercontent.com/juliakreutzer/masakhane/master/jw300_utils/test/test.en-tn.tn\n", "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.0.133, 151.101.64.133, 151.101.128.133, ...\n", "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.0.133|:443... connected.\n", "HTTP request sent, awaiting response... 200 OK\n", "Length: 238522 (233K) [text/plain]\n", "Saving to: ‘test.en-tn.tn’\n", "\n", "test.en-tn.tn 100%[===================>] 232.93K --.-KB/s in 0.02s \n", "\n", "2019-11-27 08:01:39 (10.3 MB/s) - ‘test.en-tn.tn’ saved [238522/238522]\n", "\n" ], "name": "stdout" } ] }, { "cell_type": "code", "metadata": { "id": "NqDG-CI28y2L", "colab_type": "code", "outputId": "7e465a00-b8d3-4e8d-8f33-398cf5c83183", "colab": { "base_uri": "https://localhost:8080/", "height": 34 } }, "source": [ "# Read the test data to filter from train and dev splits.\n", "# Store english portion in set for quick filtering checks.\n", "en_test_sents = set()\n", "filter_test_sents = \"test.en-any.en\"\n", "j = 0\n", "with open(filter_test_sents) as f:\n", " for line in f:\n", " en_test_sents.add(line.strip())\n", " j += 1\n", "print('Loaded {} global test sentences to filter from the training/dev data.'.format(j))" ], "execution_count": 6, "outputs": [ { "output_type": "stream", "text": [ "Loaded 3571 global test sentences to filter from the training/dev data.\n" ], "name": "stdout" } ] }, { "cell_type": "code", "metadata": { "colab_type": "code", "id": "3CNdwLBCfSIl", "outputId": "ac71d45d-133d-4029-eac2-5602cbb2aa43", "colab": { "base_uri": "https://localhost:8080/", "height": 153 } }, "source": [ "import pandas as pd\n", "\n", "# TMX file to dataframe\n", "source_file = 'entn_parallel.train.' + source_language\n", "target_file = 'entn_parallel.train.' + target_language\n", "\n", "source = []\n", "target = []\n", "skip_lines = [] # Collect the line numbers of the source portion to skip the same lines for the target portion.\n", "with open(source_file) as f:\n", " for i, line in enumerate(f):\n", " # Skip sentences that are contained in the test set.\n", " if line.strip() not in en_test_sents:\n", " source.append(line.strip())\n", " else:\n", " skip_lines.append(i) \n", "with open(target_file) as f:\n", " for j, line in enumerate(f):\n", " # Only add to corpus if corresponding source was not skipped.\n", " if j not in skip_lines:\n", " target.append(line.strip())\n", " \n", "print('Loaded data and skipped {}/{} lines since contained in test set.'.format(len(skip_lines), i))\n", " \n", "df = pd.DataFrame(zip(source, target), columns=['source_sentence', 'target_sentence'])\n", "# if you get TypeError: data argument can't be an iterator is because of your zip version run this below\n", "#df = pd.DataFrame(list(zip(source, target)), columns=['source_sentence', 'target_sentence'])\n", "df.head(3)" ], "execution_count": 7, "outputs": [ { "output_type": "stream", "text": [ "Loaded data and skipped 0/120867 lines since contained in test set.\n" ], "name": "stdout" }, { "output_type": "execute_result", "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
source_sentencetarget_sentence
0an interpreter is secured .Go tla batlwa moranodi
1Middle AgesMetlhagare
210 Policy manages media liaison10 Pholisi e laola puisano le metswedi ya dikgang
\n", "
" ], "text/plain": [ " source_sentence target_sentence\n", "0 an interpreter is secured . Go tla batlwa moranodi\n", "1 Middle Ages Metlhagare\n", "2 10 Policy manages media liaison 10 Pholisi e laola puisano le metswedi ya dikgang" ] }, "metadata": { "tags": [] }, "execution_count": 7 } ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "YkuK3B4p2AkN" }, "source": [ "## Pre-processing and export\n", "\n", "It is generally a good idea to remove duplicate translations and conflicting translations from the corpus. In practice, these public corpora include some number of these that need to be cleaned.\n", "\n", "In addition we will split our data into dev/test/train and export to the filesystem." ] }, { "cell_type": "code", "metadata": { "colab_type": "code", "id": "M_2ouEOH1_1q", "outputId": "97c60a58-e80b-492b-8c36-b58351e058f3", "colab": { "base_uri": "https://localhost:8080/", "height": 185 } }, "source": [ "# drop duplicate translations\n", "df_pp = df.drop_duplicates()\n", "\n", "# drop conflicting translations\n", "# (this is optional and something that you might want to comment out \n", "# depending on the size of your corpus)\n", "df_pp.drop_duplicates(subset='source_sentence', inplace=True)\n", "df_pp.drop_duplicates(subset='target_sentence', inplace=True)\n", "\n", "# Shuffle the data to remove bias in dev set selection.\n", "df_pp = df_pp.sample(frac=1, random_state=seed).reset_index(drop=True)" ], "execution_count": 8, "outputs": [ { "output_type": "stream", "text": [ "/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:6: SettingWithCopyWarning: \n", "A value is trying to be set on a copy of a slice from a DataFrame\n", "\n", "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n", " \n", "/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:7: SettingWithCopyWarning: \n", "A value is trying to be set on a copy of a slice from a DataFrame\n", "\n", "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n", " import sys\n" ], "name": "stderr" } ] }, { "cell_type": "code", "metadata": { "id": "Z_1BwAApEtMk", "colab_type": "code", "outputId": "c211c52a-3fbe-464c-9867-a73b4f6ab6f4", "colab": { "base_uri": "https://localhost:8080/", "height": 1000 } }, "source": [ "# Install fuzzy wuzzy to remove \"almost duplicate\" sentences in the\n", "# test and training sets.\n", "! pip install fuzzywuzzy\n", "! pip install python-Levenshtein\n", "import time\n", "from fuzzywuzzy import process\n", "import numpy as np\n", "\n", "# reset the index of the training set after previous filtering\n", "df_pp.reset_index(drop=False, inplace=True)\n", "\n", "# Remove samples from the training data set if they \"almost overlap\" with the\n", "# samples in the test set.\n", "\n", "# Filtering function. Adjust pad to narrow down the candidate matches to\n", "# within a certain length of characters of the given sample.\n", "def fuzzfilter(sample, candidates, pad):\n", " candidates = [x for x in candidates if len(x) <= len(sample)+pad and len(x) >= len(sample)-pad] \n", " if len(candidates) > 0:\n", " return process.extractOne(sample, candidates)[1]\n", " else:\n", " return np.nan\n", "\n", "# NOTE - This might run slow depending on the size of your training set. We are\n", "# printing some information to help you track how long it would take. \n", "scores = []\n", "start_time = time.time()\n", "for idx, row in df_pp.iterrows():\n", " scores.append(fuzzfilter(row['source_sentence'], list(en_test_sents), 5))\n", " if idx % 1000 == 0:\n", " hours, rem = divmod(time.time() - start_time, 3600)\n", " minutes, seconds = divmod(rem, 60)\n", " print(\"{:0>2}:{:0>2}:{:05.2f}\".format(int(hours),int(minutes),seconds), \"%0.2f percent complete\" % (100.0*float(idx)/float(len(df_pp))))\n", "\n", "# Filter out \"almost overlapping samples\"\n", "df_pp['scores'] = scores\n", "df_pp = df_pp[df_pp['scores'] < 95]" ], "execution_count": 9, "outputs": [ { "output_type": "stream", "text": [ "Collecting fuzzywuzzy\n", " Downloading https://files.pythonhosted.org/packages/d8/f1/5a267addb30ab7eaa1beab2b9323073815da4551076554ecc890a3595ec9/fuzzywuzzy-0.17.0-py2.py3-none-any.whl\n", "Installing collected packages: fuzzywuzzy\n", "Successfully installed fuzzywuzzy-0.17.0\n", "Collecting python-Levenshtein\n", "\u001b[?25l Downloading https://files.pythonhosted.org/packages/42/a9/d1785c85ebf9b7dfacd08938dd028209c34a0ea3b1bcdb895208bd40a67d/python-Levenshtein-0.12.0.tar.gz (48kB)\n", "\u001b[K |████████████████████████████████| 51kB 3.1MB/s \n", "\u001b[?25hRequirement already satisfied: setuptools in /usr/local/lib/python3.6/dist-packages (from python-Levenshtein) (41.6.0)\n", "Building wheels for collected packages: python-Levenshtein\n", " Building wheel for python-Levenshtein (setup.py) ... \u001b[?25l\u001b[?25hdone\n", " Created wheel for python-Levenshtein: filename=python_Levenshtein-0.12.0-cp36-cp36m-linux_x86_64.whl size=144669 sha256=2a1cafec808ec853872529c611c1af1a1a04b3f0a55f48cb359d4bfa1bf966e4\n", " Stored in directory: /root/.cache/pip/wheels/de/c2/93/660fd5f7559049268ad2dc6d81c4e39e9e36518766eaf7e342\n", "Successfully built python-Levenshtein\n", "Installing collected packages: python-Levenshtein\n", "Successfully installed python-Levenshtein-0.12.0\n", "00:00:00.03 0.00 percent complete\n", "00:00:21.31 0.84 percent complete\n", "00:00:42.78 1.68 percent complete\n", "00:01:03.78 2.52 percent complete\n", "00:01:24.93 3.36 percent complete\n", "00:01:47.88 4.20 percent complete\n", "00:02:08.26 5.03 percent complete\n", "00:02:29.12 5.87 percent complete\n", "00:02:50.11 6.71 percent complete\n", "00:03:12.12 7.55 percent complete\n", "00:03:33.40 8.39 percent complete\n", "00:03:54.84 9.23 percent complete\n", "00:04:15.38 10.07 percent complete\n", "00:04:37.06 10.91 percent complete\n", "00:04:58.46 11.75 percent complete\n", "00:05:19.37 12.59 percent complete\n", "00:05:39.97 13.42 percent complete\n", "00:06:01.19 14.26 percent complete\n", "00:06:21.83 15.10 percent complete\n", "00:06:43.21 15.94 percent complete\n", "00:07:03.76 16.78 percent complete\n", "00:07:24.92 17.62 percent complete\n", "00:07:46.44 18.46 percent complete\n", "00:08:07.82 19.30 percent complete\n", "00:08:28.17 20.14 percent complete\n", "00:08:49.01 20.98 percent complete\n", "00:09:10.69 21.82 percent complete\n", "00:09:32.61 22.65 percent complete\n", "00:09:53.07 23.49 percent complete\n", "00:10:14.10 24.33 percent complete\n", "00:10:36.51 25.17 percent complete\n", "00:10:56.97 26.01 percent complete\n", "00:11:17.98 26.85 percent complete\n", "00:11:38.18 27.69 percent complete\n", "00:11:59.68 28.53 percent complete\n", "00:12:20.12 29.37 percent complete\n", "00:12:41.30 30.21 percent complete\n", "00:13:02.24 31.04 percent complete\n", "00:13:24.16 31.88 percent complete\n", "00:13:45.47 32.72 percent complete\n", "00:14:06.43 33.56 percent complete\n", "00:14:26.71 34.40 percent complete\n", "00:14:47.96 35.24 percent complete\n", "00:15:09.71 36.08 percent complete\n", "00:15:30.75 36.92 percent complete\n", "00:15:51.57 37.76 percent complete\n", "00:16:13.62 38.60 percent complete\n", "00:16:34.00 39.43 percent complete\n", "00:16:55.40 40.27 percent complete\n", "00:17:16.18 41.11 percent complete\n", "00:17:37.02 41.95 percent complete\n", "00:17:58.96 42.79 percent complete\n", "00:18:19.92 43.63 percent complete\n", "00:18:40.92 44.47 percent complete\n", "00:19:02.28 45.31 percent complete\n", "00:19:23.42 46.15 percent complete\n", "00:19:44.38 46.99 percent complete\n", "00:20:05.75 47.83 percent complete\n", "00:20:26.79 48.66 percent complete\n", "00:20:48.00 49.50 percent complete\n", "00:21:09.57 50.34 percent complete\n", "00:21:31.60 51.18 percent complete\n", "00:21:53.22 52.02 percent complete\n", "00:22:14.65 52.86 percent complete\n", "00:22:35.09 53.70 percent complete\n", "00:22:56.17 54.54 percent complete\n", "00:23:17.21 55.38 percent complete\n", "00:23:38.61 56.22 percent complete\n", "00:23:59.61 57.05 percent complete\n", "00:24:21.19 57.89 percent complete\n", "00:24:44.03 58.73 percent complete\n", "00:25:05.71 59.57 percent complete\n", "00:25:27.04 60.41 percent complete\n", "00:25:48.84 61.25 percent complete\n", "00:26:10.69 62.09 percent complete\n", "00:26:32.50 62.93 percent complete\n", "00:26:53.30 63.77 percent complete\n", "00:27:14.23 64.61 percent complete\n", "00:27:36.45 65.45 percent complete\n", "00:27:57.16 66.28 percent complete\n", "00:28:17.69 67.12 percent complete\n", "00:28:38.72 67.96 percent complete\n", "00:28:59.84 68.80 percent complete\n", "00:29:20.92 69.64 percent complete\n", "00:29:42.99 70.48 percent complete\n", "00:30:03.91 71.32 percent complete\n", "00:30:25.60 72.16 percent complete\n", "00:30:47.54 73.00 percent complete\n", "00:31:09.70 73.84 percent complete\n", "00:31:30.26 74.67 percent complete\n", "00:31:51.84 75.51 percent complete\n", "00:32:13.13 76.35 percent complete\n", "00:32:34.68 77.19 percent complete\n", "00:32:55.93 78.03 percent complete\n", "00:33:16.85 78.87 percent complete\n", "00:33:39.54 79.71 percent complete\n", "00:34:00.61 80.55 percent complete\n", "00:34:21.65 81.39 percent complete\n", "00:34:41.67 82.23 percent complete\n", "00:35:03.15 83.06 percent complete\n", "00:35:24.74 83.90 percent complete\n", "00:35:45.65 84.74 percent complete\n", "00:36:07.63 85.58 percent complete\n", "00:36:29.40 86.42 percent complete\n", "00:36:51.07 87.26 percent complete\n", "00:37:12.08 88.10 percent complete\n", "00:37:33.99 88.94 percent complete\n", "00:37:55.16 89.78 percent complete\n", "00:38:16.40 90.62 percent complete\n", "00:38:37.20 91.46 percent complete\n", "00:38:58.52 92.29 percent complete\n", "00:39:20.81 93.13 percent complete\n", "00:39:42.25 93.97 percent complete\n", "00:40:03.25 94.81 percent complete\n", "00:40:23.97 95.65 percent complete\n", "00:40:44.58 96.49 percent complete\n", "00:41:04.61 97.33 percent complete\n", "00:41:25.49 98.17 percent complete\n", "00:41:46.76 99.01 percent complete\n", "00:42:08.75 99.85 percent complete\n" ], "name": "stdout" } ] }, { "cell_type": "code", "metadata": { "colab_type": "code", "id": "hxxBOCA-xXhy", "outputId": "2f0f77d4-069f-4110-b6bd-e6dc08133591", "colab": { "base_uri": "https://localhost:8080/", "height": 810 } }, "source": [ "# This section does the split between train/dev for the parallel corpora then saves them as separate files\n", "# We use 1000 dev test and the given test set.\n", "import csv\n", "\n", "# Do the split between dev/train and create parallel corpora\n", "num_dev_patterns = 1000\n", "\n", "# Optional: lower case the corpora - this will make it easier to generalize, but without proper casing.\n", "if lc: # Julia: making lowercasing optional\n", " df_pp[\"source_sentence\"] = df_pp[\"source_sentence\"].str.lower()\n", " df_pp[\"target_sentence\"] = df_pp[\"target_sentence\"].str.lower()\n", "\n", "# Julia: test sets are already generated\n", "dev = df_pp.tail(num_dev_patterns) # Herman: Error in original\n", "stripped = df_pp.drop(df_pp.tail(num_dev_patterns).index)\n", "\n", "with open(\"train.\"+source_language, \"w\") as src_file, open(\"train.\"+target_language, \"w\") as trg_file:\n", " for index, row in stripped.iterrows():\n", " src_file.write(row[\"source_sentence\"]+\"\\n\")\n", " trg_file.write(row[\"target_sentence\"]+\"\\n\")\n", " \n", "with open(\"dev.\"+source_language, \"w\") as src_file, open(\"dev.\"+target_language, \"w\") as trg_file:\n", " for index, row in dev.iterrows():\n", " src_file.write(row[\"source_sentence\"]+\"\\n\")\n", " trg_file.write(row[\"target_sentence\"]+\"\\n\")\n", "\n", "#stripped[[\"source_sentence\"]].to_csv(\"train.\"+source_language, header=False, index=False) # Herman: Added `header=False` everywhere\n", "#stripped[[\"target_sentence\"]].to_csv(\"train.\"+target_language, header=False, index=False) # Julia: Problematic handling of quotation marks.\n", "\n", "#dev[[\"source_sentence\"]].to_csv(\"dev.\"+source_language, header=False, index=False)\n", "#dev[[\"target_sentence\"]].to_csv(\"dev.\"+target_language, header=False, index=False)\n", "\n", "# Doublecheck the format below. There should be no extra quotation marks or weird characters.\n", "! head train.*\n", "! head dev.*" ], "execution_count": 10, "outputs": [ { "output_type": "stream", "text": [ "==> train.en <==\n", "Elsewhere Mphahlele has said of exile that ' Exile is a ghetto of the mind ' and famously referred to the ' tyranny of place ' to describe the oppressive emotions he felt at times as an exile , confronted with foreign countries , with their alien culture , values and ways of living .\n", "The letter should be accompanied by the prescribed fee which will be determined by the council .\n", "You must improve your management - planning , organising , implementation and control .\n", "Next in line is Suzelle Buys , an administrative officer from Graduation Ceremonies at the Organisation and Distribution Centre on the Potchefstroom Campus .\n", "When was this going to be rectified or changed ?\n", "2 . In Marxist ideology , the class that owns the means of production .\n", "Do you plan to sell the crop immediately or are you planning to store it and wait for a better price ?\n", "That is the story that only our poets can tell .\n", "The ward report-back forms will be important to use for all three categories of projects .\n", "Preparation for the process\n", "\n", "==> train.tn <==\n", "Go lo gongwe Mphahlele o rile ' Botshabelo ke ghetto ya tlhaloganyo ' mme a bua ka mafoko a itsegeng a ' patiko ya lefelo ' go tlhalosa maikutlo a a gatelelang a neng a ikutlwa one ka dinako dingwe kwa botshabelo , a samagane le dinaga di sele , tse di nang le setso se sele , le ditsela tsa tsone , bogolosegolo kwa dinageng tse di kopaneng sa Amerika , e e neng e le kgatlhanong le setso sa Aforika , le ditsela tse Mphahlele a neng a ikaeletse go di somarela ka gotlhegotlhe .\n", "Lekwalo le tshwanetse go paganngwa le tefo e e beilweng e e tla laolwang ke Lekgotla .\n", "O tshwanetse go tokafatsa tsamaiso ya gago - togamaano , thulaganyo , tiragatso le taolo .\n", "Morago ga moo go latela Suzelle Buys , e leng motlhankedi wa tsamaiso ya Meletlo ya Kalogo mo Setlamong le mo Bobolokelong jwa Dikumo tse di Rekisiwang jwa Khamphase ya Potchefstroom .\n", "Seno se ne se tlile go baakanngwa kgotsa go fetolwa leng ?\n", "2 . Ka kakanyo ya boMarxist , ke lephata le le ithuetseng dikatlanegiso tsa ntshetsoya moruo .\n", "A o rulaganya go rekisa dijalo ka bonako kgotsa a o rulaganya go di boloka le go emela tlhotlhwa e e botoka ?\n", "Seo ke kanelo e e ka anelwang fela ke baboki ba rona .\n", "Diforomo tsa pegelo ya woto di tlaa nna botlhokwa go dirisiwa ke mefuta yotlhe e mene ya diporojeke .\n", "Paakanyetso ya thulaganyo\n", "==> dev.en <==\n", "The services are free in South Africa\n", "( 7 ) A person or community dispossessed of property after 19 June 1913 as a result of past racially discriminatory laws or practices is entitled , to the extent provided by an Act of Parliament , either to restitution of that property or to equitable redress .\n", "ruling\n", "Skills Development course .\n", "It should be noted that the perpetrator lied during the interview .\n", "Diversifying into two or more enterprises could assist in maintaining liquidity .\n", "The prompt response of Campus Protection Services ( CPS ) recently prevented a vehicle from being destroyed by fire .\n", "He says Victor is a wonderful example and his exceptional leadership skills make an enormous difference .\n", "A part of a calendar quarter is regarded as a full quarter .\n", "THE ROLE OF THE NATIONAL ASSEMBLY\n", "\n", "==> dev.tn <==\n", "Mo Aforikaborwa , ditirelo di neelwa kwa ntle ga go duelelwa .\n", "( 7 ) Motho kgotsa setlhopha sa magae se se amogilweng dithoto morago ga la 19 Seetebosigo 1913 ka ntlha ya melao ya kgale ya tlhotlhololo kgotsa ditlwaelo , se tshwanelwa , go ya ka thulaganyetso ya Molao wa Palamente go busediwa dithoto tseo , kgotsa tshiamisetso e e e kalekanang .\n", "tshwetsotaolo\n", "Khoso ya Tlhabololo ya Bokgoni .\n", "Go tshwanetse ga kwalwa gore motlhotlhi o akile ka nako ya dipotsolotso .\n", "Dirisa mefutafuta ya dilo tse pedi kgotsa go feta go go tshegetsa go nna le kgono ya theko .\n", "Go tsiboga ka bonako bošeng jaana ga Lefapha la Ditirelo tsa Tshireletso la Khamphase kgotsa Campus Protection Services ( CPS ) go ne ga namola koloi gore e se jewe ke molelo .\n", "A re Victor ke sekao se se molemo tota e bile bokgoni jo bo seng kana ka sepe jwa go etelela pele jo a nang le jone bo thusa fela thata .\n", "Karolo ya kotara ya khalentara e lejwa jaaka kotara e e feletseng .\n", "TIRO YA NTLOKOKOANO BOSETŠHABA\n" ], "name": "stdout" } ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "epeCydmCyS8X" }, "source": [ "\n", "\n", "---\n", "\n", "\n", "## Installation of JoeyNMT\n", "\n", "JoeyNMT is a simple, minimalist NMT package which is useful for learning and teaching. Check out the documentation for JoeyNMT [here](https://joeynmt.readthedocs.io) " ] }, { "cell_type": "code", "metadata": { "colab_type": "code", "id": "iBRMm4kMxZ8L", "outputId": "5aa1f92b-a720-4254-ab27-8480f3965715", "colab": { "base_uri": "https://localhost:8080/", "height": 1000 } }, "source": [ "# Install JoeyNMT\n", "! git clone https://github.com/joeynmt/joeynmt.git\n", "! cd joeynmt; pip3 install ." ], "execution_count": 11, "outputs": [ { "output_type": "stream", "text": [ "Cloning into 'joeynmt'...\n", "remote: Enumerating objects: 15, done.\u001b[K\n", "remote: Counting objects: 6% (1/15)\u001b[K\rremote: Counting objects: 13% (2/15)\u001b[K\rremote: Counting objects: 20% (3/15)\u001b[K\rremote: Counting objects: 26% (4/15)\u001b[K\rremote: Counting objects: 33% (5/15)\u001b[K\rremote: Counting objects: 40% (6/15)\u001b[K\rremote: Counting objects: 46% (7/15)\u001b[K\rremote: Counting objects: 53% (8/15)\u001b[K\rremote: Counting objects: 60% (9/15)\u001b[K\rremote: Counting objects: 66% (10/15)\u001b[K\rremote: Counting objects: 73% (11/15)\u001b[K\rremote: Counting objects: 80% (12/15)\u001b[K\rremote: Counting objects: 86% (13/15)\u001b[K\rremote: Counting objects: 93% (14/15)\u001b[K\rremote: Counting objects: 100% (15/15)\u001b[K\rremote: Counting objects: 100% (15/15), done.\u001b[K\n", "remote: Compressing objects: 100% (12/12), done.\u001b[K\n", "remote: Total 2199 (delta 4), reused 5 (delta 3), pack-reused 2184\u001b[K\n", "Receiving objects: 100% (2199/2199), 2.60 MiB | 4.29 MiB/s, done.\n", "Resolving deltas: 100% (1525/1525), done.\n", "Processing /content/joeynmt\n", "Requirement already satisfied: future in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.1) (0.16.0)\n", "Requirement already satisfied: pillow in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.1) (4.3.0)\n", "Requirement already satisfied: numpy<2.0,>=1.14.5 in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.1) (1.17.4)\n", "Requirement already satisfied: setuptools>=41.0.0 in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.1) (41.6.0)\n", "Requirement already satisfied: torch>=1.1 in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.1) (1.3.1)\n", "Requirement already satisfied: tensorflow>=1.14 in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.1) (1.15.0)\n", "Requirement already satisfied: torchtext in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.1) (0.3.1)\n", "Collecting sacrebleu>=1.3.6\n", " Downloading https://files.pythonhosted.org/packages/0e/e5/93d252182f7cbd4b59bb3ec5797e2ce33cfd6f5aadaf327db170cf4b7887/sacrebleu-1.4.2-py3-none-any.whl\n", "Collecting subword-nmt\n", " Downloading https://files.pythonhosted.org/packages/74/60/6600a7bc09e7ab38bc53a48a20d8cae49b837f93f5842a41fe513a694912/subword_nmt-0.3.7-py2.py3-none-any.whl\n", "Requirement already satisfied: matplotlib in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.1) (3.1.1)\n", "Requirement already satisfied: seaborn in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.1) (0.9.0)\n", "Collecting pyyaml>=5.1\n", "\u001b[?25l Downloading https://files.pythonhosted.org/packages/e3/e8/b3212641ee2718d556df0f23f78de8303f068fe29cdaa7a91018849582fe/PyYAML-5.1.2.tar.gz (265kB)\n", "\u001b[K |████████████████████████████████| 266kB 18.2MB/s \n", "\u001b[?25hCollecting pylint\n", "\u001b[?25l Downloading https://files.pythonhosted.org/packages/e9/59/43fc36c5ee316bb9aeb7cf5329cdbdca89e5749c34d5602753827c0aa2dc/pylint-2.4.4-py3-none-any.whl (302kB)\n", "\u001b[K |████████████████████████████████| 307kB 40.4MB/s \n", "\u001b[?25hRequirement already satisfied: six==1.12 in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.1) (1.12.0)\n", "Requirement already satisfied: olefile in /usr/local/lib/python3.6/dist-packages (from pillow->joeynmt==0.0.1) (0.46)\n", "Requirement already satisfied: keras-applications>=1.0.8 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (1.0.8)\n", "Requirement already satisfied: gast==0.2.2 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (0.2.2)\n", "Requirement already satisfied: tensorflow-estimator==1.15.1 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (1.15.1)\n", "Requirement already satisfied: grpcio>=1.8.6 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (1.15.0)\n", "Requirement already satisfied: google-pasta>=0.1.6 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (0.1.8)\n", "Requirement already satisfied: opt-einsum>=2.3.2 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (3.1.0)\n", "Requirement already satisfied: tensorboard<1.16.0,>=1.15.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (1.15.0)\n", "Requirement already satisfied: wheel>=0.26 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (0.33.6)\n", "Requirement already satisfied: wrapt>=1.11.1 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (1.11.2)\n", "Requirement already satisfied: keras-preprocessing>=1.0.5 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (1.1.0)\n", "Requirement already satisfied: absl-py>=0.7.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (0.8.1)\n", "Requirement already satisfied: astor>=0.6.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (0.8.0)\n", "Requirement already satisfied: termcolor>=1.1.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (1.1.0)\n", "Requirement already satisfied: protobuf>=3.6.1 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (3.10.0)\n", "Requirement already satisfied: tqdm in /usr/local/lib/python3.6/dist-packages (from torchtext->joeynmt==0.0.1) (4.28.1)\n", "Requirement already satisfied: requests in /usr/local/lib/python3.6/dist-packages (from torchtext->joeynmt==0.0.1) (2.21.0)\n", "Collecting portalocker\n", " Downloading https://files.pythonhosted.org/packages/91/db/7bc703c0760df726839e0699b7f78a4d8217fdc9c7fcb1b51b39c5a22a4e/portalocker-1.5.2-py2.py3-none-any.whl\n", "Requirement already satisfied: typing in /usr/local/lib/python3.6/dist-packages (from sacrebleu>=1.3.6->joeynmt==0.0.1) (3.6.6)\n", "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.6/dist-packages (from matplotlib->joeynmt==0.0.1) (0.10.0)\n", "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib->joeynmt==0.0.1) (1.1.0)\n", "Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib->joeynmt==0.0.1) (2.4.5)\n", "Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib->joeynmt==0.0.1) (2.6.1)\n", "Requirement already satisfied: scipy>=0.14.0 in /usr/local/lib/python3.6/dist-packages (from seaborn->joeynmt==0.0.1) (1.3.2)\n", "Requirement already satisfied: pandas>=0.15.2 in /usr/local/lib/python3.6/dist-packages (from seaborn->joeynmt==0.0.1) (0.25.3)\n", "Collecting astroid<2.4,>=2.3.0\n", "\u001b[?25l Downloading https://files.pythonhosted.org/packages/ad/ae/86734823047962e7b8c8529186a1ac4a7ca19aaf1aa0c7713c022ef593fd/astroid-2.3.3-py3-none-any.whl (205kB)\n", "\u001b[K |████████████████████████████████| 215kB 48.3MB/s \n", "\u001b[?25hCollecting isort<5,>=4.2.5\n", "\u001b[?25l Downloading https://files.pythonhosted.org/packages/e5/b0/c121fd1fa3419ea9bfd55c7f9c4fedfec5143208d8c7ad3ce3db6c623c21/isort-4.3.21-py2.py3-none-any.whl (42kB)\n", "\u001b[K |████████████████████████████████| 51kB 6.8MB/s \n", "\u001b[?25hCollecting mccabe<0.7,>=0.6\n", " Downloading https://files.pythonhosted.org/packages/87/89/479dc97e18549e21354893e4ee4ef36db1d237534982482c3681ee6e7b57/mccabe-0.6.1-py2.py3-none-any.whl\n", "Requirement already satisfied: h5py in /usr/local/lib/python3.6/dist-packages (from keras-applications>=1.0.8->tensorflow>=1.14->joeynmt==0.0.1) (2.8.0)\n", "Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.6/dist-packages (from tensorboard<1.16.0,>=1.15.0->tensorflow>=1.14->joeynmt==0.0.1) (3.1.1)\n", "Requirement already satisfied: werkzeug>=0.11.15 in /usr/local/lib/python3.6/dist-packages (from tensorboard<1.16.0,>=1.15.0->tensorflow>=1.14->joeynmt==0.0.1) (0.16.0)\n", "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests->torchtext->joeynmt==0.0.1) (2019.9.11)\n", "Requirement already satisfied: idna<2.9,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests->torchtext->joeynmt==0.0.1) (2.8)\n", "Requirement already satisfied: urllib3<1.25,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests->torchtext->joeynmt==0.0.1) (1.24.3)\n", "Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests->torchtext->joeynmt==0.0.1) (3.0.4)\n", "Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.6/dist-packages (from pandas>=0.15.2->seaborn->joeynmt==0.0.1) (2018.9)\n", "Collecting typed-ast<1.5,>=1.4.0; implementation_name == \"cpython\" and python_version < \"3.8\"\n", "\u001b[?25l Downloading https://files.pythonhosted.org/packages/31/d3/9d1802c161626d0278bafb1ffb32f76b9d01e123881bbf9d91e8ccf28e18/typed_ast-1.4.0-cp36-cp36m-manylinux1_x86_64.whl (736kB)\n", "\u001b[K |████████████████████████████████| 737kB 45.6MB/s \n", "\u001b[?25hCollecting lazy-object-proxy==1.4.*\n", "\u001b[?25l Downloading https://files.pythonhosted.org/packages/0b/dd/b1e3407e9e6913cf178e506cd0dee818e58694d9a5cd1984e3f6a8b9a10f/lazy_object_proxy-1.4.3-cp36-cp36m-manylinux1_x86_64.whl (55kB)\n", "\u001b[K |████████████████████████████████| 61kB 7.7MB/s \n", "\u001b[?25hBuilding wheels for collected packages: joeynmt, pyyaml\n", " Building wheel for joeynmt (setup.py) ... \u001b[?25l\u001b[?25hdone\n", " Created wheel for joeynmt: filename=joeynmt-0.0.1-cp36-none-any.whl size=72136 sha256=8220eb7f8701af981e2a3b2cc53cced306013dadf7788c53794e1bd4e4faac33\n", " Stored in directory: /tmp/pip-ephem-wheel-cache-8uwcx2m0/wheels/db/01/db/751cc9f3e7f6faec127c43644ba250a3ea7ad200594aeda70a\n", " Building wheel for pyyaml (setup.py) ... \u001b[?25l\u001b[?25hdone\n", " Created wheel for pyyaml: filename=PyYAML-5.1.2-cp36-cp36m-linux_x86_64.whl size=44104 sha256=a96dfafda5af340469ec506c2c84d46b08782e6e123f138f56fc2b5212efbdd5\n", " Stored in directory: /root/.cache/pip/wheels/d9/45/dd/65f0b38450c47cf7e5312883deb97d065e030c5cca0a365030\n", "Successfully built joeynmt pyyaml\n", "Installing collected packages: portalocker, sacrebleu, subword-nmt, pyyaml, typed-ast, lazy-object-proxy, astroid, isort, mccabe, pylint, joeynmt\n", " Found existing installation: PyYAML 3.13\n", " Uninstalling PyYAML-3.13:\n", " Successfully uninstalled PyYAML-3.13\n", "Successfully installed astroid-2.3.3 isort-4.3.21 joeynmt-0.0.1 lazy-object-proxy-1.4.3 mccabe-0.6.1 portalocker-1.5.2 pylint-2.4.4 pyyaml-5.1.2 sacrebleu-1.4.2 subword-nmt-0.3.7 typed-ast-1.4.0\n" ], "name": "stdout" } ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "AaE77Tcppex9" }, "source": [ "# Preprocessing the Data into Subword BPE Tokens\n", "\n", "- One of the most powerful improvements for agglutinative languages (a feature of most Bantu languages) is using BPE tokenization [ (Sennrich, 2015) ](https://arxiv.org/abs/1508.07909).\n", "\n", "- It was also shown that by optimizing the umber of BPE codes we significantly improve results for low-resourced languages [(Sennrich, 2019)](https://www.aclweb.org/anthology/P19-1021) [(Martinus, 2019)](https://arxiv.org/abs/1906.05685)\n", "\n", "- Below we have the scripts for doing BPE tokenization of our data. We use 4000 tokens as recommended by [(Sennrich, 2019)](https://www.aclweb.org/anthology/P19-1021). You do not need to change anything. Simply running the below will be suitable. " ] }, { "cell_type": "code", "metadata": { "colab_type": "code", "id": "H-TyjtmXB1mL", "outputId": "9c3a1151-b327-42ef-d974-5fa3b2f05ad6", "colab": { "base_uri": "https://localhost:8080/", "height": 423 } }, "source": [ "# One of the huge boosts in NMT performance was to use a different method of tokenizing. \n", "# Usually, NMT would tokenize by words. However, using a method called BPE gave amazing boosts to performance\n", "\n", "# Do subword NMT\n", "from os import path\n", "os.environ[\"src\"] = source_language # Sets them in bash as well, since we often use bash scripts\n", "os.environ[\"tgt\"] = target_language\n", "\n", "# Learn BPEs on the training data.\n", "os.environ[\"data_path\"] = path.join(\"joeynmt\", \"data\", source_language + target_language) # Herman! \n", "! subword-nmt learn-joint-bpe-and-vocab --input train.$src train.$tgt -s 4000 -o bpe.codes.4000 --write-vocabulary vocab.$src vocab.$tgt\n", "\n", "# Apply BPE splits to the development and test data.\n", "! subword-nmt apply-bpe -c bpe.codes.4000 --vocabulary vocab.$src < train.$src > train.bpe.$src\n", "! subword-nmt apply-bpe -c bpe.codes.4000 --vocabulary vocab.$tgt < train.$tgt > train.bpe.$tgt\n", "\n", "! subword-nmt apply-bpe -c bpe.codes.4000 --vocabulary vocab.$src < dev.$src > dev.bpe.$src\n", "! subword-nmt apply-bpe -c bpe.codes.4000 --vocabulary vocab.$tgt < dev.$tgt > dev.bpe.$tgt\n", "! subword-nmt apply-bpe -c bpe.codes.4000 --vocabulary vocab.$src < test.$src > test.bpe.$src\n", "! subword-nmt apply-bpe -c bpe.codes.4000 --vocabulary vocab.$tgt < test.$tgt > test.bpe.$tgt\n", "\n", "# Create directory, move everyone we care about to the correct location\n", "! mkdir -p $data_path\n", "! cp train.* $data_path\n", "! cp test.* $data_path\n", "! cp dev.* $data_path\n", "! cp bpe.codes.4000 $data_path\n", "! ls $data_path\n", "\n", "# Also move everything we care about to a mounted location in google drive (relevant if running in colab) at gdrive_path\n", "! cp train.* \"$gdrive_path\"\n", "! cp test.* \"$gdrive_path\"\n", "! cp dev.* \"$gdrive_path\"\n", "! cp bpe.codes.4000 \"$gdrive_path\"\n", "! ls \"$gdrive_path\"\n", "\n", "# Create that vocab using build_vocab\n", "! sudo chmod 777 joeynmt/scripts/build_vocab.py\n", "! joeynmt/scripts/build_vocab.py joeynmt/data/$src$tgt/train.bpe.$src joeynmt/data/$src$tgt/train.bpe.$tgt --output_path joeynmt/data/$src$tgt/vocab.txt\n", "\n", "# Some output\n", "! echo \"BPE Setswana Sentences\"\n", "! tail -n 5 test.bpe.$tgt\n", "! echo \"Combined BPE Vocab\"\n", "! tail -n 10 joeynmt/data/$src$tgt/vocab.txt # Herman" ], "execution_count": 12, "outputs": [ { "output_type": "stream", "text": [ "bpe.codes.4000\tdev.en\t test.bpe.tn test.tn\t train.en\n", "dev.bpe.en\tdev.tn\t test.en\t train.bpe.en train.tn\n", "dev.bpe.tn\ttest.bpe.en test.en-any.en train.bpe.tn\n", "bpe.codes.4000\tdev.en\t test.bpe.tn test.tn\t train.en\n", "dev.bpe.en\tdev.tn\t test.en\t train.bpe.en train.tn\n", "dev.bpe.tn\ttest.bpe.en test.en-any.en train.bpe.tn\n", "BPE Setswana Sentences\n", "Se@@ o se ile sa dira gore ke its@@ iwe ke le motho yo o sa ik@@ any@@ eg@@ eng .\n", "Fa ke sena go ithuta bo@@ am@@ ma@@ ar@@ uri , ke ne ka g@@ ana go tswelela ka mokgwa o@@ o tota le fa tiro eo e ne e n@@ tu@@ ela bon@@ tle tota .\n", "Ke tlhom@@ ela bom@@ or@@ wa@@ a@@ ke ba bab@@ edi sekao se se molemo e bile ke kgon@@ ne go tshwan@@ eleg@@ ela go newa ditsh@@ i@@ amelo mo ph@@ u@@ the@@ g@@ ong .\n", "Ba@@ tlhatlho@@ b@@ i ba lekgetho mmogo le batho ba bangwe ba ke g@@ web@@ is@@ anang le bone , ba n@@ ki@@ tse ke le motho yo o ik@@ any@@ egang . ”\n", "R@@ u@@ the o ne a fu@@ du@@ g@@ ela kwa I@@ ser@@ a@@ ele kwa a neng a ka kgona go ob@@ am@@ ela Mo@@ dimo wa bo@@ am@@ ma@@ ar@@ uri gone .\n", "Combined BPE Vocab\n", "Ó@@\n", "²@@\n", "ś@@\n", "‐@@\n", "\n", "¬@@\n", "‚@@\n", "\\\n", "nology\n", "£\n" ], "name": "stdout" } ] }, { "cell_type": "code", "metadata": { "colab_type": "code", "id": "IlMitUHR8Qy-", "outputId": "177c01a8-907d-46c0-8a63-8a6edb091399", "colab": { "base_uri": "https://localhost:8080/", "height": 67 } }, "source": [ "# Also move everything we care about to a mounted location in google drive (relevant if running in colab) at gdrive_path\n", "! cp train.* \"$gdrive_path\"\n", "! cp test.* \"$gdrive_path\"\n", "! cp dev.* \"$gdrive_path\"\n", "! cp bpe.codes.4000 \"$gdrive_path\"\n", "! ls \"$gdrive_path\"" ], "execution_count": 13, "outputs": [ { "output_type": "stream", "text": [ "bpe.codes.4000\tdev.en\t test.bpe.tn test.tn\t train.en\n", "dev.bpe.en\tdev.tn\t test.en\t train.bpe.en train.tn\n", "dev.bpe.tn\ttest.bpe.en test.en-any.en train.bpe.tn\n" ], "name": "stdout" } ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "Ixmzi60WsUZ8" }, "source": [ "# Creating the JoeyNMT Config\n", "\n", "JoeyNMT requires a yaml config. We provide a template below. We've also set a number of defaults with it, that you may play with!\n", "\n", "- We used Transformer architecture \n", "- We set our dropout to reasonably high: 0.3 (recommended in [(Sennrich, 2019)](https://www.aclweb.org/anthology/P19-1021))\n", "\n", "Things worth playing with:\n", "- The batch size (also recommended to change for low-resourced languages)\n", "- The number of epochs (we've set it at 30 just so it runs in about an hour, for testing purposes)\n", "- The decoder options (beam_size, alpha)\n", "- Evaluation metrics (BLEU versus Crhf4)" ] }, { "cell_type": "code", "metadata": { "colab_type": "code", "id": "PIs1lY2hxMsl", "colab": {} }, "source": [ "# This creates the config file for our JoeyNMT system. It might seem overwhelming so we've provided a couple of useful parameters you'll need to update\n", "# (You can of course play with all the parameters if you'd like!)\n", "\n", "name = '%s%s' % (source_language, target_language)\n", "gdrive_path = os.environ[\"gdrive_path\"]\n", "\n", "# Create the config\n", "config = \"\"\"\n", "name: \"{name}_transformer\"\n", "\n", "data:\n", " src: \"{source_language}\"\n", " trg: \"{target_language}\"\n", " train: \"data/{name}/train.bpe\"\n", " dev: \"data/{name}/dev.bpe\"\n", " test: \"data/{name}/test.bpe\"\n", " level: \"bpe\"\n", " lowercase: False\n", " max_sent_length: 100\n", " src_vocab: \"data/{name}/vocab.txt\"\n", " trg_vocab: \"data/{name}/vocab.txt\"\n", "\n", "testing:\n", " beam_size: 5\n", " alpha: 1.0\n", "\n", "training:\n", " #load_model: \"{gdrive_path}/models/{name}_transformer/1.ckpt\" # if uncommented, load a pre-trained model from this checkpoint\n", " random_seed: 42\n", " optimizer: \"adam\"\n", " normalization: \"tokens\"\n", " adam_betas: [0.9, 0.999] \n", " scheduling: \"plateau\" # TODO: try switching from plateau to Noam scheduling\n", " patience: 5 # For plateau: decrease learning rate by decrease_factor if validation score has not improved for this many validation rounds.\n", " learning_rate_factor: 0.5 # factor for Noam scheduler (used with Transformer)\n", " learning_rate_warmup: 1000 # warmup steps for Noam scheduler (used with Transformer)\n", " decrease_factor: 0.7\n", " loss: \"crossentropy\"\n", " learning_rate: 0.0003\n", " learning_rate_min: 0.00000001\n", " weight_decay: 0.0\n", " label_smoothing: 0.1\n", " batch_size: 4096\n", " batch_type: \"token\"\n", " eval_batch_size: 3600\n", " eval_batch_type: \"token\"\n", " batch_multiplier: 1\n", " early_stopping_metric: \"ppl\"\n", " epochs: 100 # TODO: Decrease for when playing around and checking of working. Around 30 is sufficient to check if its working at all\n", " validation_freq: 2000 # TODO: Set to at least once per epoch.\n", " logging_freq: 100\n", " eval_metric: \"bleu\"\n", " model_dir: \"models/{name}_transformer\"\n", " overwrite: False # TODO: Set to True if you want to overwrite possibly existing models. \n", " shuffle: True\n", " use_cuda: True\n", " max_output_length: 100\n", " print_valid_sents: [0, 1, 2, 3]\n", " keep_last_ckpts: 3\n", "\n", "model:\n", " initializer: \"xavier\"\n", " bias_initializer: \"zeros\"\n", " init_gain: 1.0\n", " embed_initializer: \"xavier\"\n", " embed_init_gain: 1.0\n", " tied_embeddings: True\n", " tied_softmax: True\n", " encoder:\n", " type: \"transformer\"\n", " num_layers: 6\n", " num_heads: 4 # TODO: Increase to 8 for larger data.\n", " embeddings:\n", " embedding_dim: 256 # TODO: Increase to 512 for larger data.\n", " scale: True\n", " dropout: 0.2\n", " # typically ff_size = 4 x hidden_size\n", " hidden_size: 256 # TODO: Increase to 512 for larger data.\n", " ff_size: 1024 # TODO: Increase to 2048 for larger data.\n", " dropout: 0.3\n", " decoder:\n", " type: \"transformer\"\n", " num_layers: 6\n", " num_heads: 4 # TODO: Increase to 8 for larger data.\n", " embeddings:\n", " embedding_dim: 256 # TODO: Increase to 512 for larger data.\n", " scale: True\n", " dropout: 0.2\n", " # typically ff_size = 4 x hidden_size\n", " hidden_size: 256 # TODO: Increase to 512 for larger data.\n", " ff_size: 1024 # TODO: Increase to 2048 for larger data.\n", " dropout: 0.3\n", "\"\"\".format(name=name, gdrive_path=os.environ[\"gdrive_path\"], source_language=source_language, target_language=target_language)\n", "with open(\"joeynmt/configs/transformer_{name}.yaml\".format(name=name),'w') as f:\n", " f.write(config)" ], "execution_count": 0, "outputs": [] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "pIifxE3Qzuvs" }, "source": [ "# Train the Model\n", "\n", "This single line of joeynmt runs the training using the config we made above" ] }, { "cell_type": "code", "metadata": { "colab_type": "code", "id": "6ZBPFwT94WpI", "outputId": "1270b54f-786d-4e48-98af-276d6e272524", "colab": { "base_uri": "https://localhost:8080/", "height": 1000 } }, "source": [ "# Train the model\n", "# You can press Ctrl-C to stop. And then run the next cell to save your checkpoints! \n", "!cd joeynmt; python3 -m joeynmt train configs/transformer_$src$tgt.yaml" ], "execution_count": 15, "outputs": [ { "output_type": "stream", "text": [ "2019-11-27 08:48:18,492 Hello! This is Joey-NMT.\n", "2019-11-27 08:48:20,425 Total params: 12147456\n", "2019-11-27 08:48:20,426 Trainable parameters: ['decoder.layer_norm.bias', 'decoder.layer_norm.weight', 'decoder.layers.0.dec_layer_norm.bias', 'decoder.layers.0.dec_layer_norm.weight', 'decoder.layers.0.feed_forward.layer_norm.bias', 'decoder.layers.0.feed_forward.layer_norm.weight', 'decoder.layers.0.feed_forward.pwff_layer.0.bias', 'decoder.layers.0.feed_forward.pwff_layer.0.weight', 'decoder.layers.0.feed_forward.pwff_layer.3.bias', 'decoder.layers.0.feed_forward.pwff_layer.3.weight', 'decoder.layers.0.src_trg_att.k_layer.bias', 'decoder.layers.0.src_trg_att.k_layer.weight', 'decoder.layers.0.src_trg_att.output_layer.bias', 'decoder.layers.0.src_trg_att.output_layer.weight', 'decoder.layers.0.src_trg_att.q_layer.bias', 'decoder.layers.0.src_trg_att.q_layer.weight', 'decoder.layers.0.src_trg_att.v_layer.bias', 'decoder.layers.0.src_trg_att.v_layer.weight', 'decoder.layers.0.trg_trg_att.k_layer.bias', 'decoder.layers.0.trg_trg_att.k_layer.weight', 'decoder.layers.0.trg_trg_att.output_layer.bias', 'decoder.layers.0.trg_trg_att.output_layer.weight', 'decoder.layers.0.trg_trg_att.q_layer.bias', 'decoder.layers.0.trg_trg_att.q_layer.weight', 'decoder.layers.0.trg_trg_att.v_layer.bias', 'decoder.layers.0.trg_trg_att.v_layer.weight', 'decoder.layers.0.x_layer_norm.bias', 'decoder.layers.0.x_layer_norm.weight', 'decoder.layers.1.dec_layer_norm.bias', 'decoder.layers.1.dec_layer_norm.weight', 'decoder.layers.1.feed_forward.layer_norm.bias', 'decoder.layers.1.feed_forward.layer_norm.weight', 'decoder.layers.1.feed_forward.pwff_layer.0.bias', 'decoder.layers.1.feed_forward.pwff_layer.0.weight', 'decoder.layers.1.feed_forward.pwff_layer.3.bias', 'decoder.layers.1.feed_forward.pwff_layer.3.weight', 'decoder.layers.1.src_trg_att.k_layer.bias', 'decoder.layers.1.src_trg_att.k_layer.weight', 'decoder.layers.1.src_trg_att.output_layer.bias', 'decoder.layers.1.src_trg_att.output_layer.weight', 'decoder.layers.1.src_trg_att.q_layer.bias', 'decoder.layers.1.src_trg_att.q_layer.weight', 'decoder.layers.1.src_trg_att.v_layer.bias', 'decoder.layers.1.src_trg_att.v_layer.weight', 'decoder.layers.1.trg_trg_att.k_layer.bias', 'decoder.layers.1.trg_trg_att.k_layer.weight', 'decoder.layers.1.trg_trg_att.output_layer.bias', 'decoder.layers.1.trg_trg_att.output_layer.weight', 'decoder.layers.1.trg_trg_att.q_layer.bias', 'decoder.layers.1.trg_trg_att.q_layer.weight', 'decoder.layers.1.trg_trg_att.v_layer.bias', 'decoder.layers.1.trg_trg_att.v_layer.weight', 'decoder.layers.1.x_layer_norm.bias', 'decoder.layers.1.x_layer_norm.weight', 'decoder.layers.2.dec_layer_norm.bias', 'decoder.layers.2.dec_layer_norm.weight', 'decoder.layers.2.feed_forward.layer_norm.bias', 'decoder.layers.2.feed_forward.layer_norm.weight', 'decoder.layers.2.feed_forward.pwff_layer.0.bias', 'decoder.layers.2.feed_forward.pwff_layer.0.weight', 'decoder.layers.2.feed_forward.pwff_layer.3.bias', 'decoder.layers.2.feed_forward.pwff_layer.3.weight', 'decoder.layers.2.src_trg_att.k_layer.bias', 'decoder.layers.2.src_trg_att.k_layer.weight', 'decoder.layers.2.src_trg_att.output_layer.bias', 'decoder.layers.2.src_trg_att.output_layer.weight', 'decoder.layers.2.src_trg_att.q_layer.bias', 'decoder.layers.2.src_trg_att.q_layer.weight', 'decoder.layers.2.src_trg_att.v_layer.bias', 'decoder.layers.2.src_trg_att.v_layer.weight', 'decoder.layers.2.trg_trg_att.k_layer.bias', 'decoder.layers.2.trg_trg_att.k_layer.weight', 'decoder.layers.2.trg_trg_att.output_layer.bias', 'decoder.layers.2.trg_trg_att.output_layer.weight', 'decoder.layers.2.trg_trg_att.q_layer.bias', 'decoder.layers.2.trg_trg_att.q_layer.weight', 'decoder.layers.2.trg_trg_att.v_layer.bias', 'decoder.layers.2.trg_trg_att.v_layer.weight', 'decoder.layers.2.x_layer_norm.bias', 'decoder.layers.2.x_layer_norm.weight', 'decoder.layers.3.dec_layer_norm.bias', 'decoder.layers.3.dec_layer_norm.weight', 'decoder.layers.3.feed_forward.layer_norm.bias', 'decoder.layers.3.feed_forward.layer_norm.weight', 'decoder.layers.3.feed_forward.pwff_layer.0.bias', 'decoder.layers.3.feed_forward.pwff_layer.0.weight', 'decoder.layers.3.feed_forward.pwff_layer.3.bias', 'decoder.layers.3.feed_forward.pwff_layer.3.weight', 'decoder.layers.3.src_trg_att.k_layer.bias', 'decoder.layers.3.src_trg_att.k_layer.weight', 'decoder.layers.3.src_trg_att.output_layer.bias', 'decoder.layers.3.src_trg_att.output_layer.weight', 'decoder.layers.3.src_trg_att.q_layer.bias', 'decoder.layers.3.src_trg_att.q_layer.weight', 'decoder.layers.3.src_trg_att.v_layer.bias', 'decoder.layers.3.src_trg_att.v_layer.weight', 'decoder.layers.3.trg_trg_att.k_layer.bias', 'decoder.layers.3.trg_trg_att.k_layer.weight', 'decoder.layers.3.trg_trg_att.output_layer.bias', 'decoder.layers.3.trg_trg_att.output_layer.weight', 'decoder.layers.3.trg_trg_att.q_layer.bias', 'decoder.layers.3.trg_trg_att.q_layer.weight', 'decoder.layers.3.trg_trg_att.v_layer.bias', 'decoder.layers.3.trg_trg_att.v_layer.weight', 'decoder.layers.3.x_layer_norm.bias', 'decoder.layers.3.x_layer_norm.weight', 'decoder.layers.4.dec_layer_norm.bias', 'decoder.layers.4.dec_layer_norm.weight', 'decoder.layers.4.feed_forward.layer_norm.bias', 'decoder.layers.4.feed_forward.layer_norm.weight', 'decoder.layers.4.feed_forward.pwff_layer.0.bias', 'decoder.layers.4.feed_forward.pwff_layer.0.weight', 'decoder.layers.4.feed_forward.pwff_layer.3.bias', 'decoder.layers.4.feed_forward.pwff_layer.3.weight', 'decoder.layers.4.src_trg_att.k_layer.bias', 'decoder.layers.4.src_trg_att.k_layer.weight', 'decoder.layers.4.src_trg_att.output_layer.bias', 'decoder.layers.4.src_trg_att.output_layer.weight', 'decoder.layers.4.src_trg_att.q_layer.bias', 'decoder.layers.4.src_trg_att.q_layer.weight', 'decoder.layers.4.src_trg_att.v_layer.bias', 'decoder.layers.4.src_trg_att.v_layer.weight', 'decoder.layers.4.trg_trg_att.k_layer.bias', 'decoder.layers.4.trg_trg_att.k_layer.weight', 'decoder.layers.4.trg_trg_att.output_layer.bias', 'decoder.layers.4.trg_trg_att.output_layer.weight', 'decoder.layers.4.trg_trg_att.q_layer.bias', 'decoder.layers.4.trg_trg_att.q_layer.weight', 'decoder.layers.4.trg_trg_att.v_layer.bias', 'decoder.layers.4.trg_trg_att.v_layer.weight', 'decoder.layers.4.x_layer_norm.bias', 'decoder.layers.4.x_layer_norm.weight', 'decoder.layers.5.dec_layer_norm.bias', 'decoder.layers.5.dec_layer_norm.weight', 'decoder.layers.5.feed_forward.layer_norm.bias', 'decoder.layers.5.feed_forward.layer_norm.weight', 'decoder.layers.5.feed_forward.pwff_layer.0.bias', 'decoder.layers.5.feed_forward.pwff_layer.0.weight', 'decoder.layers.5.feed_forward.pwff_layer.3.bias', 'decoder.layers.5.feed_forward.pwff_layer.3.weight', 'decoder.layers.5.src_trg_att.k_layer.bias', 'decoder.layers.5.src_trg_att.k_layer.weight', 'decoder.layers.5.src_trg_att.output_layer.bias', 'decoder.layers.5.src_trg_att.output_layer.weight', 'decoder.layers.5.src_trg_att.q_layer.bias', 'decoder.layers.5.src_trg_att.q_layer.weight', 'decoder.layers.5.src_trg_att.v_layer.bias', 'decoder.layers.5.src_trg_att.v_layer.weight', 'decoder.layers.5.trg_trg_att.k_layer.bias', 'decoder.layers.5.trg_trg_att.k_layer.weight', 'decoder.layers.5.trg_trg_att.output_layer.bias', 'decoder.layers.5.trg_trg_att.output_layer.weight', 'decoder.layers.5.trg_trg_att.q_layer.bias', 'decoder.layers.5.trg_trg_att.q_layer.weight', 'decoder.layers.5.trg_trg_att.v_layer.bias', 'decoder.layers.5.trg_trg_att.v_layer.weight', 'decoder.layers.5.x_layer_norm.bias', 'decoder.layers.5.x_layer_norm.weight', 'encoder.layer_norm.bias', 'encoder.layer_norm.weight', 'encoder.layers.0.feed_forward.layer_norm.bias', 'encoder.layers.0.feed_forward.layer_norm.weight', 'encoder.layers.0.feed_forward.pwff_layer.0.bias', 'encoder.layers.0.feed_forward.pwff_layer.0.weight', 'encoder.layers.0.feed_forward.pwff_layer.3.bias', 'encoder.layers.0.feed_forward.pwff_layer.3.weight', 'encoder.layers.0.layer_norm.bias', 'encoder.layers.0.layer_norm.weight', 'encoder.layers.0.src_src_att.k_layer.bias', 'encoder.layers.0.src_src_att.k_layer.weight', 'encoder.layers.0.src_src_att.output_layer.bias', 'encoder.layers.0.src_src_att.output_layer.weight', 'encoder.layers.0.src_src_att.q_layer.bias', 'encoder.layers.0.src_src_att.q_layer.weight', 'encoder.layers.0.src_src_att.v_layer.bias', 'encoder.layers.0.src_src_att.v_layer.weight', 'encoder.layers.1.feed_forward.layer_norm.bias', 'encoder.layers.1.feed_forward.layer_norm.weight', 'encoder.layers.1.feed_forward.pwff_layer.0.bias', 'encoder.layers.1.feed_forward.pwff_layer.0.weight', 'encoder.layers.1.feed_forward.pwff_layer.3.bias', 'encoder.layers.1.feed_forward.pwff_layer.3.weight', 'encoder.layers.1.layer_norm.bias', 'encoder.layers.1.layer_norm.weight', 'encoder.layers.1.src_src_att.k_layer.bias', 'encoder.layers.1.src_src_att.k_layer.weight', 'encoder.layers.1.src_src_att.output_layer.bias', 'encoder.layers.1.src_src_att.output_layer.weight', 'encoder.layers.1.src_src_att.q_layer.bias', 'encoder.layers.1.src_src_att.q_layer.weight', 'encoder.layers.1.src_src_att.v_layer.bias', 'encoder.layers.1.src_src_att.v_layer.weight', 'encoder.layers.2.feed_forward.layer_norm.bias', 'encoder.layers.2.feed_forward.layer_norm.weight', 'encoder.layers.2.feed_forward.pwff_layer.0.bias', 'encoder.layers.2.feed_forward.pwff_layer.0.weight', 'encoder.layers.2.feed_forward.pwff_layer.3.bias', 'encoder.layers.2.feed_forward.pwff_layer.3.weight', 'encoder.layers.2.layer_norm.bias', 'encoder.layers.2.layer_norm.weight', 'encoder.layers.2.src_src_att.k_layer.bias', 'encoder.layers.2.src_src_att.k_layer.weight', 'encoder.layers.2.src_src_att.output_layer.bias', 'encoder.layers.2.src_src_att.output_layer.weight', 'encoder.layers.2.src_src_att.q_layer.bias', 'encoder.layers.2.src_src_att.q_layer.weight', 'encoder.layers.2.src_src_att.v_layer.bias', 'encoder.layers.2.src_src_att.v_layer.weight', 'encoder.layers.3.feed_forward.layer_norm.bias', 'encoder.layers.3.feed_forward.layer_norm.weight', 'encoder.layers.3.feed_forward.pwff_layer.0.bias', 'encoder.layers.3.feed_forward.pwff_layer.0.weight', 'encoder.layers.3.feed_forward.pwff_layer.3.bias', 'encoder.layers.3.feed_forward.pwff_layer.3.weight', 'encoder.layers.3.layer_norm.bias', 'encoder.layers.3.layer_norm.weight', 'encoder.layers.3.src_src_att.k_layer.bias', 'encoder.layers.3.src_src_att.k_layer.weight', 'encoder.layers.3.src_src_att.output_layer.bias', 'encoder.layers.3.src_src_att.output_layer.weight', 'encoder.layers.3.src_src_att.q_layer.bias', 'encoder.layers.3.src_src_att.q_layer.weight', 'encoder.layers.3.src_src_att.v_layer.bias', 'encoder.layers.3.src_src_att.v_layer.weight', 'encoder.layers.4.feed_forward.layer_norm.bias', 'encoder.layers.4.feed_forward.layer_norm.weight', 'encoder.layers.4.feed_forward.pwff_layer.0.bias', 'encoder.layers.4.feed_forward.pwff_layer.0.weight', 'encoder.layers.4.feed_forward.pwff_layer.3.bias', 'encoder.layers.4.feed_forward.pwff_layer.3.weight', 'encoder.layers.4.layer_norm.bias', 'encoder.layers.4.layer_norm.weight', 'encoder.layers.4.src_src_att.k_layer.bias', 'encoder.layers.4.src_src_att.k_layer.weight', 'encoder.layers.4.src_src_att.output_layer.bias', 'encoder.layers.4.src_src_att.output_layer.weight', 'encoder.layers.4.src_src_att.q_layer.bias', 'encoder.layers.4.src_src_att.q_layer.weight', 'encoder.layers.4.src_src_att.v_layer.bias', 'encoder.layers.4.src_src_att.v_layer.weight', 'encoder.layers.5.feed_forward.layer_norm.bias', 'encoder.layers.5.feed_forward.layer_norm.weight', 'encoder.layers.5.feed_forward.pwff_layer.0.bias', 'encoder.layers.5.feed_forward.pwff_layer.0.weight', 'encoder.layers.5.feed_forward.pwff_layer.3.bias', 'encoder.layers.5.feed_forward.pwff_layer.3.weight', 'encoder.layers.5.layer_norm.bias', 'encoder.layers.5.layer_norm.weight', 'encoder.layers.5.src_src_att.k_layer.bias', 'encoder.layers.5.src_src_att.k_layer.weight', 'encoder.layers.5.src_src_att.output_layer.bias', 'encoder.layers.5.src_src_att.output_layer.weight', 'encoder.layers.5.src_src_att.q_layer.bias', 'encoder.layers.5.src_src_att.q_layer.weight', 'encoder.layers.5.src_src_att.v_layer.bias', 'encoder.layers.5.src_src_att.v_layer.weight', 'src_embed.lut.weight']\n", "2019-11-27 08:48:25,675 cfg.name : entn_transformer\n", "2019-11-27 08:48:25,675 cfg.data.src : en\n", "2019-11-27 08:48:25,675 cfg.data.trg : tn\n", "2019-11-27 08:48:25,675 cfg.data.train : data/entn/train.bpe\n", "2019-11-27 08:48:25,675 cfg.data.dev : data/entn/dev.bpe\n", "2019-11-27 08:48:25,675 cfg.data.test : data/entn/test.bpe\n", "2019-11-27 08:48:25,675 cfg.data.level : bpe\n", "2019-11-27 08:48:25,676 cfg.data.lowercase : False\n", "2019-11-27 08:48:25,676 cfg.data.max_sent_length : 100\n", "2019-11-27 08:48:25,676 cfg.data.src_vocab : data/entn/vocab.txt\n", "2019-11-27 08:48:25,676 cfg.data.trg_vocab : data/entn/vocab.txt\n", "2019-11-27 08:48:25,676 cfg.testing.beam_size : 5\n", "2019-11-27 08:48:25,676 cfg.testing.alpha : 1.0\n", "2019-11-27 08:48:25,676 cfg.training.random_seed : 42\n", "2019-11-27 08:48:25,676 cfg.training.optimizer : adam\n", "2019-11-27 08:48:25,676 cfg.training.normalization : tokens\n", "2019-11-27 08:48:25,677 cfg.training.adam_betas : [0.9, 0.999]\n", "2019-11-27 08:48:25,677 cfg.training.scheduling : plateau\n", "2019-11-27 08:48:25,677 cfg.training.patience : 5\n", "2019-11-27 08:48:25,677 cfg.training.learning_rate_factor : 0.5\n", "2019-11-27 08:48:25,677 cfg.training.learning_rate_warmup : 1000\n", "2019-11-27 08:48:25,677 cfg.training.decrease_factor : 0.7\n", "2019-11-27 08:48:25,677 cfg.training.loss : crossentropy\n", "2019-11-27 08:48:25,677 cfg.training.learning_rate : 0.0003\n", "2019-11-27 08:48:25,677 cfg.training.learning_rate_min : 1e-08\n", "2019-11-27 08:48:25,677 cfg.training.weight_decay : 0.0\n", "2019-11-27 08:48:25,678 cfg.training.label_smoothing : 0.1\n", "2019-11-27 08:48:25,678 cfg.training.batch_size : 4096\n", "2019-11-27 08:48:25,678 cfg.training.batch_type : token\n", "2019-11-27 08:48:25,678 cfg.training.eval_batch_size : 3600\n", "2019-11-27 08:48:25,678 cfg.training.eval_batch_type : token\n", "2019-11-27 08:48:25,678 cfg.training.batch_multiplier : 1\n", "2019-11-27 08:48:25,678 cfg.training.early_stopping_metric : ppl\n", "2019-11-27 08:48:25,678 cfg.training.epochs : 100\n", "2019-11-27 08:48:25,678 cfg.training.validation_freq : 2000\n", "2019-11-27 08:48:25,678 cfg.training.logging_freq : 100\n", "2019-11-27 08:48:25,678 cfg.training.eval_metric : bleu\n", "2019-11-27 08:48:25,679 cfg.training.model_dir : models/entn_transformer\n", "2019-11-27 08:48:25,679 cfg.training.overwrite : False\n", "2019-11-27 08:48:25,679 cfg.training.shuffle : True\n", "2019-11-27 08:48:25,679 cfg.training.use_cuda : True\n", "2019-11-27 08:48:25,679 cfg.training.max_output_length : 100\n", "2019-11-27 08:48:25,679 cfg.training.print_valid_sents : [0, 1, 2, 3]\n", "2019-11-27 08:48:25,679 cfg.training.keep_last_ckpts : 3\n", "2019-11-27 08:48:25,679 cfg.model.initializer : xavier\n", "2019-11-27 08:48:25,679 cfg.model.bias_initializer : zeros\n", "2019-11-27 08:48:25,679 cfg.model.init_gain : 1.0\n", "2019-11-27 08:48:25,680 cfg.model.embed_initializer : xavier\n", "2019-11-27 08:48:25,680 cfg.model.embed_init_gain : 1.0\n", "2019-11-27 08:48:25,680 cfg.model.tied_embeddings : True\n", "2019-11-27 08:48:25,680 cfg.model.tied_softmax : True\n", "2019-11-27 08:48:25,680 cfg.model.encoder.type : transformer\n", "2019-11-27 08:48:25,680 cfg.model.encoder.num_layers : 6\n", "2019-11-27 08:48:25,680 cfg.model.encoder.num_heads : 4\n", "2019-11-27 08:48:25,680 cfg.model.encoder.embeddings.embedding_dim : 256\n", "2019-11-27 08:48:25,680 cfg.model.encoder.embeddings.scale : True\n", "2019-11-27 08:48:25,681 cfg.model.encoder.embeddings.dropout : 0.2\n", "2019-11-27 08:48:25,681 cfg.model.encoder.hidden_size : 256\n", "2019-11-27 08:48:25,681 cfg.model.encoder.ff_size : 1024\n", "2019-11-27 08:48:25,681 cfg.model.encoder.dropout : 0.3\n", "2019-11-27 08:48:25,681 cfg.model.decoder.type : transformer\n", "2019-11-27 08:48:25,681 cfg.model.decoder.num_layers : 6\n", "2019-11-27 08:48:25,681 cfg.model.decoder.num_heads : 4\n", "2019-11-27 08:48:25,681 cfg.model.decoder.embeddings.embedding_dim : 256\n", "2019-11-27 08:48:25,681 cfg.model.decoder.embeddings.scale : True\n", "2019-11-27 08:48:25,681 cfg.model.decoder.embeddings.dropout : 0.2\n", "2019-11-27 08:48:25,682 cfg.model.decoder.hidden_size : 256\n", "2019-11-27 08:48:25,682 cfg.model.decoder.ff_size : 1024\n", "2019-11-27 08:48:25,682 cfg.model.decoder.dropout : 0.3\n", "2019-11-27 08:48:25,682 Data set sizes: \n", "\ttrain 116891,\n", "\tvalid 1000,\n", "\ttest 2714\n", "2019-11-27 08:48:25,682 First training example:\n", "\t[SRC] The let@@ ter should be ac@@ comp@@ an@@ ied by the pres@@ cri@@ bed fe@@ e which will be determin@@ ed by the council .\n", "\t[TRG] L@@ ek@@ walo le tshwanetse go p@@ ag@@ anngwa le te@@ fo e e be@@ ilweng e e tla la@@ ol@@ wang ke Lekgotla .\n", "2019-11-27 08:48:25,682 First 10 words (src): (0) (1) (2) (3) (4) . (5) go (6) , (7) e (8) the (9) le\n", "2019-11-27 08:48:25,683 First 10 words (trg): (0) (1) (2) (3) (4) . (5) go (6) , (7) e (8) the (9) le\n", "2019-11-27 08:48:25,683 Number of Src words (types): 4247\n", "2019-11-27 08:48:25,683 Number of Trg words (types): 4247\n", "2019-11-27 08:48:25,684 Model(\n", "\tencoder=TransformerEncoder(num_layers=6, num_heads=4),\n", "\tdecoder=TransformerDecoder(num_layers=6, num_heads=4),\n", "\tsrc_embed=Embeddings(embedding_dim=256, vocab_size=4247),\n", "\ttrg_embed=Embeddings(embedding_dim=256, vocab_size=4247))\n", "2019-11-27 08:48:25,688 EPOCH 1\n", "2019-11-27 08:48:57,890 Epoch 1 Step: 100 Batch Loss: 5.304708 Tokens per Sec: 7472, Lr: 0.000300\n", "2019-11-27 08:49:29,401 Epoch 1 Step: 200 Batch Loss: 5.332230 Tokens per Sec: 7499, Lr: 0.000300\n", "2019-11-27 08:50:01,320 Epoch 1 Step: 300 Batch Loss: 5.046978 Tokens per Sec: 7576, Lr: 0.000300\n", "2019-11-27 08:50:32,907 Epoch 1 Step: 400 Batch Loss: 5.025369 Tokens per Sec: 7403, Lr: 0.000300\n", "2019-11-27 08:51:05,042 Epoch 1 Step: 500 Batch Loss: 4.541200 Tokens per Sec: 7504, Lr: 0.000300\n", "2019-11-27 08:51:37,100 Epoch 1 Step: 600 Batch Loss: 4.753702 Tokens per Sec: 7446, Lr: 0.000300\n", "2019-11-27 08:52:09,388 Epoch 1 Step: 700 Batch Loss: 4.360945 Tokens per Sec: 7397, Lr: 0.000300\n", "2019-11-27 08:52:41,757 Epoch 1 Step: 800 Batch Loss: 4.070097 Tokens per Sec: 7389, Lr: 0.000300\n", "2019-11-27 08:53:14,409 Epoch 1 Step: 900 Batch Loss: 4.049336 Tokens per Sec: 7628, Lr: 0.000300\n", "2019-11-27 08:53:46,884 Epoch 1 Step: 1000 Batch Loss: 3.957411 Tokens per Sec: 7547, Lr: 0.000300\n", "2019-11-27 08:54:18,797 Epoch 1 Step: 1100 Batch Loss: 3.884557 Tokens per Sec: 7484, Lr: 0.000300\n", "2019-11-27 08:54:51,047 Epoch 1 Step: 1200 Batch Loss: 3.973519 Tokens per Sec: 7420, Lr: 0.000300\n", "2019-11-27 08:55:23,090 Epoch 1 Step: 1300 Batch Loss: 3.949845 Tokens per Sec: 7465, Lr: 0.000300\n", "2019-11-27 08:55:30,305 Epoch 1: total training loss 5995.40\n", "2019-11-27 08:55:30,306 EPOCH 2\n", "2019-11-27 08:55:54,987 Epoch 2 Step: 1400 Batch Loss: 3.818370 Tokens per Sec: 7457, Lr: 0.000300\n", "2019-11-27 08:56:27,086 Epoch 2 Step: 1500 Batch Loss: 3.531626 Tokens per Sec: 7507, Lr: 0.000300\n", "2019-11-27 08:56:58,813 Epoch 2 Step: 1600 Batch Loss: 3.752235 Tokens per Sec: 7507, Lr: 0.000300\n", "2019-11-27 08:57:30,855 Epoch 2 Step: 1700 Batch Loss: 3.673694 Tokens per Sec: 7543, Lr: 0.000300\n", "2019-11-27 08:58:02,496 Epoch 2 Step: 1800 Batch Loss: 3.532929 Tokens per Sec: 7542, Lr: 0.000300\n", "2019-11-27 08:58:34,499 Epoch 2 Step: 1900 Batch Loss: 3.248276 Tokens per Sec: 7477, Lr: 0.000300\n", "2019-11-27 08:59:06,657 Epoch 2 Step: 2000 Batch Loss: 3.659960 Tokens per Sec: 7457, Lr: 0.000300\n", "2019-11-27 09:00:49,418 Hooray! New best validation result [ppl]!\n", "2019-11-27 09:00:49,418 Saving new checkpoint.\n", "2019-11-27 09:00:49,670 Example #0\n", "2019-11-27 09:00:49,670 \tSource: The services are free in South Africa\n", "2019-11-27 09:00:49,671 \tReference: Mo Aforikaborwa , ditirelo di neelwa kwa ntle ga go duelelwa .\n", "2019-11-27 09:00:49,671 \tHypothesis: Ditshwanelo tsa Aforikaborwa di ne di le dintsi tsa Aforikaborwa\n", "2019-11-27 09:00:49,671 Example #1\n", "2019-11-27 09:00:49,671 \tSource: ( 7 ) A person or community dispossessed of property after 19 June 1913 as a result of past racially discriminatory laws or practices is entitled , to the extent provided by an Act of Parliament , either to restitution of that property or to equitable redress .\n", "2019-11-27 09:00:49,671 \tReference: ( 7 ) Motho kgotsa setlhopha sa magae se se amogilweng dithoto morago ga la 19 Seetebosigo 1913 ka ntlha ya melao ya kgale ya tlhotlhololo kgotsa ditlwaelo , se tshwanelwa , go ya ka thulaganyetso ya Molao wa Palamente go busediwa dithoto tseo , kgotsa tshiamisetso e e e kalekanang .\n", "2019-11-27 09:00:49,671 \tHypothesis: ( 2 ) Fa o na le seabe sa go ya ka ga go ya ka ga go ya ka 1992 ( 1 ) , go ya ka fa go na le go dira gore go dira gore go na le go dira gore go dira gore go dira gore go dira ka ga ga go dira ka ga ga ga ga ga ga ga ga go dira ka ga ga ga ga ga ga ga go dira ka ga ga ga ga ga ga ga ga ga ga ga ga ga go dira jalo .\n", "2019-11-27 09:00:49,671 Example #2\n", "2019-11-27 09:00:49,672 \tSource: ruling\n", "2019-11-27 09:00:49,672 \tReference: tshwetsotaolo\n", "2019-11-27 09:00:49,672 \tHypothesis: setlala\n", "2019-11-27 09:00:49,672 Example #3\n", "2019-11-27 09:00:49,672 \tSource: Skills Development course .\n", "2019-11-27 09:00:49,673 \tReference: Khoso ya Tlhabololo ya Bokgoni .\n", "2019-11-27 09:00:49,673 \tHypothesis: Diporofense tsa Aforikaborwa .\n", "2019-11-27 09:00:49,673 Validation result (greedy) at epoch 2, step 2000: bleu: 1.28, loss: 93091.5078, ppl: 27.8501, duration: 103.0152s\n", "2019-11-27 09:01:21,645 Epoch 2 Step: 2100 Batch Loss: 3.278017 Tokens per Sec: 7557, Lr: 0.000300\n", "2019-11-27 09:01:53,581 Epoch 2 Step: 2200 Batch Loss: 3.579289 Tokens per Sec: 7561, Lr: 0.000300\n", "2019-11-27 09:02:25,321 Epoch 2 Step: 2300 Batch Loss: 3.459375 Tokens per Sec: 7461, Lr: 0.000300\n", "2019-11-27 09:02:57,319 Epoch 2 Step: 2400 Batch Loss: 3.053100 Tokens per Sec: 7506, Lr: 0.000300\n", "2019-11-27 09:03:29,621 Epoch 2 Step: 2500 Batch Loss: 2.949571 Tokens per Sec: 7524, Lr: 0.000300\n", "2019-11-27 09:04:01,658 Epoch 2 Step: 2600 Batch Loss: 2.839225 Tokens per Sec: 7542, Lr: 0.000300\n", "2019-11-27 09:04:16,188 Epoch 2: total training loss 4585.73\n", "2019-11-27 09:04:16,188 EPOCH 3\n", "2019-11-27 09:04:33,807 Epoch 3 Step: 2700 Batch Loss: 3.286301 Tokens per Sec: 7550, Lr: 0.000300\n", "2019-11-27 09:05:05,402 Epoch 3 Step: 2800 Batch Loss: 3.218746 Tokens per Sec: 7378, Lr: 0.000300\n", "2019-11-27 09:05:37,682 Epoch 3 Step: 2900 Batch Loss: 3.263718 Tokens per Sec: 7697, Lr: 0.000300\n", "2019-11-27 09:06:09,401 Epoch 3 Step: 3000 Batch Loss: 3.094793 Tokens per Sec: 7518, Lr: 0.000300\n", "2019-11-27 09:06:41,360 Epoch 3 Step: 3100 Batch Loss: 3.084072 Tokens per Sec: 7642, Lr: 0.000300\n", "2019-11-27 09:07:12,891 Epoch 3 Step: 3200 Batch Loss: 3.354316 Tokens per Sec: 7576, Lr: 0.000300\n", "2019-11-27 09:07:44,813 Epoch 3 Step: 3300 Batch Loss: 3.104450 Tokens per Sec: 7554, Lr: 0.000300\n", "2019-11-27 09:08:16,950 Epoch 3 Step: 3400 Batch Loss: 2.998196 Tokens per Sec: 7719, Lr: 0.000300\n", "2019-11-27 09:08:48,448 Epoch 3 Step: 3500 Batch Loss: 3.110984 Tokens per Sec: 7501, Lr: 0.000300\n", "2019-11-27 09:09:20,159 Epoch 3 Step: 3600 Batch Loss: 2.711035 Tokens per Sec: 7656, Lr: 0.000300\n", "2019-11-27 09:09:51,390 Epoch 3 Step: 3700 Batch Loss: 2.966396 Tokens per Sec: 7418, Lr: 0.000300\n", "2019-11-27 09:10:23,404 Epoch 3 Step: 3800 Batch Loss: 3.203114 Tokens per Sec: 7640, Lr: 0.000300\n", "2019-11-27 09:10:55,049 Epoch 3 Step: 3900 Batch Loss: 3.031595 Tokens per Sec: 7570, Lr: 0.000300\n", "2019-11-27 09:11:15,549 Epoch 3: total training loss 3993.49\n", "2019-11-27 09:11:15,550 EPOCH 4\n", "2019-11-27 09:11:26,750 Epoch 4 Step: 4000 Batch Loss: 2.838067 Tokens per Sec: 7204, Lr: 0.000300\n", "2019-11-27 09:13:09,348 Hooray! New best validation result [ppl]!\n", "2019-11-27 09:13:09,348 Saving new checkpoint.\n", "2019-11-27 09:13:09,601 Example #0\n", "2019-11-27 09:13:09,602 \tSource: The services are free in South Africa\n", "2019-11-27 09:13:09,602 \tReference: Mo Aforikaborwa , ditirelo di neelwa kwa ntle ga go duelelwa .\n", "2019-11-27 09:13:09,602 \tHypothesis: Ditirelo tsa ditirelo di a dumelang mo Aforikaborwa\n", "2019-11-27 09:13:09,602 Example #1\n", "2019-11-27 09:13:09,603 \tSource: ( 7 ) A person or community dispossessed of property after 19 June 1913 as a result of past racially discriminatory laws or practices is entitled , to the extent provided by an Act of Parliament , either to restitution of that property or to equitable redress .\n", "2019-11-27 09:13:09,603 \tReference: ( 7 ) Motho kgotsa setlhopha sa magae se se amogilweng dithoto morago ga la 19 Seetebosigo 1913 ka ntlha ya melao ya kgale ya tlhotlhololo kgotsa ditlwaelo , se tshwanelwa , go ya ka thulaganyetso ya Molao wa Palamente go busediwa dithoto tseo , kgotsa tshiamisetso e e e kalekanang .\n", "2019-11-27 09:13:09,603 \tHypothesis: ( 7 ) Motho kgotsa batho ba ba nang le seabe sa go ya ka 1999 fa tlase ga 1999 ka 1999 , ka go ya ka go dirisa madi a a farologaneng a a a leng teng ka go ya ka Molao wa Palamente , go ya ka Molao wa go dira gore go nne le tshwanelo ya go ka nna le molao o o o tshwanetseng go dira kopo ya ka Molao wa go dira gore go ka tshwanelo .\n", "2019-11-27 09:13:09,603 Example #2\n", "2019-11-27 09:13:09,603 \tSource: ruling\n", "2019-11-27 09:13:09,603 \tReference: tshwetsotaolo\n", "2019-11-27 09:13:09,604 \tHypothesis: go dirisa\n", "2019-11-27 09:13:09,604 Example #3\n", "2019-11-27 09:13:09,604 \tSource: Skills Development course .\n", "2019-11-27 09:13:09,604 \tReference: Khoso ya Tlhabololo ya Bokgoni .\n", "2019-11-27 09:13:09,604 \tHypothesis: Tlhabololo ya Tlhabololo ya Tlhabololo .\n", "2019-11-27 09:13:09,604 Validation result (greedy) at epoch 4, step 4000: bleu: 5.86, loss: 77095.5312, ppl: 15.7239, duration: 102.8541s\n", "2019-11-27 09:13:41,449 Epoch 4 Step: 4100 Batch Loss: 3.150059 Tokens per Sec: 7557, Lr: 0.000300\n", "2019-11-27 09:14:13,109 Epoch 4 Step: 4200 Batch Loss: 2.778692 Tokens per Sec: 7566, Lr: 0.000300\n", "2019-11-27 09:14:44,930 Epoch 4 Step: 4300 Batch Loss: 2.946270 Tokens per Sec: 7661, Lr: 0.000300\n", "2019-11-27 09:15:16,431 Epoch 4 Step: 4400 Batch Loss: 2.572032 Tokens per Sec: 7571, Lr: 0.000300\n", "2019-11-27 09:15:47,777 Epoch 4 Step: 4500 Batch Loss: 2.432274 Tokens per Sec: 7586, Lr: 0.000300\n", "2019-11-27 09:16:19,565 Epoch 4 Step: 4600 Batch Loss: 2.860476 Tokens per Sec: 7504, Lr: 0.000300\n", "2019-11-27 09:16:51,473 Epoch 4 Step: 4700 Batch Loss: 2.953434 Tokens per Sec: 7630, Lr: 0.000300\n", "2019-11-27 09:17:23,272 Epoch 4 Step: 4800 Batch Loss: 3.018531 Tokens per Sec: 7635, Lr: 0.000300\n", "2019-11-27 09:17:54,811 Epoch 4 Step: 4900 Batch Loss: 2.612107 Tokens per Sec: 7589, Lr: 0.000300\n", "2019-11-27 09:18:26,457 Epoch 4 Step: 5000 Batch Loss: 2.660981 Tokens per Sec: 7712, Lr: 0.000300\n", "2019-11-27 09:18:57,973 Epoch 4 Step: 5100 Batch Loss: 2.805457 Tokens per Sec: 7556, Lr: 0.000300\n", "2019-11-27 09:19:29,830 Epoch 4 Step: 5200 Batch Loss: 2.511969 Tokens per Sec: 7684, Lr: 0.000300\n", "2019-11-27 09:19:56,744 Epoch 4: total training loss 3660.12\n", "2019-11-27 09:19:56,744 EPOCH 5\n", "2019-11-27 09:20:02,014 Epoch 5 Step: 5300 Batch Loss: 2.830742 Tokens per Sec: 7322, Lr: 0.000300\n", "2019-11-27 09:20:33,656 Epoch 5 Step: 5400 Batch Loss: 2.801837 Tokens per Sec: 7541, Lr: 0.000300\n", "2019-11-27 09:21:05,377 Epoch 5 Step: 5500 Batch Loss: 2.864887 Tokens per Sec: 7589, Lr: 0.000300\n", "2019-11-27 09:21:37,449 Epoch 5 Step: 5600 Batch Loss: 2.618131 Tokens per Sec: 7624, Lr: 0.000300\n", "2019-11-27 09:22:09,202 Epoch 5 Step: 5700 Batch Loss: 2.287569 Tokens per Sec: 7584, Lr: 0.000300\n", "2019-11-27 09:22:41,153 Epoch 5 Step: 5800 Batch Loss: 2.794070 Tokens per Sec: 7560, Lr: 0.000300\n", "2019-11-27 09:23:12,571 Epoch 5 Step: 5900 Batch Loss: 2.688682 Tokens per Sec: 7372, Lr: 0.000300\n", "2019-11-27 09:23:44,360 Epoch 5 Step: 6000 Batch Loss: 2.547146 Tokens per Sec: 7708, Lr: 0.000300\n", "2019-11-27 09:25:27,025 Hooray! New best validation result [ppl]!\n", "2019-11-27 09:25:27,025 Saving new checkpoint.\n", "2019-11-27 09:25:27,293 Example #0\n", "2019-11-27 09:25:27,293 \tSource: The services are free in South Africa\n", "2019-11-27 09:25:27,293 \tReference: Mo Aforikaborwa , ditirelo di neelwa kwa ntle ga go duelelwa .\n", "2019-11-27 09:25:27,294 \tHypothesis: Ditirelo tsa ditirelo tse di sa letlelweng mo Aforikaborwa\n", "2019-11-27 09:25:27,294 Example #1\n", "2019-11-27 09:25:27,294 \tSource: ( 7 ) A person or community dispossessed of property after 19 June 1913 as a result of past racially discriminatory laws or practices is entitled , to the extent provided by an Act of Parliament , either to restitution of that property or to equitable redress .\n", "2019-11-27 09:25:27,294 \tReference: ( 7 ) Motho kgotsa setlhopha sa magae se se amogilweng dithoto morago ga la 19 Seetebosigo 1913 ka ntlha ya melao ya kgale ya tlhotlhololo kgotsa ditlwaelo , se tshwanelwa , go ya ka thulaganyetso ya Molao wa Palamente go busediwa dithoto tseo , kgotsa tshiamisetso e e e kalekanang .\n", "2019-11-27 09:25:27,294 \tHypothesis: ( 7 ) Motho kgotsa moagi wa baagi o o tlhophilweng morago ga 1919 Mopitlwe 1999 , ka ntlha ya mabaka a a fetileng , a a a sa letleleleng go ya ka molao o o fetileng , kgotsa a a a newang ke molao wa Palamente , go ya ka Molao wa Palamente , go ya ka Molao o o o ka se nang le tshwanelo .\n", "2019-11-27 09:25:27,294 Example #2\n", "2019-11-27 09:25:27,295 \tSource: ruling\n", "2019-11-27 09:25:27,295 \tReference: tshwetsotaolo\n", "2019-11-27 09:25:27,295 \tHypothesis: go ntsha\n", "2019-11-27 09:25:27,295 Example #3\n", "2019-11-27 09:25:27,295 \tSource: Skills Development course .\n", "2019-11-27 09:25:27,295 \tReference: Khoso ya Tlhabololo ya Bokgoni .\n", "2019-11-27 09:25:27,295 \tHypothesis: Go tlhabolola tlhabololo ya go ithuta .\n", "2019-11-27 09:25:27,296 Validation result (greedy) at epoch 5, step 6000: bleu: 9.38, loss: 68491.0625, ppl: 11.5615, duration: 102.9350s\n", "2019-11-27 09:25:59,225 Epoch 5 Step: 6100 Batch Loss: 2.186814 Tokens per Sec: 7564, Lr: 0.000300\n", "2019-11-27 09:26:31,207 Epoch 5 Step: 6200 Batch Loss: 2.570421 Tokens per Sec: 7666, Lr: 0.000300\n", "2019-11-27 09:27:03,237 Epoch 5 Step: 6300 Batch Loss: 2.276635 Tokens per Sec: 7604, Lr: 0.000300\n", "2019-11-27 09:27:34,818 Epoch 5 Step: 6400 Batch Loss: 2.338682 Tokens per Sec: 7558, Lr: 0.000300\n", "2019-11-27 09:28:06,944 Epoch 5 Step: 6500 Batch Loss: 2.383706 Tokens per Sec: 7667, Lr: 0.000300\n", "2019-11-27 09:28:38,575 Epoch 5 Step: 6600 Batch Loss: 2.201855 Tokens per Sec: 7546, Lr: 0.000300\n", "2019-11-27 09:28:38,738 Epoch 5: total training loss 3408.62\n", "2019-11-27 09:28:38,738 EPOCH 6\n", "2019-11-27 09:29:10,393 Epoch 6 Step: 6700 Batch Loss: 2.612136 Tokens per Sec: 7499, Lr: 0.000300\n", "2019-11-27 09:29:42,409 Epoch 6 Step: 6800 Batch Loss: 2.472360 Tokens per Sec: 7672, Lr: 0.000300\n", "2019-11-27 09:30:14,054 Epoch 6 Step: 6900 Batch Loss: 2.360480 Tokens per Sec: 7562, Lr: 0.000300\n", "2019-11-27 09:30:45,949 Epoch 6 Step: 7000 Batch Loss: 2.385783 Tokens per Sec: 7514, Lr: 0.000300\n", "2019-11-27 09:31:17,404 Epoch 6 Step: 7100 Batch Loss: 2.846020 Tokens per Sec: 7487, Lr: 0.000300\n", "2019-11-27 09:31:49,464 Epoch 6 Step: 7200 Batch Loss: 2.586819 Tokens per Sec: 7734, Lr: 0.000300\n", "2019-11-27 09:32:21,092 Epoch 6 Step: 7300 Batch Loss: 2.599326 Tokens per Sec: 7518, Lr: 0.000300\n", "2019-11-27 09:32:52,639 Epoch 6 Step: 7400 Batch Loss: 2.242715 Tokens per Sec: 7582, Lr: 0.000300\n", "2019-11-27 09:33:24,478 Epoch 6 Step: 7500 Batch Loss: 2.300281 Tokens per Sec: 7640, Lr: 0.000300\n", "2019-11-27 09:33:56,332 Epoch 6 Step: 7600 Batch Loss: 2.013496 Tokens per Sec: 7617, Lr: 0.000300\n", "2019-11-27 09:34:28,379 Epoch 6 Step: 7700 Batch Loss: 2.629783 Tokens per Sec: 7542, Lr: 0.000300\n", "2019-11-27 09:35:00,286 Epoch 6 Step: 7800 Batch Loss: 2.498090 Tokens per Sec: 7554, Lr: 0.000300\n", "2019-11-27 09:35:32,448 Epoch 6 Step: 7900 Batch Loss: 2.138571 Tokens per Sec: 7727, Lr: 0.000300\n", "2019-11-27 09:35:37,482 Epoch 6: total training loss 3203.02\n", "2019-11-27 09:35:37,483 EPOCH 7\n", "2019-11-27 09:36:04,011 Epoch 7 Step: 8000 Batch Loss: 2.415092 Tokens per Sec: 7519, Lr: 0.000300\n", "2019-11-27 09:37:46,476 Hooray! New best validation result [ppl]!\n", "2019-11-27 09:37:46,476 Saving new checkpoint.\n", "2019-11-27 09:37:46,760 Example #0\n", "2019-11-27 09:37:46,761 \tSource: The services are free in South Africa\n", "2019-11-27 09:37:46,761 \tReference: Mo Aforikaborwa , ditirelo di neelwa kwa ntle ga go duelelwa .\n", "2019-11-27 09:37:46,762 \tHypothesis: Ditirelo tsa Aforikaborwa di a se duelelwe mo Aforikaborwa\n", "2019-11-27 09:37:46,762 Example #1\n", "2019-11-27 09:37:46,762 \tSource: ( 7 ) A person or community dispossessed of property after 19 June 1913 as a result of past racially discriminatory laws or practices is entitled , to the extent provided by an Act of Parliament , either to restitution of that property or to equitable redress .\n", "2019-11-27 09:37:46,762 \tReference: ( 7 ) Motho kgotsa setlhopha sa magae se se amogilweng dithoto morago ga la 19 Seetebosigo 1913 ka ntlha ya melao ya kgale ya tlhotlhololo kgotsa ditlwaelo , se tshwanelwa , go ya ka thulaganyetso ya Molao wa Palamente go busediwa dithoto tseo , kgotsa tshiamisetso e e e kalekanang .\n", "2019-11-27 09:37:46,762 \tHypothesis: ( 7 ) Motho kgotsa baagi ba ba tlhophilweng morago ga 19 Ferikgong 1913 ka ntlha ya gore go na le mekgwa e e fetileng e e fetileng e e dirwang ke molao o o sa letleleleng , e e leng ya semolao , e e neetsweng ke Molao wa Palamente , e e neetsweng ke Molao wa Palamente .\n", "2019-11-27 09:37:46,762 Example #2\n", "2019-11-27 09:37:46,763 \tSource: ruling\n", "2019-11-27 09:37:46,763 \tReference: tshwetsotaolo\n", "2019-11-27 09:37:46,763 \tHypothesis: go ya ka\n", "2019-11-27 09:37:46,763 Example #3\n", "2019-11-27 09:37:46,764 \tSource: Skills Development course .\n", "2019-11-27 09:37:46,764 \tReference: Khoso ya Tlhabololo ya Bokgoni .\n", "2019-11-27 09:37:46,764 \tHypothesis: Khoso ya Tlhabololo .\n", "2019-11-27 09:37:46,764 Validation result (greedy) at epoch 7, step 8000: bleu: 12.99, loss: 62278.0508, ppl: 9.2595, duration: 102.7525s\n", "2019-11-27 09:38:18,512 Epoch 7 Step: 8100 Batch Loss: 2.435576 Tokens per Sec: 7638, Lr: 0.000300\n", "2019-11-27 09:38:50,128 Epoch 7 Step: 8200 Batch Loss: 2.250937 Tokens per Sec: 7571, Lr: 0.000300\n", "2019-11-27 09:39:21,601 Epoch 7 Step: 8300 Batch Loss: 2.167715 Tokens per Sec: 7408, Lr: 0.000300\n", "2019-11-27 09:39:53,324 Epoch 7 Step: 8400 Batch Loss: 2.326120 Tokens per Sec: 7678, Lr: 0.000300\n", "2019-11-27 09:40:25,008 Epoch 7 Step: 8500 Batch Loss: 2.365795 Tokens per Sec: 7474, Lr: 0.000300\n", "2019-11-27 09:40:56,725 Epoch 7 Step: 8600 Batch Loss: 2.451020 Tokens per Sec: 7637, Lr: 0.000300\n", "2019-11-27 09:41:28,310 Epoch 7 Step: 8700 Batch Loss: 2.421954 Tokens per Sec: 7548, Lr: 0.000300\n", "2019-11-27 09:42:00,170 Epoch 7 Step: 8800 Batch Loss: 2.153945 Tokens per Sec: 7599, Lr: 0.000300\n", "2019-11-27 09:42:32,233 Epoch 7 Step: 8900 Batch Loss: 2.306963 Tokens per Sec: 7482, Lr: 0.000300\n", "2019-11-27 09:43:03,815 Epoch 7 Step: 9000 Batch Loss: 2.084206 Tokens per Sec: 7514, Lr: 0.000300\n", "2019-11-27 09:43:35,952 Epoch 7 Step: 9100 Batch Loss: 2.325266 Tokens per Sec: 7585, Lr: 0.000300\n", "2019-11-27 09:44:07,943 Epoch 7 Step: 9200 Batch Loss: 2.101885 Tokens per Sec: 7650, Lr: 0.000300\n", "2019-11-27 09:44:20,020 Epoch 7: total training loss 3062.50\n", "2019-11-27 09:44:20,020 EPOCH 8\n", "2019-11-27 09:44:39,723 Epoch 8 Step: 9300 Batch Loss: 2.507118 Tokens per Sec: 7413, Lr: 0.000300\n", "2019-11-27 09:45:11,672 Epoch 8 Step: 9400 Batch Loss: 2.195923 Tokens per Sec: 7721, Lr: 0.000300\n", "2019-11-27 09:45:43,477 Epoch 8 Step: 9500 Batch Loss: 2.233500 Tokens per Sec: 7555, Lr: 0.000300\n", "2019-11-27 09:46:15,325 Epoch 8 Step: 9600 Batch Loss: 2.297036 Tokens per Sec: 7665, Lr: 0.000300\n", "2019-11-27 09:46:47,208 Epoch 8 Step: 9700 Batch Loss: 2.501548 Tokens per Sec: 7657, Lr: 0.000300\n", "2019-11-27 09:47:19,083 Epoch 8 Step: 9800 Batch Loss: 1.903674 Tokens per Sec: 7628, Lr: 0.000300\n", "2019-11-27 09:47:50,927 Epoch 8 Step: 9900 Batch Loss: 2.013997 Tokens per Sec: 7562, Lr: 0.000300\n", "2019-11-27 09:48:22,628 Epoch 8 Step: 10000 Batch Loss: 2.204835 Tokens per Sec: 7630, Lr: 0.000300\n", "2019-11-27 09:50:04,808 Hooray! New best validation result [ppl]!\n", "2019-11-27 09:50:04,809 Saving new checkpoint.\n", "2019-11-27 09:50:05,117 Example #0\n", "2019-11-27 09:50:05,118 \tSource: The services are free in South Africa\n", "2019-11-27 09:50:05,118 \tReference: Mo Aforikaborwa , ditirelo di neelwa kwa ntle ga go duelelwa .\n", "2019-11-27 09:50:05,118 \tHypothesis: Ditirelo tsa ditirelo di gololotswe mo Aforikaborwa\n", "2019-11-27 09:50:05,118 Example #1\n", "2019-11-27 09:50:05,119 \tSource: ( 7 ) A person or community dispossessed of property after 19 June 1913 as a result of past racially discriminatory laws or practices is entitled , to the extent provided by an Act of Parliament , either to restitution of that property or to equitable redress .\n", "2019-11-27 09:50:05,119 \tReference: ( 7 ) Motho kgotsa setlhopha sa magae se se amogilweng dithoto morago ga la 19 Seetebosigo 1913 ka ntlha ya melao ya kgale ya tlhotlhololo kgotsa ditlwaelo , se tshwanelwa , go ya ka thulaganyetso ya Molao wa Palamente go busediwa dithoto tseo , kgotsa tshiamisetso e e e kalekanang .\n", "2019-11-27 09:50:05,119 \tHypothesis: ( 7 ) Motho kgotsa baagi ba ba tlhophilweng morago ga 19 Seetebosigo 1913 ka ntlha ya gore melao e e fetileng e e tlhomilweng kgotsa e e sa tlhomameng , e e neilweng ke Molao o o tlhomilweng ke Palamente , e e ka nna ya taelo ya go dira gore e nne le tshwanelo ya go dira gore e nne le tshwanelo ya go dira .\n", "2019-11-27 09:50:05,119 Example #2\n", "2019-11-27 09:50:05,119 \tSource: ruling\n", "2019-11-27 09:50:05,119 \tReference: tshwetsotaolo\n", "2019-11-27 09:50:05,119 \tHypothesis: go kgoreletsa\n", "2019-11-27 09:50:05,119 Example #3\n", "2019-11-27 09:50:05,120 \tSource: Skills Development course .\n", "2019-11-27 09:50:05,120 \tReference: Khoso ya Tlhabololo ya Bokgoni .\n", "2019-11-27 09:50:05,120 \tHypothesis: Khoso ya Tlhabololo ya Dikgono .\n", "2019-11-27 09:50:05,120 Validation result (greedy) at epoch 8, step 10000: bleu: 15.45, loss: 57997.6445, ppl: 7.9461, duration: 102.4913s\n", "2019-11-27 09:50:36,880 Epoch 8 Step: 10100 Batch Loss: 2.395377 Tokens per Sec: 7540, Lr: 0.000300\n", "2019-11-27 09:51:08,776 Epoch 8 Step: 10200 Batch Loss: 2.394763 Tokens per Sec: 7651, Lr: 0.000300\n", "2019-11-27 09:51:40,527 Epoch 8 Step: 10300 Batch Loss: 2.256118 Tokens per Sec: 7662, Lr: 0.000300\n", "2019-11-27 09:52:12,273 Epoch 8 Step: 10400 Batch Loss: 2.351913 Tokens per Sec: 7544, Lr: 0.000300\n", "2019-11-27 09:52:43,872 Epoch 8 Step: 10500 Batch Loss: 2.669105 Tokens per Sec: 7541, Lr: 0.000300\n", "2019-11-27 09:53:00,295 Epoch 8: total training loss 2907.54\n", "2019-11-27 09:53:00,295 EPOCH 9\n", "2019-11-27 09:53:15,755 Epoch 9 Step: 10600 Batch Loss: 2.037788 Tokens per Sec: 7530, Lr: 0.000300\n", "2019-11-27 09:53:47,393 Epoch 9 Step: 10700 Batch Loss: 2.128225 Tokens per Sec: 7592, Lr: 0.000300\n", "2019-11-27 09:54:18,829 Epoch 9 Step: 10800 Batch Loss: 2.340935 Tokens per Sec: 7552, Lr: 0.000300\n", "2019-11-27 09:54:50,709 Epoch 9 Step: 10900 Batch Loss: 1.747643 Tokens per Sec: 7598, Lr: 0.000300\n", "2019-11-27 09:55:22,813 Epoch 9 Step: 11000 Batch Loss: 2.066226 Tokens per Sec: 7586, Lr: 0.000300\n", "2019-11-27 09:55:54,910 Epoch 9 Step: 11100 Batch Loss: 2.402025 Tokens per Sec: 7613, Lr: 0.000300\n", "2019-11-27 09:56:26,876 Epoch 9 Step: 11200 Batch Loss: 2.289386 Tokens per Sec: 7556, Lr: 0.000300\n", "2019-11-27 09:56:58,874 Epoch 9 Step: 11300 Batch Loss: 1.810267 Tokens per Sec: 7616, Lr: 0.000300\n", "2019-11-27 09:57:30,148 Epoch 9 Step: 11400 Batch Loss: 1.982777 Tokens per Sec: 7426, Lr: 0.000300\n", "2019-11-27 09:58:01,984 Epoch 9 Step: 11500 Batch Loss: 2.155839 Tokens per Sec: 7718, Lr: 0.000300\n", "2019-11-27 09:58:33,757 Epoch 9 Step: 11600 Batch Loss: 2.626151 Tokens per Sec: 7603, Lr: 0.000300\n", "2019-11-27 09:59:05,193 Epoch 9 Step: 11700 Batch Loss: 1.921330 Tokens per Sec: 7500, Lr: 0.000300\n", "2019-11-27 09:59:36,648 Epoch 9 Step: 11800 Batch Loss: 2.203984 Tokens per Sec: 7553, Lr: 0.000300\n", "2019-11-27 09:59:59,771 Epoch 9: total training loss 2813.03\n", "2019-11-27 09:59:59,771 EPOCH 10\n", "2019-11-27 10:00:08,434 Epoch 10 Step: 11900 Batch Loss: 1.822659 Tokens per Sec: 7260, Lr: 0.000300\n", "2019-11-27 10:00:40,143 Epoch 10 Step: 12000 Batch Loss: 2.237783 Tokens per Sec: 7582, Lr: 0.000300\n", "2019-11-27 10:02:22,352 Hooray! New best validation result [ppl]!\n", "2019-11-27 10:02:22,352 Saving new checkpoint.\n", "2019-11-27 10:02:22,634 Example #0\n", "2019-11-27 10:02:22,635 \tSource: The services are free in South Africa\n", "2019-11-27 10:02:22,635 \tReference: Mo Aforikaborwa , ditirelo di neelwa kwa ntle ga go duelelwa .\n", "2019-11-27 10:02:22,635 \tHypothesis: Ditirelo di gololosegile mo Aforikaborwa\n", "2019-11-27 10:02:22,635 Example #1\n", "2019-11-27 10:02:22,635 \tSource: ( 7 ) A person or community dispossessed of property after 19 June 1913 as a result of past racially discriminatory laws or practices is entitled , to the extent provided by an Act of Parliament , either to restitution of that property or to equitable redress .\n", "2019-11-27 10:02:22,635 \tReference: ( 7 ) Motho kgotsa setlhopha sa magae se se amogilweng dithoto morago ga la 19 Seetebosigo 1913 ka ntlha ya melao ya kgale ya tlhotlhololo kgotsa ditlwaelo , se tshwanelwa , go ya ka thulaganyetso ya Molao wa Palamente go busediwa dithoto tseo , kgotsa tshiamisetso e e e kalekanang .\n", "2019-11-27 10:02:22,636 \tHypothesis: ( 7 ) Motho kgotsa baagi ba ba nang le tshiamelo ya 19 morago ga 19 Seetebosigo 1913 ka ntlha ya diphoso tse di fetileng , kgotsa mekgwa e e sa tlhomameng , e e newang ke Palamente , ka ntlha ya go dira gore go nne le tshwanelo ya go dira gore go nne le tshwanelo ya go dira gore go se na le tshwanelo ya go se lekalekanang .\n", "2019-11-27 10:02:22,636 Example #2\n", "2019-11-27 10:02:22,636 \tSource: ruling\n", "2019-11-27 10:02:22,636 \tReference: tshwetsotaolo\n", "2019-11-27 10:02:22,636 \tHypothesis: go tsosolosa\n", "2019-11-27 10:02:22,636 Example #3\n", "2019-11-27 10:02:22,637 \tSource: Skills Development course .\n", "2019-11-27 10:02:22,637 \tReference: Khoso ya Tlhabololo ya Bokgoni .\n", "2019-11-27 10:02:22,637 \tHypothesis: Khoso ya Tlhabololo ya Bokgoni .\n", "2019-11-27 10:02:22,637 Validation result (greedy) at epoch 10, step 12000: bleu: 18.12, loss: 54562.8672, ppl: 7.0282, duration: 102.4934s\n", "2019-11-27 10:02:54,317 Epoch 10 Step: 12100 Batch Loss: 2.404394 Tokens per Sec: 7568, Lr: 0.000300\n", "2019-11-27 10:03:25,970 Epoch 10 Step: 12200 Batch Loss: 2.248341 Tokens per Sec: 7608, Lr: 0.000300\n", "2019-11-27 10:03:57,742 Epoch 10 Step: 12300 Batch Loss: 2.018736 Tokens per Sec: 7687, Lr: 0.000300\n", "2019-11-27 10:04:29,615 Epoch 10 Step: 12400 Batch Loss: 1.784761 Tokens per Sec: 7728, Lr: 0.000300\n", "2019-11-27 10:05:01,326 Epoch 10 Step: 12500 Batch Loss: 2.005187 Tokens per Sec: 7520, Lr: 0.000300\n", "2019-11-27 10:05:33,075 Epoch 10 Step: 12600 Batch Loss: 1.824576 Tokens per Sec: 7574, Lr: 0.000300\n", "2019-11-27 10:06:04,354 Epoch 10 Step: 12700 Batch Loss: 1.806907 Tokens per Sec: 7468, Lr: 0.000300\n", "2019-11-27 10:06:36,035 Epoch 10 Step: 12800 Batch Loss: 2.189607 Tokens per Sec: 7597, Lr: 0.000300\n", "2019-11-27 10:07:07,779 Epoch 10 Step: 12900 Batch Loss: 1.832859 Tokens per Sec: 7453, Lr: 0.000300\n", "2019-11-27 10:07:39,333 Epoch 10 Step: 13000 Batch Loss: 1.791401 Tokens per Sec: 7666, Lr: 0.000300\n", "2019-11-27 10:08:10,998 Epoch 10 Step: 13100 Batch Loss: 2.266404 Tokens per Sec: 7557, Lr: 0.000300\n", "2019-11-27 10:08:41,010 Epoch 10: total training loss 2712.23\n", "2019-11-27 10:08:41,010 EPOCH 11\n", "2019-11-27 10:08:43,479 Epoch 11 Step: 13200 Batch Loss: 1.877542 Tokens per Sec: 7680, Lr: 0.000300\n", "2019-11-27 10:09:14,701 Epoch 11 Step: 13300 Batch Loss: 1.868406 Tokens per Sec: 7448, Lr: 0.000300\n", "2019-11-27 10:09:46,358 Epoch 11 Step: 13400 Batch Loss: 2.316263 Tokens per Sec: 7639, Lr: 0.000300\n", "2019-11-27 10:10:18,219 Epoch 11 Step: 13500 Batch Loss: 1.762514 Tokens per Sec: 7666, Lr: 0.000300\n", "2019-11-27 10:10:49,659 Epoch 11 Step: 13600 Batch Loss: 2.081477 Tokens per Sec: 7610, Lr: 0.000300\n", "2019-11-27 10:11:21,506 Epoch 11 Step: 13700 Batch Loss: 1.984580 Tokens per Sec: 7656, Lr: 0.000300\n", "2019-11-27 10:11:53,265 Epoch 11 Step: 13800 Batch Loss: 1.803510 Tokens per Sec: 7564, Lr: 0.000300\n", "2019-11-27 10:12:25,673 Epoch 11 Step: 13900 Batch Loss: 2.148779 Tokens per Sec: 7697, Lr: 0.000300\n", "2019-11-27 10:12:57,369 Epoch 11 Step: 14000 Batch Loss: 2.024479 Tokens per Sec: 7501, Lr: 0.000300\n", "2019-11-27 10:14:40,022 Hooray! New best validation result [ppl]!\n", "2019-11-27 10:14:40,022 Saving new checkpoint.\n", "2019-11-27 10:14:40,348 Example #0\n", "2019-11-27 10:14:40,349 \tSource: The services are free in South Africa\n", "2019-11-27 10:14:40,349 \tReference: Mo Aforikaborwa , ditirelo di neelwa kwa ntle ga go duelelwa .\n", "2019-11-27 10:14:40,349 \tHypothesis: Ditirelo di gololesegile mo Aforikaborwa\n", "2019-11-27 10:14:40,349 Example #1\n", "2019-11-27 10:14:40,349 \tSource: ( 7 ) A person or community dispossessed of property after 19 June 1913 as a result of past racially discriminatory laws or practices is entitled , to the extent provided by an Act of Parliament , either to restitution of that property or to equitable redress .\n", "2019-11-27 10:14:40,350 \tReference: ( 7 ) Motho kgotsa setlhopha sa magae se se amogilweng dithoto morago ga la 19 Seetebosigo 1913 ka ntlha ya melao ya kgale ya tlhotlhololo kgotsa ditlwaelo , se tshwanelwa , go ya ka thulaganyetso ya Molao wa Palamente go busediwa dithoto tseo , kgotsa tshiamisetso e e e kalekanang .\n", "2019-11-27 10:14:40,350 \tHypothesis: ( 7 ) Motho kgotsa baagi ba ba neng ba gatisiwa go feta , morago ga 19 Seetebosigo 1913 ka ntlha ya melao e e fetileng kgotsa mekgwa e e sa tlwaelegang , e e neelwang ke Molao wa Palamente , go ya ka Molao wa Palamente , go ya ka tshwanelo kgotsa go sa lekalekane .\n", "2019-11-27 10:14:40,350 Example #2\n", "2019-11-27 10:14:40,350 \tSource: ruling\n", "2019-11-27 10:14:40,350 \tReference: tshwetsotaolo\n", "2019-11-27 10:14:40,350 \tHypothesis: go tsosolosa\n", "2019-11-27 10:14:40,352 Example #3\n", "2019-11-27 10:14:40,352 \tSource: Skills Development course .\n", "2019-11-27 10:14:40,353 \tReference: Khoso ya Tlhabololo ya Bokgoni .\n", "2019-11-27 10:14:40,353 \tHypothesis: Khoso ya Tlhabololo ya Metswedi .\n", "2019-11-27 10:14:40,353 Validation result (greedy) at epoch 11, step 14000: bleu: 19.95, loss: 51923.9336, ppl: 6.3957, duration: 102.9831s\n", "2019-11-27 10:15:11,980 Epoch 11 Step: 14100 Batch Loss: 2.417223 Tokens per Sec: 7490, Lr: 0.000300\n", "2019-11-27 10:15:44,125 Epoch 11 Step: 14200 Batch Loss: 1.973081 Tokens per Sec: 7575, Lr: 0.000300\n", "2019-11-27 10:16:16,132 Epoch 11 Step: 14300 Batch Loss: 2.249529 Tokens per Sec: 7636, Lr: 0.000300\n", "2019-11-27 10:16:47,861 Epoch 11 Step: 14400 Batch Loss: 1.978169 Tokens per Sec: 7450, Lr: 0.000300\n", "2019-11-27 10:17:19,677 Epoch 11 Step: 14500 Batch Loss: 2.042983 Tokens per Sec: 7581, Lr: 0.000300\n", "2019-11-27 10:17:23,081 Epoch 11: total training loss 2630.12\n", "2019-11-27 10:17:23,082 EPOCH 12\n", "2019-11-27 10:17:51,364 Epoch 12 Step: 14600 Batch Loss: 1.636456 Tokens per Sec: 7402, Lr: 0.000300\n", "2019-11-27 10:18:23,307 Epoch 12 Step: 14700 Batch Loss: 2.017878 Tokens per Sec: 7562, Lr: 0.000300\n", "2019-11-27 10:18:55,438 Epoch 12 Step: 14800 Batch Loss: 2.079649 Tokens per Sec: 7552, Lr: 0.000300\n", "2019-11-27 10:19:27,329 Epoch 12 Step: 14900 Batch Loss: 1.765992 Tokens per Sec: 7635, Lr: 0.000300\n", "2019-11-27 10:19:59,115 Epoch 12 Step: 15000 Batch Loss: 1.675653 Tokens per Sec: 7603, Lr: 0.000300\n", "2019-11-27 10:20:30,569 Epoch 12 Step: 15100 Batch Loss: 1.720103 Tokens per Sec: 7534, Lr: 0.000300\n", "2019-11-27 10:21:02,352 Epoch 12 Step: 15200 Batch Loss: 1.980757 Tokens per Sec: 7589, Lr: 0.000300\n", "2019-11-27 10:21:33,757 Epoch 12 Step: 15300 Batch Loss: 1.821656 Tokens per Sec: 7602, Lr: 0.000300\n", "2019-11-27 10:22:05,266 Epoch 12 Step: 15400 Batch Loss: 1.732499 Tokens per Sec: 7529, Lr: 0.000300\n", "2019-11-27 10:22:36,688 Epoch 12 Step: 15500 Batch Loss: 1.815525 Tokens per Sec: 7527, Lr: 0.000300\n", "2019-11-27 10:23:08,637 Epoch 12 Step: 15600 Batch Loss: 2.088042 Tokens per Sec: 7632, Lr: 0.000300\n", "2019-11-27 10:23:40,136 Epoch 12 Step: 15700 Batch Loss: 2.095356 Tokens per Sec: 7501, Lr: 0.000300\n", "2019-11-27 10:24:12,288 Epoch 12 Step: 15800 Batch Loss: 2.289366 Tokens per Sec: 7592, Lr: 0.000300\n", "2019-11-27 10:24:23,072 Epoch 12: total training loss 2563.41\n", "2019-11-27 10:24:23,072 EPOCH 13\n", "2019-11-27 10:24:44,559 Epoch 13 Step: 15900 Batch Loss: 1.635685 Tokens per Sec: 7493, Lr: 0.000300\n", "2019-11-27 10:25:16,008 Epoch 13 Step: 16000 Batch Loss: 2.254515 Tokens per Sec: 7504, Lr: 0.000300\n", "2019-11-27 10:26:58,590 Hooray! New best validation result [ppl]!\n", "2019-11-27 10:26:58,591 Saving new checkpoint.\n", "2019-11-27 10:26:58,917 Example #0\n", "2019-11-27 10:26:58,918 \tSource: The services are free in South Africa\n", "2019-11-27 10:26:58,918 \tReference: Mo Aforikaborwa , ditirelo di neelwa kwa ntle ga go duelelwa .\n", "2019-11-27 10:26:58,918 \tHypothesis: Ditirelo di gololesegile mo Aforikaborwa\n", "2019-11-27 10:26:58,918 Example #1\n", "2019-11-27 10:26:58,919 \tSource: ( 7 ) A person or community dispossessed of property after 19 June 1913 as a result of past racially discriminatory laws or practices is entitled , to the extent provided by an Act of Parliament , either to restitution of that property or to equitable redress .\n", "2019-11-27 10:26:58,919 \tReference: ( 7 ) Motho kgotsa setlhopha sa magae se se amogilweng dithoto morago ga la 19 Seetebosigo 1913 ka ntlha ya melao ya kgale ya tlhotlhololo kgotsa ditlwaelo , se tshwanelwa , go ya ka thulaganyetso ya Molao wa Palamente go busediwa dithoto tseo , kgotsa tshiamisetso e e e kalekanang .\n", "2019-11-27 10:26:58,919 \tHypothesis: ( 7 ) Motho kgotsa baagi ba ba neng ba ganwa ka thoto morago ga 19 Seetebosigo 1913 ka ntlha ya melao e e fetileng ya melao e e sa tlwaelegang kgotsa e e sa siamang , e e neilweng ke Molao wa Palamente , go ya ka tlhokego ya go se lekalekane kgotsa go sa siama .\n", "2019-11-27 10:26:58,919 Example #2\n", "2019-11-27 10:26:58,919 \tSource: ruling\n", "2019-11-27 10:26:58,920 \tReference: tshwetsotaolo\n", "2019-11-27 10:26:58,920 \tHypothesis: go kgoreletsa\n", "2019-11-27 10:26:58,920 Example #3\n", "2019-11-27 10:26:58,920 \tSource: Skills Development course .\n", "2019-11-27 10:26:58,920 \tReference: Khoso ya Tlhabololo ya Bokgoni .\n", "2019-11-27 10:26:58,920 \tHypothesis: Khoso ya Tlhabololo ya Bokgoni .\n", "2019-11-27 10:26:58,921 Validation result (greedy) at epoch 13, step 16000: bleu: 20.94, loss: 50232.1211, ppl: 6.0204, duration: 102.9131s\n", "2019-11-27 10:27:30,524 Epoch 13 Step: 16100 Batch Loss: 1.799220 Tokens per Sec: 7723, Lr: 0.000300\n", "2019-11-27 10:28:02,460 Epoch 13 Step: 16200 Batch Loss: 1.800014 Tokens per Sec: 7687, Lr: 0.000300\n", "2019-11-27 10:28:34,056 Epoch 13 Step: 16300 Batch Loss: 2.156416 Tokens per Sec: 7383, Lr: 0.000300\n", "2019-11-27 10:29:05,997 Epoch 13 Step: 16400 Batch Loss: 2.017004 Tokens per Sec: 7475, Lr: 0.000300\n", "2019-11-27 10:29:37,561 Epoch 13 Step: 16500 Batch Loss: 1.962373 Tokens per Sec: 7548, Lr: 0.000300\n", "2019-11-27 10:30:09,603 Epoch 13 Step: 16600 Batch Loss: 2.057327 Tokens per Sec: 7583, Lr: 0.000300\n", "2019-11-27 10:30:41,406 Epoch 13 Step: 16700 Batch Loss: 1.954224 Tokens per Sec: 7604, Lr: 0.000300\n", "2019-11-27 10:31:12,949 Epoch 13 Step: 16800 Batch Loss: 1.896105 Tokens per Sec: 7484, Lr: 0.000300\n", "2019-11-27 10:31:44,445 Epoch 13 Step: 16900 Batch Loss: 1.824259 Tokens per Sec: 7526, Lr: 0.000300\n", "2019-11-27 10:32:16,502 Epoch 13 Step: 17000 Batch Loss: 1.966074 Tokens per Sec: 7692, Lr: 0.000300\n", "2019-11-27 10:32:48,699 Epoch 13 Step: 17100 Batch Loss: 1.764410 Tokens per Sec: 7595, Lr: 0.000300\n", "2019-11-27 10:33:05,897 Epoch 13: total training loss 2497.23\n", "2019-11-27 10:33:05,897 EPOCH 14\n", "2019-11-27 10:33:20,599 Epoch 14 Step: 17200 Batch Loss: 1.955580 Tokens per Sec: 7558, Lr: 0.000300\n", "2019-11-27 10:33:52,359 Epoch 14 Step: 17300 Batch Loss: 2.100826 Tokens per Sec: 7562, Lr: 0.000300\n", "2019-11-27 10:34:23,930 Epoch 14 Step: 17400 Batch Loss: 1.742390 Tokens per Sec: 7489, Lr: 0.000300\n", "2019-11-27 10:34:56,234 Epoch 14 Step: 17500 Batch Loss: 1.751768 Tokens per Sec: 7639, Lr: 0.000300\n", "2019-11-27 10:35:27,897 Epoch 14 Step: 17600 Batch Loss: 1.804232 Tokens per Sec: 7543, Lr: 0.000300\n", "2019-11-27 10:35:59,622 Epoch 14 Step: 17700 Batch Loss: 1.642770 Tokens per Sec: 7551, Lr: 0.000300\n", "2019-11-27 10:36:31,449 Epoch 14 Step: 17800 Batch Loss: 1.705540 Tokens per Sec: 7609, Lr: 0.000300\n", "2019-11-27 10:37:03,293 Epoch 14 Step: 17900 Batch Loss: 1.629975 Tokens per Sec: 7677, Lr: 0.000300\n", "2019-11-27 10:37:34,929 Epoch 14 Step: 18000 Batch Loss: 1.935024 Tokens per Sec: 7533, Lr: 0.000300\n", "2019-11-27 10:39:17,550 Hooray! New best validation result [ppl]!\n", "2019-11-27 10:39:17,550 Saving new checkpoint.\n", "2019-11-27 10:39:17,870 Example #0\n", "2019-11-27 10:39:17,870 \tSource: The services are free in South Africa\n", "2019-11-27 10:39:17,870 \tReference: Mo Aforikaborwa , ditirelo di neelwa kwa ntle ga go duelelwa .\n", "2019-11-27 10:39:17,871 \tHypothesis: Ditirelo di gololesegile mo Aforikaborwa\n", "2019-11-27 10:39:17,871 Example #1\n", "2019-11-27 10:39:17,871 \tSource: ( 7 ) A person or community dispossessed of property after 19 June 1913 as a result of past racially discriminatory laws or practices is entitled , to the extent provided by an Act of Parliament , either to restitution of that property or to equitable redress .\n", "2019-11-27 10:39:17,871 \tReference: ( 7 ) Motho kgotsa setlhopha sa magae se se amogilweng dithoto morago ga la 19 Seetebosigo 1913 ka ntlha ya melao ya kgale ya tlhotlhololo kgotsa ditlwaelo , se tshwanelwa , go ya ka thulaganyetso ya Molao wa Palamente go busediwa dithoto tseo , kgotsa tshiamisetso e e e kalekanang .\n", "2019-11-27 10:39:17,871 \tHypothesis: ( 7 ) Motho kgotsa baagi ba ba tlhophilweng ke dithoto morago ga 19 Seetebosigo 1913 ka ntlha ya melao e e fetileng kgotsa mekgwa e e sa siamang , e e neelwang ke Molao wa Palamente , e ka nna ya tshoganyetso ya go lekana kgotsa go sa lekalekane .\n", "2019-11-27 10:39:17,871 Example #2\n", "2019-11-27 10:39:17,872 \tSource: ruling\n", "2019-11-27 10:39:17,872 \tReference: tshwetsotaolo\n", "2019-11-27 10:39:17,872 \tHypothesis: go tsala\n", "2019-11-27 10:39:17,872 Example #3\n", "2019-11-27 10:39:17,872 \tSource: Skills Development course .\n", "2019-11-27 10:39:17,872 \tReference: Khoso ya Tlhabololo ya Bokgoni .\n", "2019-11-27 10:39:17,873 \tHypothesis: Khoso ya Tlhabololo ya Bokgoni .\n", "2019-11-27 10:39:17,873 Validation result (greedy) at epoch 14, step 18000: bleu: 22.27, loss: 48756.1445, ppl: 5.7111, duration: 102.9438s\n", "2019-11-27 10:39:50,196 Epoch 14 Step: 18100 Batch Loss: 1.698276 Tokens per Sec: 7619, Lr: 0.000300\n", "2019-11-27 10:40:22,534 Epoch 14 Step: 18200 Batch Loss: 2.008240 Tokens per Sec: 7559, Lr: 0.000300\n", "2019-11-27 10:40:54,867 Epoch 14 Step: 18300 Batch Loss: 1.484115 Tokens per Sec: 7309, Lr: 0.000300\n", "2019-11-27 10:41:27,030 Epoch 14 Step: 18400 Batch Loss: 1.710978 Tokens per Sec: 7416, Lr: 0.000300\n", "2019-11-27 10:41:49,900 Epoch 14: total training loss 2432.72\n", "2019-11-27 10:41:49,900 EPOCH 15\n", "2019-11-27 10:41:59,366 Epoch 15 Step: 18500 Batch Loss: 1.564605 Tokens per Sec: 7447, Lr: 0.000300\n", "2019-11-27 10:42:31,608 Epoch 15 Step: 18600 Batch Loss: 1.870029 Tokens per Sec: 7546, Lr: 0.000300\n", "2019-11-27 10:43:03,952 Epoch 15 Step: 18700 Batch Loss: 1.562552 Tokens per Sec: 7526, Lr: 0.000300\n", "2019-11-27 10:43:36,297 Epoch 15 Step: 18800 Batch Loss: 1.647856 Tokens per Sec: 7464, Lr: 0.000300\n", "2019-11-27 10:44:08,447 Epoch 15 Step: 18900 Batch Loss: 2.012269 Tokens per Sec: 7365, Lr: 0.000300\n", "2019-11-27 10:44:40,253 Epoch 15 Step: 19000 Batch Loss: 1.550156 Tokens per Sec: 7427, Lr: 0.000300\n", "2019-11-27 10:45:12,580 Epoch 15 Step: 19100 Batch Loss: 1.667629 Tokens per Sec: 7396, Lr: 0.000300\n", "2019-11-27 10:45:44,714 Epoch 15 Step: 19200 Batch Loss: 2.012200 Tokens per Sec: 7569, Lr: 0.000300\n", "2019-11-27 10:46:16,679 Epoch 15 Step: 19300 Batch Loss: 1.848879 Tokens per Sec: 7580, Lr: 0.000300\n", "2019-11-27 10:46:48,995 Epoch 15 Step: 19400 Batch Loss: 1.588713 Tokens per Sec: 7588, Lr: 0.000300\n", "2019-11-27 10:47:20,859 Epoch 15 Step: 19500 Batch Loss: 1.834715 Tokens per Sec: 7503, Lr: 0.000300\n", "2019-11-27 10:47:51,956 Epoch 15 Step: 19600 Batch Loss: 2.047431 Tokens per Sec: 7437, Lr: 0.000300\n", "2019-11-27 10:48:23,797 Epoch 15 Step: 19700 Batch Loss: 1.814978 Tokens per Sec: 7666, Lr: 0.000300\n", "2019-11-27 10:48:52,563 Epoch 15: total training loss 2388.55\n", "2019-11-27 10:48:52,564 EPOCH 16\n", "2019-11-27 10:48:56,027 Epoch 16 Step: 19800 Batch Loss: 1.946938 Tokens per Sec: 7486, Lr: 0.000300\n", "2019-11-27 10:49:28,489 Epoch 16 Step: 19900 Batch Loss: 1.808570 Tokens per Sec: 7592, Lr: 0.000300\n", "2019-11-27 10:49:59,744 Epoch 16 Step: 20000 Batch Loss: 1.818748 Tokens per Sec: 7433, Lr: 0.000300\n", "2019-11-27 10:51:42,036 Hooray! New best validation result [ppl]!\n", "2019-11-27 10:51:42,037 Saving new checkpoint.\n", "2019-11-27 10:51:42,334 Example #0\n", "2019-11-27 10:51:42,335 \tSource: The services are free in South Africa\n", "2019-11-27 10:51:42,335 \tReference: Mo Aforikaborwa , ditirelo di neelwa kwa ntle ga go duelelwa .\n", "2019-11-27 10:51:42,335 \tHypothesis: Ditirelo di gololesegile mo Aforikaborwa\n", "2019-11-27 10:51:42,335 Example #1\n", "2019-11-27 10:51:42,335 \tSource: ( 7 ) A person or community dispossessed of property after 19 June 1913 as a result of past racially discriminatory laws or practices is entitled , to the extent provided by an Act of Parliament , either to restitution of that property or to equitable redress .\n", "2019-11-27 10:51:42,336 \tReference: ( 7 ) Motho kgotsa setlhopha sa magae se se amogilweng dithoto morago ga la 19 Seetebosigo 1913 ka ntlha ya melao ya kgale ya tlhotlhololo kgotsa ditlwaelo , se tshwanelwa , go ya ka thulaganyetso ya Molao wa Palamente go busediwa dithoto tseo , kgotsa tshiamisetso e e e kalekanang .\n", "2019-11-27 10:51:42,336 \tHypothesis: ( 7 ) Motho kgotsa baagi ba ba ganetsang thoto morago ga 19 Seetebosigo 1913 ka ntlha ya melao e e fetileng kgotsa mekgwa e e sa tlwaelegang , e e newang ke Molao wa Palamente , e ka nna ya tsholwa kgotsa e e sa siamang .\n", "2019-11-27 10:51:42,336 Example #2\n", "2019-11-27 10:51:42,336 \tSource: ruling\n", "2019-11-27 10:51:42,336 \tReference: tshwetsotaolo\n", "2019-11-27 10:51:42,336 \tHypothesis: go kgoreletsa\n", "2019-11-27 10:51:42,337 Example #3\n", "2019-11-27 10:51:42,337 \tSource: Skills Development course .\n", "2019-11-27 10:51:42,337 \tReference: Khoso ya Tlhabololo ya Bokgoni .\n", "2019-11-27 10:51:42,337 \tHypothesis: Khoso ya Tlhabololo ya Bokgoni .\n", "2019-11-27 10:51:42,337 Validation result (greedy) at epoch 16, step 20000: bleu: 22.95, loss: 47692.8711, ppl: 5.4982, duration: 102.5931s\n", "2019-11-27 10:52:13,913 Epoch 16 Step: 20100 Batch Loss: 1.471170 Tokens per Sec: 7545, Lr: 0.000300\n", "2019-11-27 10:52:46,286 Epoch 16 Step: 20200 Batch Loss: 1.760224 Tokens per Sec: 7418, Lr: 0.000300\n", "2019-11-27 10:53:18,203 Epoch 16 Step: 20300 Batch Loss: 1.713607 Tokens per Sec: 7442, Lr: 0.000300\n", "2019-11-27 10:53:49,954 Epoch 16 Step: 20400 Batch Loss: 1.606376 Tokens per Sec: 7514, Lr: 0.000300\n", "2019-11-27 10:54:22,372 Epoch 16 Step: 20500 Batch Loss: 2.052690 Tokens per Sec: 7614, Lr: 0.000300\n", "2019-11-27 10:54:55,129 Epoch 16 Step: 20600 Batch Loss: 1.872383 Tokens per Sec: 7480, Lr: 0.000300\n", "2019-11-27 10:55:27,162 Epoch 16 Step: 20700 Batch Loss: 1.830910 Tokens per Sec: 7514, Lr: 0.000300\n", "2019-11-27 10:55:59,375 Epoch 16 Step: 20800 Batch Loss: 1.612850 Tokens per Sec: 7551, Lr: 0.000300\n", "2019-11-27 10:56:31,079 Epoch 16 Step: 20900 Batch Loss: 1.787618 Tokens per Sec: 7344, Lr: 0.000300\n", "2019-11-27 10:57:03,230 Epoch 16 Step: 21000 Batch Loss: 1.864138 Tokens per Sec: 7559, Lr: 0.000300\n", "2019-11-27 10:57:35,318 Epoch 16 Step: 21100 Batch Loss: 1.619395 Tokens per Sec: 7463, Lr: 0.000300\n", "2019-11-27 10:57:38,562 Epoch 16: total training loss 2348.81\n", "2019-11-27 10:57:38,562 EPOCH 17\n", "2019-11-27 10:58:07,158 Epoch 17 Step: 21200 Batch Loss: 1.727051 Tokens per Sec: 7293, Lr: 0.000300\n", "2019-11-27 10:58:39,103 Epoch 17 Step: 21300 Batch Loss: 1.757831 Tokens per Sec: 7428, Lr: 0.000300\n", "2019-11-27 10:59:11,087 Epoch 17 Step: 21400 Batch Loss: 1.595175 Tokens per Sec: 7568, Lr: 0.000300\n", "2019-11-27 10:59:43,126 Epoch 17 Step: 21500 Batch Loss: 1.459811 Tokens per Sec: 7578, Lr: 0.000300\n", "2019-11-27 11:00:15,041 Epoch 17 Step: 21600 Batch Loss: 1.863920 Tokens per Sec: 7601, Lr: 0.000300\n", "2019-11-27 11:00:47,025 Epoch 17 Step: 21700 Batch Loss: 1.819965 Tokens per Sec: 7546, Lr: 0.000300\n", "2019-11-27 11:01:18,733 Epoch 17 Step: 21800 Batch Loss: 1.921723 Tokens per Sec: 7647, Lr: 0.000300\n", "2019-11-27 11:01:50,673 Epoch 17 Step: 21900 Batch Loss: 2.005459 Tokens per Sec: 7557, Lr: 0.000300\n", "2019-11-27 11:02:22,836 Epoch 17 Step: 22000 Batch Loss: 1.897895 Tokens per Sec: 7453, Lr: 0.000300\n", "2019-11-27 11:04:05,977 Hooray! New best validation result [ppl]!\n", "2019-11-27 11:04:05,977 Saving new checkpoint.\n", "2019-11-27 11:04:06,270 Example #0\n", "2019-11-27 11:04:06,271 \tSource: The services are free in South Africa\n", "2019-11-27 11:04:06,271 \tReference: Mo Aforikaborwa , ditirelo di neelwa kwa ntle ga go duelelwa .\n", "2019-11-27 11:04:06,271 \tHypothesis: Ditirelo di gololesegile mo Aforikaborwa\n", "2019-11-27 11:04:06,271 Example #1\n", "2019-11-27 11:04:06,272 \tSource: ( 7 ) A person or community dispossessed of property after 19 June 1913 as a result of past racially discriminatory laws or practices is entitled , to the extent provided by an Act of Parliament , either to restitution of that property or to equitable redress .\n", "2019-11-27 11:04:06,272 \tReference: ( 7 ) Motho kgotsa setlhopha sa magae se se amogilweng dithoto morago ga la 19 Seetebosigo 1913 ka ntlha ya melao ya kgale ya tlhotlhololo kgotsa ditlwaelo , se tshwanelwa , go ya ka thulaganyetso ya Molao wa Palamente go busediwa dithoto tseo , kgotsa tshiamisetso e e e kalekanang .\n", "2019-11-27 11:04:06,272 \tHypothesis: ( 7 ) Motho kgotsa baagi ba ba sa tshwanelang go tsewa tsia morago ga 19 Seetebosigo 1913 ka ntlha ya melao e e fetileng kgotsa mekgwatiriso e e sa siamang , e e neilweng ke Molao wa Palamente , e ka nna ya go dira gore go nne le tshwanelo kgotsa go sa lekalekane .\n", "2019-11-27 11:04:06,272 Example #2\n", "2019-11-27 11:04:06,273 \tSource: ruling\n", "2019-11-27 11:04:06,273 \tReference: tshwetsotaolo\n", "2019-11-27 11:04:06,273 \tHypothesis: go kgoreletsa\n", "2019-11-27 11:04:06,273 Example #3\n", "2019-11-27 11:04:06,273 \tSource: Skills Development course .\n", "2019-11-27 11:04:06,273 \tReference: Khoso ya Tlhabololo ya Bokgoni .\n", "2019-11-27 11:04:06,273 \tHypothesis: Khoso ya Tlhabololo ya Bokgoni .\n", "2019-11-27 11:04:06,274 Validation result (greedy) at epoch 17, step 22000: bleu: 23.78, loss: 46253.8672, ppl: 5.2226, duration: 103.4373s\n", "2019-11-27 11:04:38,457 Epoch 17 Step: 22100 Batch Loss: 1.917826 Tokens per Sec: 7523, Lr: 0.000300\n", "2019-11-27 11:05:10,367 Epoch 17 Step: 22200 Batch Loss: 1.806039 Tokens per Sec: 7446, Lr: 0.000300\n", "2019-11-27 11:05:42,333 Epoch 17 Step: 22300 Batch Loss: 1.848647 Tokens per Sec: 7548, Lr: 0.000300\n", "2019-11-27 11:06:14,534 Epoch 17 Step: 22400 Batch Loss: 1.506181 Tokens per Sec: 7419, Lr: 0.000300\n", "2019-11-27 11:06:24,873 Epoch 17: total training loss 2311.35\n", "2019-11-27 11:06:24,873 EPOCH 18\n", "2019-11-27 11:06:47,139 Epoch 18 Step: 22500 Batch Loss: 1.779034 Tokens per Sec: 7350, Lr: 0.000300\n", "2019-11-27 11:07:18,772 Epoch 18 Step: 22600 Batch Loss: 1.559733 Tokens per Sec: 7424, Lr: 0.000300\n", "2019-11-27 11:07:50,748 Epoch 18 Step: 22700 Batch Loss: 1.782120 Tokens per Sec: 7518, Lr: 0.000300\n", "2019-11-27 11:08:22,754 Epoch 18 Step: 22800 Batch Loss: 1.600652 Tokens per Sec: 7624, Lr: 0.000300\n", "2019-11-27 11:08:54,755 Epoch 18 Step: 22900 Batch Loss: 1.825630 Tokens per Sec: 7754, Lr: 0.000300\n", "2019-11-27 11:09:26,793 Epoch 18 Step: 23000 Batch Loss: 1.452477 Tokens per Sec: 7518, Lr: 0.000300\n", "2019-11-27 11:09:58,890 Epoch 18 Step: 23100 Batch Loss: 1.702956 Tokens per Sec: 7650, Lr: 0.000300\n", "2019-11-27 11:10:30,703 Epoch 18 Step: 23200 Batch Loss: 1.822166 Tokens per Sec: 7439, Lr: 0.000300\n", "2019-11-27 11:11:02,543 Epoch 18 Step: 23300 Batch Loss: 1.838295 Tokens per Sec: 7458, Lr: 0.000300\n", "2019-11-27 11:11:34,488 Epoch 18 Step: 23400 Batch Loss: 1.939657 Tokens per Sec: 7579, Lr: 0.000300\n", "2019-11-27 11:12:06,340 Epoch 18 Step: 23500 Batch Loss: 1.736556 Tokens per Sec: 7753, Lr: 0.000300\n", "2019-11-27 11:12:37,871 Epoch 18 Step: 23600 Batch Loss: 1.595957 Tokens per Sec: 7569, Lr: 0.000300\n", "2019-11-27 11:13:09,425 Epoch 18 Step: 23700 Batch Loss: 1.598570 Tokens per Sec: 7546, Lr: 0.000300\n", "2019-11-27 11:13:24,962 Epoch 18: total training loss 2262.74\n", "2019-11-27 11:13:24,962 EPOCH 19\n", "2019-11-27 11:13:41,390 Epoch 19 Step: 23800 Batch Loss: 1.451131 Tokens per Sec: 7539, Lr: 0.000300\n", "2019-11-27 11:14:13,126 Epoch 19 Step: 23900 Batch Loss: 1.502333 Tokens per Sec: 7526, Lr: 0.000300\n", "2019-11-27 11:14:44,742 Epoch 19 Step: 24000 Batch Loss: 1.729751 Tokens per Sec: 7475, Lr: 0.000300\n", "2019-11-27 11:16:27,388 Hooray! New best validation result [ppl]!\n", "2019-11-27 11:16:27,389 Saving new checkpoint.\n", "2019-11-27 11:16:27,702 Example #0\n", "2019-11-27 11:16:27,702 \tSource: The services are free in South Africa\n", "2019-11-27 11:16:27,703 \tReference: Mo Aforikaborwa , ditirelo di neelwa kwa ntle ga go duelelwa .\n", "2019-11-27 11:16:27,703 \tHypothesis: Ditirelo di gololesegile mo Aforikaborwa\n", "2019-11-27 11:16:27,703 Example #1\n", "2019-11-27 11:16:27,703 \tSource: ( 7 ) A person or community dispossessed of property after 19 June 1913 as a result of past racially discriminatory laws or practices is entitled , to the extent provided by an Act of Parliament , either to restitution of that property or to equitable redress .\n", "2019-11-27 11:16:27,703 \tReference: ( 7 ) Motho kgotsa setlhopha sa magae se se amogilweng dithoto morago ga la 19 Seetebosigo 1913 ka ntlha ya melao ya kgale ya tlhotlhololo kgotsa ditlwaelo , se tshwanelwa , go ya ka thulaganyetso ya Molao wa Palamente go busediwa dithoto tseo , kgotsa tshiamisetso e e e kalekanang .\n", "2019-11-27 11:16:27,703 \tHypothesis: ( 7 ) Motho kgotsa baagi ba ba sa kgetholegeng dithoto morago ga 19 Seetebosigo 1913 ka ntlha ya melao ya bosiamisi kgotsa mekgwa e e sa siamang , go ya ka Molao wa Palamente , go ka nna le tshwanelo ya go sa lekalekane kgotsa go sa lekalekane .\n", "2019-11-27 11:16:27,703 Example #2\n", "2019-11-27 11:16:27,704 \tSource: ruling\n", "2019-11-27 11:16:27,704 \tReference: tshwetsotaolo\n", "2019-11-27 11:16:27,704 \tHypothesis: sekgoreletsi\n", "2019-11-27 11:16:27,704 Example #3\n", "2019-11-27 11:16:27,704 \tSource: Skills Development course .\n", "2019-11-27 11:16:27,704 \tReference: Khoso ya Tlhabololo ya Bokgoni .\n", "2019-11-27 11:16:27,705 \tHypothesis: Khoso ya Tlhabololo ya Bokgoni .\n", "2019-11-27 11:16:27,705 Validation result (greedy) at epoch 19, step 24000: bleu: 24.48, loss: 45594.0430, ppl: 5.1008, duration: 102.9621s\n", "2019-11-27 11:16:59,481 Epoch 19 Step: 24100 Batch Loss: 1.920007 Tokens per Sec: 7483, Lr: 0.000300\n", "2019-11-27 11:17:32,184 Epoch 19 Step: 24200 Batch Loss: 1.830553 Tokens per Sec: 7720, Lr: 0.000300\n", "2019-11-27 11:18:04,153 Epoch 19 Step: 24300 Batch Loss: 1.586019 Tokens per Sec: 7231, Lr: 0.000300\n", "2019-11-27 11:18:36,574 Epoch 19 Step: 24400 Batch Loss: 1.811028 Tokens per Sec: 7378, Lr: 0.000300\n", "2019-11-27 11:19:09,032 Epoch 19 Step: 24500 Batch Loss: 1.814666 Tokens per Sec: 7473, Lr: 0.000300\n", "2019-11-27 11:19:41,611 Epoch 19 Step: 24600 Batch Loss: 1.702794 Tokens per Sec: 7466, Lr: 0.000300\n", "2019-11-27 11:20:13,470 Epoch 19 Step: 24700 Batch Loss: 1.752976 Tokens per Sec: 7490, Lr: 0.000300\n", "2019-11-27 11:20:45,655 Epoch 19 Step: 24800 Batch Loss: 1.622045 Tokens per Sec: 7553, Lr: 0.000300\n", "2019-11-27 11:21:17,874 Epoch 19 Step: 24900 Batch Loss: 1.834907 Tokens per Sec: 7466, Lr: 0.000300\n", "2019-11-27 11:21:49,970 Epoch 19 Step: 25000 Batch Loss: 1.772614 Tokens per Sec: 7409, Lr: 0.000300\n", "2019-11-27 11:22:12,760 Epoch 19: total training loss 2240.00\n", "2019-11-27 11:22:12,761 EPOCH 20\n", "2019-11-27 11:22:22,000 Epoch 20 Step: 25100 Batch Loss: 1.511060 Tokens per Sec: 7009, Lr: 0.000300\n", "2019-11-27 11:22:54,028 Epoch 20 Step: 25200 Batch Loss: 1.490772 Tokens per Sec: 7478, Lr: 0.000300\n", "2019-11-27 11:23:26,252 Epoch 20 Step: 25300 Batch Loss: 1.757785 Tokens per Sec: 7477, Lr: 0.000300\n", "2019-11-27 11:23:57,993 Epoch 20 Step: 25400 Batch Loss: 1.479680 Tokens per Sec: 7485, Lr: 0.000300\n", "2019-11-27 11:24:29,825 Epoch 20 Step: 25500 Batch Loss: 1.720403 Tokens per Sec: 7624, Lr: 0.000300\n", "2019-11-27 11:25:01,497 Epoch 20 Step: 25600 Batch Loss: 1.459862 Tokens per Sec: 7434, Lr: 0.000300\n", "2019-11-27 11:25:33,464 Epoch 20 Step: 25700 Batch Loss: 1.913007 Tokens per Sec: 7519, Lr: 0.000300\n", "2019-11-27 11:26:05,985 Epoch 20 Step: 25800 Batch Loss: 1.815313 Tokens per Sec: 7522, Lr: 0.000300\n", "2019-11-27 11:26:38,380 Epoch 20 Step: 25900 Batch Loss: 1.474030 Tokens per Sec: 7503, Lr: 0.000300\n", "2019-11-27 11:27:10,643 Epoch 20 Step: 26000 Batch Loss: 1.543901 Tokens per Sec: 7564, Lr: 0.000300\n", "2019-11-27 11:28:54,341 Hooray! New best validation result [ppl]!\n", "2019-11-27 11:28:54,341 Saving new checkpoint.\n", "2019-11-27 11:28:54,629 Example #0\n", "2019-11-27 11:28:54,630 \tSource: The services are free in South Africa\n", "2019-11-27 11:28:54,630 \tReference: Mo Aforikaborwa , ditirelo di neelwa kwa ntle ga go duelelwa .\n", "2019-11-27 11:28:54,630 \tHypothesis: Ditirelo di gololesegile mo Aforikaborwa\n", "2019-11-27 11:28:54,630 Example #1\n", "2019-11-27 11:28:54,631 \tSource: ( 7 ) A person or community dispossessed of property after 19 June 1913 as a result of past racially discriminatory laws or practices is entitled , to the extent provided by an Act of Parliament , either to restitution of that property or to equitable redress .\n", "2019-11-27 11:28:54,631 \tReference: ( 7 ) Motho kgotsa setlhopha sa magae se se amogilweng dithoto morago ga la 19 Seetebosigo 1913 ka ntlha ya melao ya kgale ya tlhotlhololo kgotsa ditlwaelo , se tshwanelwa , go ya ka thulaganyetso ya Molao wa Palamente go busediwa dithoto tseo , kgotsa tshiamisetso e e e kalekanang .\n", "2019-11-27 11:28:54,631 \tHypothesis: ( 7 ) Motho kgotsa baagi ba ba sa kgetholegeng thoto morago ga 19 Seetebosigo 1913 ka ntlha ya melao e e fetileng kgotsa mekgwatsamaiso e e sa siamang , e e neilweng ke Molao wa Palamente , e ka nna go sa siama kgotsa go sa lekalekane le go tlhoka go lekane le le siameng .\n", "2019-11-27 11:28:54,631 Example #2\n", "2019-11-27 11:28:54,631 \tSource: ruling\n", "2019-11-27 11:28:54,631 \tReference: tshwetsotaolo\n", "2019-11-27 11:28:54,631 \tHypothesis: go tsoga\n", "2019-11-27 11:28:54,632 Example #3\n", "2019-11-27 11:28:54,632 \tSource: Skills Development course .\n", "2019-11-27 11:28:54,632 \tReference: Khoso ya Tlhabololo ya Bokgoni .\n", "2019-11-27 11:28:54,632 \tHypothesis: Khoso ya Tlhabololo ya Bokgoni .\n", "2019-11-27 11:28:54,632 Validation result (greedy) at epoch 20, step 26000: bleu: 25.23, loss: 44392.1367, ppl: 4.8864, duration: 103.9891s\n", "2019-11-27 11:29:26,444 Epoch 20 Step: 26100 Batch Loss: 1.437025 Tokens per Sec: 7678, Lr: 0.000300\n", "2019-11-27 11:29:58,086 Epoch 20 Step: 26200 Batch Loss: 1.471626 Tokens per Sec: 7478, Lr: 0.000300\n", "2019-11-27 11:30:30,195 Epoch 20 Step: 26300 Batch Loss: 1.473710 Tokens per Sec: 7580, Lr: 0.000300\n", "2019-11-27 11:30:58,974 Epoch 20: total training loss 2203.41\n", "2019-11-27 11:30:58,975 EPOCH 21\n", "2019-11-27 11:31:01,916 Epoch 21 Step: 26400 Batch Loss: 1.451157 Tokens per Sec: 6789, Lr: 0.000300\n", "2019-11-27 11:31:33,892 Epoch 21 Step: 26500 Batch Loss: 1.702116 Tokens per Sec: 7520, Lr: 0.000300\n", "2019-11-27 11:32:05,932 Epoch 21 Step: 26600 Batch Loss: 1.440231 Tokens per Sec: 7595, Lr: 0.000300\n", "2019-11-27 11:32:37,756 Epoch 21 Step: 26700 Batch Loss: 1.361406 Tokens per Sec: 7614, Lr: 0.000300\n", "2019-11-27 11:33:09,559 Epoch 21 Step: 26800 Batch Loss: 1.813076 Tokens per Sec: 7639, Lr: 0.000300\n", "2019-11-27 11:33:41,257 Epoch 21 Step: 26900 Batch Loss: 1.579151 Tokens per Sec: 7513, Lr: 0.000300\n", "2019-11-27 11:34:12,974 Epoch 21 Step: 27000 Batch Loss: 1.761373 Tokens per Sec: 7501, Lr: 0.000300\n", "2019-11-27 11:34:44,913 Epoch 21 Step: 27100 Batch Loss: 1.555657 Tokens per Sec: 7664, Lr: 0.000300\n", "2019-11-27 11:35:16,369 Epoch 21 Step: 27200 Batch Loss: 1.847673 Tokens per Sec: 7529, Lr: 0.000300\n", "2019-11-27 11:35:47,915 Epoch 21 Step: 27300 Batch Loss: 1.604585 Tokens per Sec: 7372, Lr: 0.000300\n", "2019-11-27 11:36:19,709 Epoch 21 Step: 27400 Batch Loss: 1.457386 Tokens per Sec: 7572, Lr: 0.000300\n", "2019-11-27 11:36:51,831 Epoch 21 Step: 27500 Batch Loss: 1.816985 Tokens per Sec: 7729, Lr: 0.000300\n", "2019-11-27 11:37:23,438 Epoch 21 Step: 27600 Batch Loss: 1.845428 Tokens per Sec: 7581, Lr: 0.000300\n", "2019-11-27 11:37:55,346 Epoch 21 Step: 27700 Batch Loss: 1.957841 Tokens per Sec: 7584, Lr: 0.000300\n", "2019-11-27 11:37:59,074 Epoch 21: total training loss 2177.99\n", "2019-11-27 11:37:59,074 EPOCH 22\n", "2019-11-27 11:38:26,968 Epoch 22 Step: 27800 Batch Loss: 1.690364 Tokens per Sec: 7503, Lr: 0.000300\n", "2019-11-27 11:38:58,418 Epoch 22 Step: 27900 Batch Loss: 1.464409 Tokens per Sec: 7515, Lr: 0.000300\n", "2019-11-27 11:39:29,884 Epoch 22 Step: 28000 Batch Loss: 1.818248 Tokens per Sec: 7574, Lr: 0.000300\n", "2019-11-27 11:41:12,462 Hooray! New best validation result [ppl]!\n", "2019-11-27 11:41:12,462 Saving new checkpoint.\n", "2019-11-27 11:41:12,796 Example #0\n", "2019-11-27 11:41:12,797 \tSource: The services are free in South Africa\n", "2019-11-27 11:41:12,797 \tReference: Mo Aforikaborwa , ditirelo di neelwa kwa ntle ga go duelelwa .\n", "2019-11-27 11:41:12,797 \tHypothesis: Ditirelo di gololesegile mo Aforikaborwa\n", "2019-11-27 11:41:12,797 Example #1\n", "2019-11-27 11:41:12,797 \tSource: ( 7 ) A person or community dispossessed of property after 19 June 1913 as a result of past racially discriminatory laws or practices is entitled , to the extent provided by an Act of Parliament , either to restitution of that property or to equitable redress .\n", "2019-11-27 11:41:12,797 \tReference: ( 7 ) Motho kgotsa setlhopha sa magae se se amogilweng dithoto morago ga la 19 Seetebosigo 1913 ka ntlha ya melao ya kgale ya tlhotlhololo kgotsa ditlwaelo , se tshwanelwa , go ya ka thulaganyetso ya Molao wa Palamente go busediwa dithoto tseo , kgotsa tshiamisetso e e e kalekanang .\n", "2019-11-27 11:41:12,797 \tHypothesis: ( 7 ) Motho kgotsa baagi ba ba neng ba ganwa ka thoto morago ga 19 Seetebosigo 1913 ka ntlha ya melao e e fetileng kgotsa mekgwa e e sa tlwaelegang e tshwanetse go latelwa , go ya ka Molao wa Palamente , e ka nna ya nna le tshiamelo ya go se lekalekane kgotsa go sa lekalekane .\n", "2019-11-27 11:41:12,797 Example #2\n", "2019-11-27 11:41:12,797 \tSource: ruling\n", "2019-11-27 11:41:12,797 \tReference: tshwetsotaolo\n", "2019-11-27 11:41:12,798 \tHypothesis: go goga\n", "2019-11-27 11:41:12,798 Example #3\n", "2019-11-27 11:41:12,798 \tSource: Skills Development course .\n", "2019-11-27 11:41:12,798 \tReference: Khoso ya Tlhabololo ya Bokgoni .\n", "2019-11-27 11:41:12,798 \tHypothesis: Khoso ya Tlhabololo ya Bokgoni .\n", "2019-11-27 11:41:12,798 Validation result (greedy) at epoch 22, step 28000: bleu: 25.46, loss: 44094.1836, ppl: 4.8346, duration: 102.9134s\n", "2019-11-27 11:41:44,254 Epoch 22 Step: 28100 Batch Loss: 1.696291 Tokens per Sec: 7588, Lr: 0.000300\n", "2019-11-27 11:42:16,107 Epoch 22 Step: 28200 Batch Loss: 1.842358 Tokens per Sec: 7666, Lr: 0.000300\n", "2019-11-27 11:42:47,628 Epoch 22 Step: 28300 Batch Loss: 1.701993 Tokens per Sec: 7547, Lr: 0.000300\n", "2019-11-27 11:43:19,367 Epoch 22 Step: 28400 Batch Loss: 1.963636 Tokens per Sec: 7645, Lr: 0.000300\n", "2019-11-27 11:43:51,105 Epoch 22 Step: 28500 Batch Loss: 1.578465 Tokens per Sec: 7657, Lr: 0.000300\n", "2019-11-27 11:44:23,332 Epoch 22 Step: 28600 Batch Loss: 1.279055 Tokens per Sec: 7736, Lr: 0.000300\n", "2019-11-27 11:44:54,882 Epoch 22 Step: 28700 Batch Loss: 1.782416 Tokens per Sec: 7572, Lr: 0.000300\n", "2019-11-27 11:45:26,637 Epoch 22 Step: 28800 Batch Loss: 1.722763 Tokens per Sec: 7516, Lr: 0.000300\n", "2019-11-27 11:45:58,212 Epoch 22 Step: 28900 Batch Loss: 1.561923 Tokens per Sec: 7678, Lr: 0.000300\n", "2019-11-27 11:46:29,841 Epoch 22 Step: 29000 Batch Loss: 1.712644 Tokens per Sec: 7694, Lr: 0.000300\n", "2019-11-27 11:46:39,129 Epoch 22: total training loss 2144.62\n", "2019-11-27 11:46:39,129 EPOCH 23\n", "2019-11-27 11:47:01,922 Epoch 23 Step: 29100 Batch Loss: 1.584973 Tokens per Sec: 7698, Lr: 0.000300\n", "2019-11-27 11:47:33,668 Epoch 23 Step: 29200 Batch Loss: 1.561042 Tokens per Sec: 7582, Lr: 0.000300\n", "2019-11-27 11:48:05,500 Epoch 23 Step: 29300 Batch Loss: 1.721926 Tokens per Sec: 7586, Lr: 0.000300\n", "2019-11-27 11:48:37,012 Epoch 23 Step: 29400 Batch Loss: 1.338379 Tokens per Sec: 7598, Lr: 0.000300\n", "2019-11-27 11:49:08,811 Epoch 23 Step: 29500 Batch Loss: 1.459262 Tokens per Sec: 7554, Lr: 0.000300\n", "2019-11-27 11:49:40,333 Epoch 23 Step: 29600 Batch Loss: 1.578105 Tokens per Sec: 7518, Lr: 0.000300\n", "2019-11-27 11:50:11,432 Epoch 23 Step: 29700 Batch Loss: 1.382825 Tokens per Sec: 7462, Lr: 0.000300\n", "2019-11-27 11:50:43,344 Epoch 23 Step: 29800 Batch Loss: 1.740782 Tokens per Sec: 7614, Lr: 0.000300\n", "2019-11-27 11:51:15,087 Epoch 23 Step: 29900 Batch Loss: 1.778173 Tokens per Sec: 7652, Lr: 0.000300\n", "2019-11-27 11:51:46,566 Epoch 23 Step: 30000 Batch Loss: 1.312358 Tokens per Sec: 7565, Lr: 0.000300\n", "2019-11-27 11:53:28,741 Hooray! New best validation result [ppl]!\n", "2019-11-27 11:53:28,741 Saving new checkpoint.\n", "2019-11-27 11:53:29,042 Example #0\n", "2019-11-27 11:53:29,042 \tSource: The services are free in South Africa\n", "2019-11-27 11:53:29,043 \tReference: Mo Aforikaborwa , ditirelo di neelwa kwa ntle ga go duelelwa .\n", "2019-11-27 11:53:29,043 \tHypothesis: Ditirelo di gololesegile mo Aforikaborwa\n", "2019-11-27 11:53:29,043 Example #1\n", "2019-11-27 11:53:29,043 \tSource: ( 7 ) A person or community dispossessed of property after 19 June 1913 as a result of past racially discriminatory laws or practices is entitled , to the extent provided by an Act of Parliament , either to restitution of that property or to equitable redress .\n", "2019-11-27 11:53:29,043 \tReference: ( 7 ) Motho kgotsa setlhopha sa magae se se amogilweng dithoto morago ga la 19 Seetebosigo 1913 ka ntlha ya melao ya kgale ya tlhotlhololo kgotsa ditlwaelo , se tshwanelwa , go ya ka thulaganyetso ya Molao wa Palamente go busediwa dithoto tseo , kgotsa tshiamisetso e e e kalekanang .\n", "2019-11-27 11:53:29,043 \tHypothesis: ( 7 ) Motho kgotsa baagi ba ba sa kgethololeng thoto morago ga 19 Seetebosigo 1913 ka ntlha ya melao ya bosenyi e e fetileng kgotsa mekgwa e e sa siamang , go ya ka Molao wa Palamente , e ka nna ya siamisa go dira gore dithoto kgotsa go siame kgotsa go fitlhelela phitlhelelo e e lekanang .\n", "2019-11-27 11:53:29,044 Example #2\n", "2019-11-27 11:53:29,044 \tSource: ruling\n", "2019-11-27 11:53:29,044 \tReference: tshwetsotaolo\n", "2019-11-27 11:53:29,044 \tHypothesis: go goga\n", "2019-11-27 11:53:29,044 Example #3\n", "2019-11-27 11:53:29,044 \tSource: Skills Development course .\n", "2019-11-27 11:53:29,045 \tReference: Khoso ya Tlhabololo ya Bokgoni .\n", "2019-11-27 11:53:29,045 \tHypothesis: Khoso ya Tlhabololo ya Bokgoni .\n", "2019-11-27 11:53:29,045 Validation result (greedy) at epoch 23, step 30000: bleu: 25.59, loss: 43571.1758, ppl: 4.7451, duration: 102.4780s\n", "2019-11-27 11:54:00,513 Epoch 23 Step: 30100 Batch Loss: 1.868418 Tokens per Sec: 7607, Lr: 0.000300\n", "2019-11-27 11:54:32,309 Epoch 23 Step: 30200 Batch Loss: 1.675581 Tokens per Sec: 7626, Lr: 0.000300\n", "2019-11-27 11:55:03,871 Epoch 23 Step: 30300 Batch Loss: 1.807264 Tokens per Sec: 7638, Lr: 0.000300\n", "2019-11-27 11:55:20,118 Epoch 23: total training loss 2124.69\n", "2019-11-27 11:55:20,118 EPOCH 24\n", "2019-11-27 11:55:36,014 Epoch 24 Step: 30400 Batch Loss: 1.799792 Tokens per Sec: 7724, Lr: 0.000300\n", "2019-11-27 11:56:07,703 Epoch 24 Step: 30500 Batch Loss: 1.544193 Tokens per Sec: 7619, Lr: 0.000300\n", "2019-11-27 11:56:39,489 Epoch 24 Step: 30600 Batch Loss: 1.230513 Tokens per Sec: 7478, Lr: 0.000300\n", "2019-11-27 11:57:11,177 Epoch 24 Step: 30700 Batch Loss: 1.415668 Tokens per Sec: 7676, Lr: 0.000300\n", "2019-11-27 11:57:42,709 Epoch 24 Step: 30800 Batch Loss: 1.667269 Tokens per Sec: 7612, Lr: 0.000300\n", "2019-11-27 11:58:14,537 Epoch 24 Step: 30900 Batch Loss: 1.554264 Tokens per Sec: 7636, Lr: 0.000300\n", "2019-11-27 11:58:46,013 Epoch 24 Step: 31000 Batch Loss: 1.783296 Tokens per Sec: 7469, Lr: 0.000300\n", "2019-11-27 11:59:17,688 Epoch 24 Step: 31100 Batch Loss: 1.328379 Tokens per Sec: 7681, Lr: 0.000300\n", "2019-11-27 11:59:49,750 Epoch 24 Step: 31200 Batch Loss: 1.797975 Tokens per Sec: 7478, Lr: 0.000300\n", "2019-11-27 12:00:21,346 Epoch 24 Step: 31300 Batch Loss: 1.702019 Tokens per Sec: 7526, Lr: 0.000300\n", "2019-11-27 12:00:53,471 Epoch 24 Step: 31400 Batch Loss: 1.355064 Tokens per Sec: 7652, Lr: 0.000300\n", "2019-11-27 12:01:24,840 Epoch 24 Step: 31500 Batch Loss: 1.402638 Tokens per Sec: 7536, Lr: 0.000300\n", "2019-11-27 12:01:56,395 Epoch 24 Step: 31600 Batch Loss: 1.613888 Tokens per Sec: 7699, Lr: 0.000300\n", "2019-11-27 12:02:18,242 Epoch 24: total training loss 2095.61\n", "2019-11-27 12:02:18,243 EPOCH 25\n", "2019-11-27 12:02:28,387 Epoch 25 Step: 31700 Batch Loss: 1.277563 Tokens per Sec: 7620, Lr: 0.000300\n", "2019-11-27 12:03:00,227 Epoch 25 Step: 31800 Batch Loss: 1.338673 Tokens per Sec: 7663, Lr: 0.000300\n", "2019-11-27 12:03:31,808 Epoch 25 Step: 31900 Batch Loss: 1.665285 Tokens per Sec: 7538, Lr: 0.000300\n", "2019-11-27 12:04:03,574 Epoch 25 Step: 32000 Batch Loss: 1.787052 Tokens per Sec: 7718, Lr: 0.000300\n", "2019-11-27 12:05:45,825 Hooray! New best validation result [ppl]!\n", "2019-11-27 12:05:45,825 Saving new checkpoint.\n", "2019-11-27 12:05:46,108 Example #0\n", "2019-11-27 12:05:46,108 \tSource: The services are free in South Africa\n", "2019-11-27 12:05:46,108 \tReference: Mo Aforikaborwa , ditirelo di neelwa kwa ntle ga go duelelwa .\n", "2019-11-27 12:05:46,109 \tHypothesis: Ditirelo di gololesegile mo Aforikaborwa\n", "2019-11-27 12:05:46,109 Example #1\n", "2019-11-27 12:05:46,109 \tSource: ( 7 ) A person or community dispossessed of property after 19 June 1913 as a result of past racially discriminatory laws or practices is entitled , to the extent provided by an Act of Parliament , either to restitution of that property or to equitable redress .\n", "2019-11-27 12:05:46,109 \tReference: ( 7 ) Motho kgotsa setlhopha sa magae se se amogilweng dithoto morago ga la 19 Seetebosigo 1913 ka ntlha ya melao ya kgale ya tlhotlhololo kgotsa ditlwaelo , se tshwanelwa , go ya ka thulaganyetso ya Molao wa Palamente go busediwa dithoto tseo , kgotsa tshiamisetso e e e kalekanang .\n", "2019-11-27 12:05:46,109 \tHypothesis: ( 7 ) Motho kgotsa batho ba ba sa amogelwang thoto morago ga 19 Seetebosigo 1913 ka ntlha ya melao ya maloba e e fetileng kgotsa mekgwatiriso e e tshwanetseng go dirwa , go ya ka Molao wa Palamente , e ka nna ya siamisa go sa lekalekane kgotsa go sa lekane le le tshwanetseng go siamisa kgatelelo .\n", "2019-11-27 12:05:46,109 Example #2\n", "2019-11-27 12:05:46,110 \tSource: ruling\n", "2019-11-27 12:05:46,110 \tReference: tshwetsotaolo\n", "2019-11-27 12:05:46,110 \tHypothesis: go ikgogela\n", "2019-11-27 12:05:46,110 Example #3\n", "2019-11-27 12:05:46,110 \tSource: Skills Development course .\n", "2019-11-27 12:05:46,110 \tReference: Khoso ya Tlhabololo ya Bokgoni .\n", "2019-11-27 12:05:46,110 \tHypothesis: Khoso ya Tlhabololo ya Bokgoni .\n", "2019-11-27 12:05:46,110 Validation result (greedy) at epoch 25, step 32000: bleu: 26.52, loss: 42917.8008, ppl: 4.6356, duration: 102.5359s\n", "2019-11-27 12:06:17,746 Epoch 25 Step: 32100 Batch Loss: 1.652790 Tokens per Sec: 7556, Lr: 0.000300\n", "2019-11-27 12:06:49,110 Epoch 25 Step: 32200 Batch Loss: 1.495177 Tokens per Sec: 7345, Lr: 0.000300\n", "2019-11-27 12:07:21,123 Epoch 25 Step: 32300 Batch Loss: 1.310986 Tokens per Sec: 7689, Lr: 0.000300\n", "2019-11-27 12:07:53,111 Epoch 25 Step: 32400 Batch Loss: 1.585905 Tokens per Sec: 7706, Lr: 0.000300\n", "2019-11-27 12:08:24,560 Epoch 25 Step: 32500 Batch Loss: 1.634917 Tokens per Sec: 7432, Lr: 0.000300\n", "2019-11-27 12:08:56,132 Epoch 25 Step: 32600 Batch Loss: 1.587194 Tokens per Sec: 7605, Lr: 0.000300\n", "2019-11-27 12:09:28,147 Epoch 25 Step: 32700 Batch Loss: 1.900753 Tokens per Sec: 7609, Lr: 0.000300\n", "2019-11-27 12:09:59,992 Epoch 25 Step: 32800 Batch Loss: 1.378002 Tokens per Sec: 7595, Lr: 0.000300\n", "2019-11-27 12:10:31,636 Epoch 25 Step: 32900 Batch Loss: 1.605950 Tokens per Sec: 7632, Lr: 0.000300\n", "2019-11-27 12:10:58,977 Epoch 25: total training loss 2071.84\n", "2019-11-27 12:10:58,977 EPOCH 26\n", "2019-11-27 12:11:03,457 Epoch 26 Step: 33000 Batch Loss: 1.521287 Tokens per Sec: 7001, Lr: 0.000300\n", "2019-11-27 12:11:35,214 Epoch 26 Step: 33100 Batch Loss: 1.378317 Tokens per Sec: 7588, Lr: 0.000300\n", "2019-11-27 12:12:06,667 Epoch 26 Step: 33200 Batch Loss: 1.492661 Tokens per Sec: 7453, Lr: 0.000300\n", "2019-11-27 12:12:38,671 Epoch 26 Step: 33300 Batch Loss: 1.685629 Tokens per Sec: 7506, Lr: 0.000300\n", "2019-11-27 12:13:10,751 Epoch 26 Step: 33400 Batch Loss: 1.531162 Tokens per Sec: 7636, Lr: 0.000300\n", "2019-11-27 12:13:42,638 Epoch 26 Step: 33500 Batch Loss: 1.266288 Tokens per Sec: 7500, Lr: 0.000300\n", "2019-11-27 12:14:14,630 Epoch 26 Step: 33600 Batch Loss: 1.782213 Tokens per Sec: 7670, Lr: 0.000300\n", "2019-11-27 12:14:46,123 Epoch 26 Step: 33700 Batch Loss: 1.320277 Tokens per Sec: 7390, Lr: 0.000300\n", "2019-11-27 12:15:17,949 Epoch 26 Step: 33800 Batch Loss: 1.437633 Tokens per Sec: 7575, Lr: 0.000300\n", "2019-11-27 12:15:50,008 Epoch 26 Step: 33900 Batch Loss: 1.471593 Tokens per Sec: 7643, Lr: 0.000300\n", "2019-11-27 12:16:21,881 Epoch 26 Step: 34000 Batch Loss: 1.258225 Tokens per Sec: 7587, Lr: 0.000300\n", "2019-11-27 12:18:05,100 Hooray! New best validation result [ppl]!\n", "2019-11-27 12:18:05,100 Saving new checkpoint.\n", "2019-11-27 12:18:05,415 Example #0\n", "2019-11-27 12:18:05,415 \tSource: The services are free in South Africa\n", "2019-11-27 12:18:05,416 \tReference: Mo Aforikaborwa , ditirelo di neelwa kwa ntle ga go duelelwa .\n", "2019-11-27 12:18:05,416 \tHypothesis: Ditirelo di gololesegile mo Aforikaborwa\n", "2019-11-27 12:18:05,416 Example #1\n", "2019-11-27 12:18:05,416 \tSource: ( 7 ) A person or community dispossessed of property after 19 June 1913 as a result of past racially discriminatory laws or practices is entitled , to the extent provided by an Act of Parliament , either to restitution of that property or to equitable redress .\n", "2019-11-27 12:18:05,416 \tReference: ( 7 ) Motho kgotsa setlhopha sa magae se se amogilweng dithoto morago ga la 19 Seetebosigo 1913 ka ntlha ya melao ya kgale ya tlhotlhololo kgotsa ditlwaelo , se tshwanelwa , go ya ka thulaganyetso ya Molao wa Palamente go busediwa dithoto tseo , kgotsa tshiamisetso e e e kalekanang .\n", "2019-11-27 12:18:05,416 \tHypothesis: ( 7 ) Motho kgotsa baagi ba ba nang le dithoto morago ga 19 Seetebosigo 1913 ka ntlha ya melao ya kgethololo e e fetileng kgotsa mekgwatiriso e e sa siamang , go ya ka Molao o o neilweng ke Palamente , e ka nna go dira gore go nne le go siame ga go lekana kgotsa go sa lekalekane .\n", "2019-11-27 12:18:05,417 Example #2\n", "2019-11-27 12:18:05,417 \tSource: ruling\n", "2019-11-27 12:18:05,417 \tReference: tshwetsotaolo\n", "2019-11-27 12:18:05,417 \tHypothesis: phuthelo\n", "2019-11-27 12:18:05,417 Example #3\n", "2019-11-27 12:18:05,417 \tSource: Skills Development course .\n", "2019-11-27 12:18:05,418 \tReference: Khoso ya Tlhabololo ya Bokgoni .\n", "2019-11-27 12:18:05,418 \tHypothesis: Khoso ya Tlhabololo ya Bokgoni .\n", "2019-11-27 12:18:05,418 Validation result (greedy) at epoch 26, step 34000: bleu: 26.15, loss: 42369.9492, ppl: 4.5457, duration: 103.5360s\n", "2019-11-27 12:18:37,081 Epoch 26 Step: 34100 Batch Loss: 1.759353 Tokens per Sec: 7566, Lr: 0.000300\n", "2019-11-27 12:19:08,842 Epoch 26 Step: 34200 Batch Loss: 1.735673 Tokens per Sec: 7585, Lr: 0.000300\n", "2019-11-27 12:19:40,201 Epoch 26 Step: 34300 Batch Loss: 1.272740 Tokens per Sec: 7496, Lr: 0.000300\n", "2019-11-27 12:19:43,594 Epoch 26: total training loss 2064.50\n", "2019-11-27 12:19:43,595 EPOCH 27\n", "2019-11-27 12:20:11,981 Epoch 27 Step: 34400 Batch Loss: 1.632121 Tokens per Sec: 7558, Lr: 0.000300\n", "2019-11-27 12:20:43,551 Epoch 27 Step: 34500 Batch Loss: 1.428843 Tokens per Sec: 7505, Lr: 0.000300\n", "2019-11-27 12:21:15,282 Epoch 27 Step: 34600 Batch Loss: 1.705959 Tokens per Sec: 7623, Lr: 0.000300\n", "2019-11-27 12:21:47,350 Epoch 27 Step: 34700 Batch Loss: 1.647930 Tokens per Sec: 7528, Lr: 0.000300\n", "2019-11-27 12:22:19,262 Epoch 27 Step: 34800 Batch Loss: 1.355222 Tokens per Sec: 7491, Lr: 0.000300\n", "2019-11-27 12:22:51,335 Epoch 27 Step: 34900 Batch Loss: 1.668518 Tokens per Sec: 7575, Lr: 0.000300\n", "2019-11-27 12:23:23,330 Epoch 27 Step: 35000 Batch Loss: 1.722842 Tokens per Sec: 7549, Lr: 0.000300\n", "2019-11-27 12:23:55,330 Epoch 27 Step: 35100 Batch Loss: 1.286787 Tokens per Sec: 7581, Lr: 0.000300\n", "2019-11-27 12:24:27,378 Epoch 27 Step: 35200 Batch Loss: 1.670881 Tokens per Sec: 7537, Lr: 0.000300\n", "2019-11-27 12:24:59,253 Epoch 27 Step: 35300 Batch Loss: 1.707505 Tokens per Sec: 7560, Lr: 0.000300\n", "2019-11-27 12:25:31,524 Epoch 27 Step: 35400 Batch Loss: 1.262991 Tokens per Sec: 7548, Lr: 0.000300\n", "2019-11-27 12:26:03,544 Epoch 27 Step: 35500 Batch Loss: 1.419637 Tokens per Sec: 7512, Lr: 0.000300\n", "2019-11-27 12:26:35,502 Epoch 27 Step: 35600 Batch Loss: 1.597602 Tokens per Sec: 7468, Lr: 0.000300\n", "2019-11-27 12:26:44,664 Epoch 27: total training loss 2029.82\n", "2019-11-27 12:26:44,664 EPOCH 28\n", "2019-11-27 12:27:07,895 Epoch 28 Step: 35700 Batch Loss: 1.758926 Tokens per Sec: 7474, Lr: 0.000300\n", "2019-11-27 12:27:40,010 Epoch 28 Step: 35800 Batch Loss: 1.574666 Tokens per Sec: 7464, Lr: 0.000300\n", "2019-11-27 12:28:11,876 Epoch 28 Step: 35900 Batch Loss: 1.394227 Tokens per Sec: 7441, Lr: 0.000300\n", "2019-11-27 12:28:43,905 Epoch 28 Step: 36000 Batch Loss: 1.354145 Tokens per Sec: 7448, Lr: 0.000300\n", "2019-11-27 12:30:28,038 Hooray! New best validation result [ppl]!\n", "2019-11-27 12:30:28,039 Saving new checkpoint.\n", "2019-11-27 12:30:28,374 Example #0\n", "2019-11-27 12:30:28,374 \tSource: The services are free in South Africa\n", "2019-11-27 12:30:28,375 \tReference: Mo Aforikaborwa , ditirelo di neelwa kwa ntle ga go duelelwa .\n", "2019-11-27 12:30:28,375 \tHypothesis: Ditirelo di gololesegile mo Aforikaborwa\n", "2019-11-27 12:30:28,375 Example #1\n", "2019-11-27 12:30:28,375 \tSource: ( 7 ) A person or community dispossessed of property after 19 June 1913 as a result of past racially discriminatory laws or practices is entitled , to the extent provided by an Act of Parliament , either to restitution of that property or to equitable redress .\n", "2019-11-27 12:30:28,375 \tReference: ( 7 ) Motho kgotsa setlhopha sa magae se se amogilweng dithoto morago ga la 19 Seetebosigo 1913 ka ntlha ya melao ya kgale ya tlhotlhololo kgotsa ditlwaelo , se tshwanelwa , go ya ka thulaganyetso ya Molao wa Palamente go busediwa dithoto tseo , kgotsa tshiamisetso e e e kalekanang .\n", "2019-11-27 12:30:28,375 \tHypothesis: ( 7 ) Motho kgotsa baagi ba ba nang le dithoto morago ga 19 Seetebosigo 1913 ka ntlha ya melao e e sa kgetholololeng kgotsa mekgwa e e tlhomilweng , go ya ka Molao wa Palamente , e ka nna ya siamisa go sa lekalekane kgotsa go sa lekalekane ga go sa lekalekane ga yona .\n", "2019-11-27 12:30:28,375 Example #2\n", "2019-11-27 12:30:28,375 \tSource: ruling\n", "2019-11-27 12:30:28,375 \tReference: tshwetsotaolo\n", "2019-11-27 12:30:28,375 \tHypothesis: go ikhutsa\n", "2019-11-27 12:30:28,375 Example #3\n", "2019-11-27 12:30:28,376 \tSource: Skills Development course .\n", "2019-11-27 12:30:28,376 \tReference: Khoso ya Tlhabololo ya Bokgoni .\n", "2019-11-27 12:30:28,376 \tHypothesis: Khoso ya Tlhabololo ya Bokgoni .\n", "2019-11-27 12:30:28,376 Validation result (greedy) at epoch 28, step 36000: bleu: 26.99, loss: 42141.7344, ppl: 4.5088, duration: 104.4706s\n", "2019-11-27 12:31:00,075 Epoch 28 Step: 36100 Batch Loss: 1.470447 Tokens per Sec: 7541, Lr: 0.000300\n", "2019-11-27 12:31:31,831 Epoch 28 Step: 36200 Batch Loss: 1.574187 Tokens per Sec: 7565, Lr: 0.000300\n", "2019-11-27 12:32:03,452 Epoch 28 Step: 36300 Batch Loss: 1.467552 Tokens per Sec: 7490, Lr: 0.000300\n", "2019-11-27 12:32:35,359 Epoch 28 Step: 36400 Batch Loss: 1.735628 Tokens per Sec: 7638, Lr: 0.000300\n", "2019-11-27 12:33:07,215 Epoch 28 Step: 36500 Batch Loss: 1.246443 Tokens per Sec: 7596, Lr: 0.000300\n", "2019-11-27 12:33:39,227 Epoch 28 Step: 36600 Batch Loss: 1.385774 Tokens per Sec: 7599, Lr: 0.000300\n", "2019-11-27 12:34:10,906 Epoch 28 Step: 36700 Batch Loss: 1.409538 Tokens per Sec: 7489, Lr: 0.000300\n", "2019-11-27 12:34:42,422 Epoch 28 Step: 36800 Batch Loss: 1.720665 Tokens per Sec: 7561, Lr: 0.000300\n", "2019-11-27 12:35:14,127 Epoch 28 Step: 36900 Batch Loss: 1.890163 Tokens per Sec: 7580, Lr: 0.000300\n", "2019-11-27 12:35:30,490 Epoch 28: total training loss 2022.90\n", "2019-11-27 12:35:30,490 EPOCH 29\n", "2019-11-27 12:35:46,313 Epoch 29 Step: 37000 Batch Loss: 1.168066 Tokens per Sec: 7495, Lr: 0.000300\n", "2019-11-27 12:36:18,168 Epoch 29 Step: 37100 Batch Loss: 1.313823 Tokens per Sec: 7495, Lr: 0.000300\n", "2019-11-27 12:36:49,516 Epoch 29 Step: 37200 Batch Loss: 1.680288 Tokens per Sec: 7380, Lr: 0.000300\n", "2019-11-27 12:37:21,294 Epoch 29 Step: 37300 Batch Loss: 1.728303 Tokens per Sec: 7508, Lr: 0.000300\n", "2019-11-27 12:37:53,481 Epoch 29 Step: 37400 Batch Loss: 1.600967 Tokens per Sec: 7551, Lr: 0.000300\n", "2019-11-27 12:38:25,580 Epoch 29 Step: 37500 Batch Loss: 1.323310 Tokens per Sec: 7448, Lr: 0.000300\n", "2019-11-27 12:38:57,813 Epoch 29 Step: 37600 Batch Loss: 1.611347 Tokens per Sec: 7568, Lr: 0.000300\n", "2019-11-27 12:39:29,724 Epoch 29 Step: 37700 Batch Loss: 1.282291 Tokens per Sec: 7389, Lr: 0.000300\n", "2019-11-27 12:40:01,710 Epoch 29 Step: 37800 Batch Loss: 1.824787 Tokens per Sec: 7457, Lr: 0.000300\n", "2019-11-27 12:40:33,896 Epoch 29 Step: 37900 Batch Loss: 1.516114 Tokens per Sec: 7389, Lr: 0.000300\n", "2019-11-27 12:41:06,362 Epoch 29 Step: 38000 Batch Loss: 1.746776 Tokens per Sec: 7562, Lr: 0.000300\n", "2019-11-27 12:42:50,489 Hooray! New best validation result [ppl]!\n", "2019-11-27 12:42:50,489 Saving new checkpoint.\n", "2019-11-27 12:42:50,807 Example #0\n", "2019-11-27 12:42:50,808 \tSource: The services are free in South Africa\n", "2019-11-27 12:42:50,808 \tReference: Mo Aforikaborwa , ditirelo di neelwa kwa ntle ga go duelelwa .\n", "2019-11-27 12:42:50,808 \tHypothesis: Ditirelo di gololesegile mo Aforikaborwa\n", "2019-11-27 12:42:50,808 Example #1\n", "2019-11-27 12:42:50,808 \tSource: ( 7 ) A person or community dispossessed of property after 19 June 1913 as a result of past racially discriminatory laws or practices is entitled , to the extent provided by an Act of Parliament , either to restitution of that property or to equitable redress .\n", "2019-11-27 12:42:50,808 \tReference: ( 7 ) Motho kgotsa setlhopha sa magae se se amogilweng dithoto morago ga la 19 Seetebosigo 1913 ka ntlha ya melao ya kgale ya tlhotlhololo kgotsa ditlwaelo , se tshwanelwa , go ya ka thulaganyetso ya Molao wa Palamente go busediwa dithoto tseo , kgotsa tshiamisetso e e e kalekanang .\n", "2019-11-27 12:42:50,809 \tHypothesis: ( 7 ) Motho kgotsa setšhaba se se sa amogelesegeng morago ga 19 Seetebosigo 1913 ka ntlha ya melao e e sa kgethololeng kgotsa mekgwatiriso e e tlhomamisiwang , go ya ka Molao wa Palamente , e ka nna go dira gore go nne le go sa lekalekane kgotsa go sa lekalekane ga go sa lekalekane .\n", "2019-11-27 12:42:50,809 Example #2\n", "2019-11-27 12:42:50,809 \tSource: ruling\n", "2019-11-27 12:42:50,809 \tReference: tshwetsotaolo\n", "2019-11-27 12:42:50,809 \tHypothesis: go ikgogela\n", "2019-11-27 12:42:50,809 Example #3\n", "2019-11-27 12:42:50,810 \tSource: Skills Development course .\n", "2019-11-27 12:42:50,810 \tReference: Khoso ya Tlhabololo ya Bokgoni .\n", "2019-11-27 12:42:50,810 \tHypothesis: Khoso ya Tlhabololo ya Bokgoni .\n", "2019-11-27 12:42:50,810 Validation result (greedy) at epoch 29, step 38000: bleu: 26.72, loss: 41620.4844, ppl: 4.4256, duration: 104.4478s\n", "2019-11-27 12:43:22,967 Epoch 29 Step: 38100 Batch Loss: 1.289660 Tokens per Sec: 7420, Lr: 0.000300\n", "2019-11-27 12:43:55,374 Epoch 29 Step: 38200 Batch Loss: 1.657576 Tokens per Sec: 7473, Lr: 0.000300\n", "2019-11-27 12:44:19,628 Epoch 29: total training loss 2006.31\n", "2019-11-27 12:44:19,629 EPOCH 30\n", "2019-11-27 12:44:28,060 Epoch 30 Step: 38300 Batch Loss: 1.599230 Tokens per Sec: 7578, Lr: 0.000300\n", "2019-11-27 12:45:00,169 Epoch 30 Step: 38400 Batch Loss: 1.362006 Tokens per Sec: 7422, Lr: 0.000300\n", "2019-11-27 12:45:32,362 Epoch 30 Step: 38500 Batch Loss: 1.626501 Tokens per Sec: 7474, Lr: 0.000300\n", "2019-11-27 12:46:04,713 Epoch 30 Step: 38600 Batch Loss: 1.627502 Tokens per Sec: 7364, Lr: 0.000300\n", "2019-11-27 12:46:36,027 Epoch 30 Step: 38700 Batch Loss: 1.553267 Tokens per Sec: 7464, Lr: 0.000300\n", "2019-11-27 12:47:08,327 Epoch 30 Step: 38800 Batch Loss: 1.392726 Tokens per Sec: 7588, Lr: 0.000300\n", "2019-11-27 12:47:40,512 Epoch 30 Step: 38900 Batch Loss: 1.608633 Tokens per Sec: 7736, Lr: 0.000300\n", "2019-11-27 12:48:12,122 Epoch 30 Step: 39000 Batch Loss: 1.609908 Tokens per Sec: 7608, Lr: 0.000300\n", "2019-11-27 12:48:43,781 Epoch 30 Step: 39100 Batch Loss: 1.590148 Tokens per Sec: 7634, Lr: 0.000300\n", "2019-11-27 12:49:15,094 Epoch 30 Step: 39200 Batch Loss: 1.582209 Tokens per Sec: 7435, Lr: 0.000300\n", "2019-11-27 12:49:46,690 Epoch 30 Step: 39300 Batch Loss: 1.705152 Tokens per Sec: 7603, Lr: 0.000300\n", "2019-11-27 12:50:18,374 Epoch 30 Step: 39400 Batch Loss: 1.571397 Tokens per Sec: 7634, Lr: 0.000300\n", "2019-11-27 12:50:50,276 Epoch 30 Step: 39500 Batch Loss: 1.517253 Tokens per Sec: 7640, Lr: 0.000300\n", "2019-11-27 12:51:20,339 Epoch 30: total training loss 1983.55\n", "2019-11-27 12:51:20,340 EPOCH 31\n", "2019-11-27 12:51:22,170 Epoch 31 Step: 39600 Batch Loss: 1.589104 Tokens per Sec: 6879, Lr: 0.000300\n", "2019-11-27 12:51:54,184 Epoch 31 Step: 39700 Batch Loss: 1.513076 Tokens per Sec: 7517, Lr: 0.000300\n", "2019-11-27 12:52:26,423 Epoch 31 Step: 39800 Batch Loss: 1.305691 Tokens per Sec: 7632, Lr: 0.000300\n", "2019-11-27 12:52:58,154 Epoch 31 Step: 39900 Batch Loss: 1.482919 Tokens per Sec: 7517, Lr: 0.000300\n", "2019-11-27 12:53:29,839 Epoch 31 Step: 40000 Batch Loss: 1.455574 Tokens per Sec: 7570, Lr: 0.000300\n", "2019-11-27 12:55:13,070 Hooray! New best validation result [ppl]!\n", "2019-11-27 12:55:13,070 Saving new checkpoint.\n", "2019-11-27 12:55:13,378 Example #0\n", "2019-11-27 12:55:13,378 \tSource: The services are free in South Africa\n", "2019-11-27 12:55:13,378 \tReference: Mo Aforikaborwa , ditirelo di neelwa kwa ntle ga go duelelwa .\n", "2019-11-27 12:55:13,378 \tHypothesis: Ditirelo di gololesegile mo Aforikaborwa\n", "2019-11-27 12:55:13,378 Example #1\n", "2019-11-27 12:55:13,379 \tSource: ( 7 ) A person or community dispossessed of property after 19 June 1913 as a result of past racially discriminatory laws or practices is entitled , to the extent provided by an Act of Parliament , either to restitution of that property or to equitable redress .\n", "2019-11-27 12:55:13,379 \tReference: ( 7 ) Motho kgotsa setlhopha sa magae se se amogilweng dithoto morago ga la 19 Seetebosigo 1913 ka ntlha ya melao ya kgale ya tlhotlhololo kgotsa ditlwaelo , se tshwanelwa , go ya ka thulaganyetso ya Molao wa Palamente go busediwa dithoto tseo , kgotsa tshiamisetso e e e kalekanang .\n", "2019-11-27 12:55:13,379 \tHypothesis: ( 7 ) Motho kgotsa baagi ba ba ganetsanang ka thoto morago ga 19 Seetebosigo 1913 ka ntlha ya melao ya kgethololo kgotsa mekgwa e e sa siamang , go ya ka Molao wa Palamente , e ka nna ya itlhokomolosa kgotsa ya lekanya le le lekanang le le le sa siamang .\n", "2019-11-27 12:55:13,379 Example #2\n", "2019-11-27 12:55:13,379 \tSource: ruling\n", "2019-11-27 12:55:13,379 \tReference: tshwetsotaolo\n", "2019-11-27 12:55:13,379 \tHypothesis: go ntsha\n", "2019-11-27 12:55:13,379 Example #3\n", "2019-11-27 12:55:13,379 \tSource: Skills Development course .\n", "2019-11-27 12:55:13,379 \tReference: Khoso ya Tlhabololo ya Bokgoni .\n", "2019-11-27 12:55:13,379 \tHypothesis: Khoso ya Tlhabololo ya Bokgoni .\n", "2019-11-27 12:55:13,380 Validation result (greedy) at epoch 31, step 40000: bleu: 27.23, loss: 41505.0781, ppl: 4.4074, duration: 103.5398s\n", "2019-11-27 12:55:44,909 Epoch 31 Step: 40100 Batch Loss: 1.502476 Tokens per Sec: 7519, Lr: 0.000300\n", "2019-11-27 12:56:16,571 Epoch 31 Step: 40200 Batch Loss: 1.284223 Tokens per Sec: 7510, Lr: 0.000300\n", "2019-11-27 12:56:48,371 Epoch 31 Step: 40300 Batch Loss: 1.504844 Tokens per Sec: 7605, Lr: 0.000300\n", "2019-11-27 12:57:20,082 Epoch 31 Step: 40400 Batch Loss: 1.480071 Tokens per Sec: 7577, Lr: 0.000300\n", "2019-11-27 12:57:51,816 Epoch 31 Step: 40500 Batch Loss: 1.700066 Tokens per Sec: 7617, Lr: 0.000300\n", "2019-11-27 12:58:23,812 Epoch 31 Step: 40600 Batch Loss: 1.519060 Tokens per Sec: 7596, Lr: 0.000300\n", "2019-11-27 12:58:55,605 Epoch 31 Step: 40700 Batch Loss: 1.537472 Tokens per Sec: 7445, Lr: 0.000300\n", "2019-11-27 12:59:27,624 Epoch 31 Step: 40800 Batch Loss: 1.669729 Tokens per Sec: 7597, Lr: 0.000300\n", "2019-11-27 12:59:59,444 Epoch 31 Step: 40900 Batch Loss: 1.701680 Tokens per Sec: 7501, Lr: 0.000300\n", "2019-11-27 13:00:04,756 Epoch 31: total training loss 1969.11\n", "2019-11-27 13:00:04,756 EPOCH 32\n", "2019-11-27 13:00:31,394 Epoch 32 Step: 41000 Batch Loss: 1.560519 Tokens per Sec: 7547, Lr: 0.000300\n", "2019-11-27 13:01:03,106 Epoch 32 Step: 41100 Batch Loss: 1.257946 Tokens per Sec: 7522, Lr: 0.000300\n", "2019-11-27 13:01:34,963 Epoch 32 Step: 41200 Batch Loss: 1.542236 Tokens per Sec: 7611, Lr: 0.000300\n", "2019-11-27 13:02:06,879 Epoch 32 Step: 41300 Batch Loss: 1.264607 Tokens per Sec: 7552, Lr: 0.000300\n", "2019-11-27 13:02:38,720 Epoch 32 Step: 41400 Batch Loss: 1.622833 Tokens per Sec: 7576, Lr: 0.000300\n", "2019-11-27 13:03:10,929 Epoch 32 Step: 41500 Batch Loss: 1.359242 Tokens per Sec: 7543, Lr: 0.000300\n", "2019-11-27 13:03:42,719 Epoch 32 Step: 41600 Batch Loss: 1.532966 Tokens per Sec: 7540, Lr: 0.000300\n", "2019-11-27 13:04:14,480 Epoch 32 Step: 41700 Batch Loss: 1.242920 Tokens per Sec: 7506, Lr: 0.000300\n", "2019-11-27 13:04:46,485 Epoch 32 Step: 41800 Batch Loss: 1.464010 Tokens per Sec: 7506, Lr: 0.000300\n", "2019-11-27 13:05:17,931 Epoch 32 Step: 41900 Batch Loss: 1.510335 Tokens per Sec: 7515, Lr: 0.000300\n", "2019-11-27 13:05:49,472 Epoch 32 Step: 42000 Batch Loss: 1.481223 Tokens per Sec: 7499, Lr: 0.000300\n", "2019-11-27 13:07:32,057 Hooray! New best validation result [ppl]!\n", "2019-11-27 13:07:32,057 Saving new checkpoint.\n", "2019-11-27 13:07:32,379 Example #0\n", "2019-11-27 13:07:32,379 \tSource: The services are free in South Africa\n", "2019-11-27 13:07:32,379 \tReference: Mo Aforikaborwa , ditirelo di neelwa kwa ntle ga go duelelwa .\n", "2019-11-27 13:07:32,379 \tHypothesis: Ditirelo di gololesegile mo Aforikaborwa\n", "2019-11-27 13:07:32,379 Example #1\n", "2019-11-27 13:07:32,380 \tSource: ( 7 ) A person or community dispossessed of property after 19 June 1913 as a result of past racially discriminatory laws or practices is entitled , to the extent provided by an Act of Parliament , either to restitution of that property or to equitable redress .\n", "2019-11-27 13:07:32,380 \tReference: ( 7 ) Motho kgotsa setlhopha sa magae se se amogilweng dithoto morago ga la 19 Seetebosigo 1913 ka ntlha ya melao ya kgale ya tlhotlhololo kgotsa ditlwaelo , se tshwanelwa , go ya ka thulaganyetso ya Molao wa Palamente go busediwa dithoto tseo , kgotsa tshiamisetso e e e kalekanang .\n", "2019-11-27 13:07:32,380 \tHypothesis: ( 7 ) Motho kgotsa baagi ba ba nang le thoto morago ga 19 Seetebosigo 1913 ka ntlha ya melao ya kgethololo ya bosenyi kgotsa mekgwa e e tshwanetseng , go ya ka Molao wa Palamente , e ka nna go dira gore go nne le go siamisa thoto eo kgotsa go siame .\n", "2019-11-27 13:07:32,380 Example #2\n", "2019-11-27 13:07:32,380 \tSource: ruling\n", "2019-11-27 13:07:32,380 \tReference: tshwetsotaolo\n", "2019-11-27 13:07:32,380 \tHypothesis: go ikgogela\n", "2019-11-27 13:07:32,380 Example #3\n", "2019-11-27 13:07:32,380 \tSource: Skills Development course .\n", "2019-11-27 13:07:32,380 \tReference: Khoso ya Tlhabololo ya Bokgoni .\n", "2019-11-27 13:07:32,380 \tHypothesis: Khoso ya Tlhabololo ya Bokgoni .\n", "2019-11-27 13:07:32,381 Validation result (greedy) at epoch 32, step 42000: bleu: 27.17, loss: 41047.1953, ppl: 4.3358, duration: 102.9081s\n", "2019-11-27 13:08:04,044 Epoch 32 Step: 42100 Batch Loss: 1.748304 Tokens per Sec: 7660, Lr: 0.000300\n", "2019-11-27 13:08:36,063 Epoch 32 Step: 42200 Batch Loss: 1.663833 Tokens per Sec: 7594, Lr: 0.000300\n", "2019-11-27 13:08:48,274 Epoch 32: total training loss 1956.85\n", "2019-11-27 13:08:48,275 EPOCH 33\n", "2019-11-27 13:09:07,951 Epoch 33 Step: 42300 Batch Loss: 1.396360 Tokens per Sec: 7512, Lr: 0.000300\n", "2019-11-27 13:09:39,912 Epoch 33 Step: 42400 Batch Loss: 1.630236 Tokens per Sec: 7646, Lr: 0.000300\n", "2019-11-27 13:10:11,426 Epoch 33 Step: 42500 Batch Loss: 1.234941 Tokens per Sec: 7553, Lr: 0.000300\n", "2019-11-27 13:10:43,117 Epoch 33 Step: 42600 Batch Loss: 1.544293 Tokens per Sec: 7577, Lr: 0.000300\n", "2019-11-27 13:11:15,002 Epoch 33 Step: 42700 Batch Loss: 1.741575 Tokens per Sec: 7607, Lr: 0.000300\n", "2019-11-27 13:11:46,919 Epoch 33 Step: 42800 Batch Loss: 1.294974 Tokens per Sec: 7531, Lr: 0.000300\n", "2019-11-27 13:12:18,400 Epoch 33 Step: 42900 Batch Loss: 1.491299 Tokens per Sec: 7459, Lr: 0.000300\n", "2019-11-27 13:12:50,262 Epoch 33 Step: 43000 Batch Loss: 1.336689 Tokens per Sec: 7511, Lr: 0.000300\n", "2019-11-27 13:13:21,833 Epoch 33 Step: 43100 Batch Loss: 1.645521 Tokens per Sec: 7645, Lr: 0.000300\n", "2019-11-27 13:13:53,483 Epoch 33 Step: 43200 Batch Loss: 1.510849 Tokens per Sec: 7623, Lr: 0.000300\n", "2019-11-27 13:14:25,411 Epoch 33 Step: 43300 Batch Loss: 1.223891 Tokens per Sec: 7590, Lr: 0.000300\n", "2019-11-27 13:14:56,810 Epoch 33 Step: 43400 Batch Loss: 1.400064 Tokens per Sec: 7455, Lr: 0.000300\n", "2019-11-27 13:15:28,485 Epoch 33 Step: 43500 Batch Loss: 1.482636 Tokens per Sec: 7553, Lr: 0.000300\n", "2019-11-27 13:15:48,508 Epoch 33: total training loss 1948.44\n", "2019-11-27 13:15:48,508 EPOCH 34\n", "2019-11-27 13:16:00,343 Epoch 34 Step: 43600 Batch Loss: 1.402205 Tokens per Sec: 7158, Lr: 0.000300\n", "2019-11-27 13:16:32,295 Epoch 34 Step: 43700 Batch Loss: 1.535597 Tokens per Sec: 7569, Lr: 0.000300\n", "2019-11-27 13:17:03,748 Epoch 34 Step: 43800 Batch Loss: 1.129035 Tokens per Sec: 7443, Lr: 0.000300\n", "2019-11-27 13:17:35,773 Epoch 34 Step: 43900 Batch Loss: 1.380437 Tokens per Sec: 7634, Lr: 0.000300\n", "2019-11-27 13:18:07,853 Epoch 34 Step: 44000 Batch Loss: 1.264002 Tokens per Sec: 7618, Lr: 0.000300\n", "2019-11-27 13:19:50,589 Hooray! New best validation result [ppl]!\n", "2019-11-27 13:19:50,589 Saving new checkpoint.\n", "2019-11-27 13:19:50,878 Example #0\n", "2019-11-27 13:19:50,879 \tSource: The services are free in South Africa\n", "2019-11-27 13:19:50,879 \tReference: Mo Aforikaborwa , ditirelo di neelwa kwa ntle ga go duelelwa .\n", "2019-11-27 13:19:50,879 \tHypothesis: Ditirelo di gololosegile mo Aforikaborwa\n", "2019-11-27 13:19:50,879 Example #1\n", "2019-11-27 13:19:50,880 \tSource: ( 7 ) A person or community dispossessed of property after 19 June 1913 as a result of past racially discriminatory laws or practices is entitled , to the extent provided by an Act of Parliament , either to restitution of that property or to equitable redress .\n", "2019-11-27 13:19:50,880 \tReference: ( 7 ) Motho kgotsa setlhopha sa magae se se amogilweng dithoto morago ga la 19 Seetebosigo 1913 ka ntlha ya melao ya kgale ya tlhotlhololo kgotsa ditlwaelo , se tshwanelwa , go ya ka thulaganyetso ya Molao wa Palamente go busediwa dithoto tseo , kgotsa tshiamisetso e e e kalekanang .\n", "2019-11-27 13:19:50,880 \tHypothesis: ( 7 ) Motho kgotsa baagi ba ba neng ba sa amogetswe thoto morago ga 19 Seetebosigo 1913 ka ntlha ya melao ya kgethololo kgotsa mekgwa e e sa siamang , e a tshwanela , go ya ka Molao wa Palamente , o ka nna wa dira gore go nne le tshwanelo ya go dira gore dithoto kgotsa go lekane le le lekanang le kgatelelo .\n", "2019-11-27 13:19:50,880 Example #2\n", "2019-11-27 13:19:50,880 \tSource: ruling\n", "2019-11-27 13:19:50,880 \tReference: tshwetsotaolo\n", "2019-11-27 13:19:50,880 \tHypothesis: go ikgogela\n", "2019-11-27 13:19:50,880 Example #3\n", "2019-11-27 13:19:50,881 \tSource: Skills Development course .\n", "2019-11-27 13:19:50,881 \tReference: Khoso ya Tlhabololo ya Bokgoni .\n", "2019-11-27 13:19:50,881 \tHypothesis: Khoso ya Tlhabololo ya Bokgoni .\n", "2019-11-27 13:19:50,881 Validation result (greedy) at epoch 34, step 44000: bleu: 27.85, loss: 40879.4180, ppl: 4.3099, duration: 103.0272s\n", "2019-11-27 13:20:22,915 Epoch 34 Step: 44100 Batch Loss: 1.471918 Tokens per Sec: 7595, Lr: 0.000300\n", "2019-11-27 13:20:54,722 Epoch 34 Step: 44200 Batch Loss: 1.491272 Tokens per Sec: 7532, Lr: 0.000300\n", "2019-11-27 13:21:26,385 Epoch 34 Step: 44300 Batch Loss: 1.485325 Tokens per Sec: 7574, Lr: 0.000300\n", "2019-11-27 13:21:58,392 Epoch 34 Step: 44400 Batch Loss: 1.400804 Tokens per Sec: 7677, Lr: 0.000300\n", "2019-11-27 13:22:30,240 Epoch 34 Step: 44500 Batch Loss: 1.215391 Tokens per Sec: 7621, Lr: 0.000300\n", "2019-11-27 13:23:01,986 Epoch 34 Step: 44600 Batch Loss: 1.430239 Tokens per Sec: 7638, Lr: 0.000300\n", "2019-11-27 13:23:33,905 Epoch 34 Step: 44700 Batch Loss: 1.713942 Tokens per Sec: 7564, Lr: 0.000300\n", "2019-11-27 13:24:05,869 Epoch 34 Step: 44800 Batch Loss: 1.502829 Tokens per Sec: 7572, Lr: 0.000300\n", "2019-11-27 13:24:31,280 Epoch 34: total training loss 1914.35\n", "2019-11-27 13:24:31,280 EPOCH 35\n", "2019-11-27 13:24:37,839 Epoch 35 Step: 44900 Batch Loss: 1.606096 Tokens per Sec: 7106, Lr: 0.000300\n", "2019-11-27 13:25:09,387 Epoch 35 Step: 45000 Batch Loss: 1.215390 Tokens per Sec: 7526, Lr: 0.000300\n", "2019-11-27 13:25:41,220 Epoch 35 Step: 45100 Batch Loss: 1.387948 Tokens per Sec: 7586, Lr: 0.000300\n", "2019-11-27 13:26:12,960 Epoch 35 Step: 45200 Batch Loss: 1.648783 Tokens per Sec: 7666, Lr: 0.000300\n", "2019-11-27 13:26:44,792 Epoch 35 Step: 45300 Batch Loss: 1.520435 Tokens per Sec: 7643, Lr: 0.000300\n", "2019-11-27 13:27:16,247 Epoch 35 Step: 45400 Batch Loss: 1.670077 Tokens per Sec: 7477, Lr: 0.000300\n", "2019-11-27 13:27:47,696 Epoch 35 Step: 45500 Batch Loss: 1.453155 Tokens per Sec: 7513, Lr: 0.000300\n", "2019-11-27 13:28:19,661 Epoch 35 Step: 45600 Batch Loss: 1.436879 Tokens per Sec: 7646, Lr: 0.000300\n", "2019-11-27 13:28:51,728 Epoch 35 Step: 45700 Batch Loss: 1.233272 Tokens per Sec: 7440, Lr: 0.000300\n", "2019-11-27 13:29:23,428 Epoch 35 Step: 45800 Batch Loss: 1.613018 Tokens per Sec: 7550, Lr: 0.000300\n", "2019-11-27 13:29:55,069 Epoch 35 Step: 45900 Batch Loss: 1.588095 Tokens per Sec: 7522, Lr: 0.000300\n", "2019-11-27 13:30:27,132 Epoch 35 Step: 46000 Batch Loss: 1.461054 Tokens per Sec: 7598, Lr: 0.000300\n", "2019-11-27 13:32:09,737 Hooray! New best validation result [ppl]!\n", "2019-11-27 13:32:09,737 Saving new checkpoint.\n", "2019-11-27 13:32:10,063 Example #0\n", "2019-11-27 13:32:10,063 \tSource: The services are free in South Africa\n", "2019-11-27 13:32:10,063 \tReference: Mo Aforikaborwa , ditirelo di neelwa kwa ntle ga go duelelwa .\n", "2019-11-27 13:32:10,063 \tHypothesis: Ditirelo di gololesegile mo Aforikaborwa\n", "2019-11-27 13:32:10,064 Example #1\n", "2019-11-27 13:32:10,064 \tSource: ( 7 ) A person or community dispossessed of property after 19 June 1913 as a result of past racially discriminatory laws or practices is entitled , to the extent provided by an Act of Parliament , either to restitution of that property or to equitable redress .\n", "2019-11-27 13:32:10,064 \tReference: ( 7 ) Motho kgotsa setlhopha sa magae se se amogilweng dithoto morago ga la 19 Seetebosigo 1913 ka ntlha ya melao ya kgale ya tlhotlhololo kgotsa ditlwaelo , se tshwanelwa , go ya ka thulaganyetso ya Molao wa Palamente go busediwa dithoto tseo , kgotsa tshiamisetso e e e kalekanang .\n", "2019-11-27 13:32:10,064 \tHypothesis: ( 7 ) Motho kgotsa baagi ba ba neng ba sa amogele thoto morago ga 19 Seetebosigo 1913 ka ntlha ya melao e e sa fediseng pelo kgotsa mekgwa e e nang le bofitlha , go ya ka Molao wa Palamente , e ka nna go tlosa thoto eo kgotsa go sa lekalekane .\n", "2019-11-27 13:32:10,064 Example #2\n", "2019-11-27 13:32:10,065 \tSource: ruling\n", "2019-11-27 13:32:10,065 \tReference: tshwetsotaolo\n", "2019-11-27 13:32:10,065 \tHypothesis: kgobalo\n", "2019-11-27 13:32:10,065 Example #3\n", "2019-11-27 13:32:10,065 \tSource: Skills Development course .\n", "2019-11-27 13:32:10,065 \tReference: Khoso ya Tlhabololo ya Bokgoni .\n", "2019-11-27 13:32:10,066 \tHypothesis: Khoso ya Tlhabololo ya Bokgoni .\n", "2019-11-27 13:32:10,066 Validation result (greedy) at epoch 35, step 46000: bleu: 27.94, loss: 40705.3516, ppl: 4.2832, duration: 102.9330s\n", "2019-11-27 13:32:41,956 Epoch 35 Step: 46100 Batch Loss: 1.317172 Tokens per Sec: 7561, Lr: 0.000300\n", "2019-11-27 13:33:13,774 Epoch 35 Step: 46200 Batch Loss: 1.563701 Tokens per Sec: 7503, Lr: 0.000300\n", "2019-11-27 13:33:14,781 Epoch 35: total training loss 1914.03\n", "2019-11-27 13:33:14,781 EPOCH 36\n", "2019-11-27 13:33:45,877 Epoch 36 Step: 46300 Batch Loss: 1.474903 Tokens per Sec: 7533, Lr: 0.000300\n", "2019-11-27 13:34:18,034 Epoch 36 Step: 46400 Batch Loss: 1.510665 Tokens per Sec: 7622, Lr: 0.000300\n", "2019-11-27 13:34:50,068 Epoch 36 Step: 46500 Batch Loss: 1.211242 Tokens per Sec: 7656, Lr: 0.000300\n", "2019-11-27 13:35:22,105 Epoch 36 Step: 46600 Batch Loss: 1.346674 Tokens per Sec: 7721, Lr: 0.000300\n", "2019-11-27 13:35:53,905 Epoch 36 Step: 46700 Batch Loss: 1.191994 Tokens per Sec: 7528, Lr: 0.000300\n", "2019-11-27 13:36:25,916 Epoch 36 Step: 46800 Batch Loss: 1.368068 Tokens per Sec: 7573, Lr: 0.000300\n", "2019-11-27 13:36:57,517 Epoch 36 Step: 46900 Batch Loss: 1.592258 Tokens per Sec: 7536, Lr: 0.000300\n", "2019-11-27 13:37:29,147 Epoch 36 Step: 47000 Batch Loss: 1.562731 Tokens per Sec: 7409, Lr: 0.000300\n", "2019-11-27 13:38:00,694 Epoch 36 Step: 47100 Batch Loss: 1.342320 Tokens per Sec: 7602, Lr: 0.000300\n", "2019-11-27 13:38:32,398 Epoch 36 Step: 47200 Batch Loss: 1.498322 Tokens per Sec: 7438, Lr: 0.000300\n", "2019-11-27 13:39:04,365 Epoch 36 Step: 47300 Batch Loss: 1.508350 Tokens per Sec: 7587, Lr: 0.000300\n", "2019-11-27 13:39:36,657 Epoch 36 Step: 47400 Batch Loss: 1.420274 Tokens per Sec: 7701, Lr: 0.000300\n", "2019-11-27 13:40:08,123 Epoch 36 Step: 47500 Batch Loss: 1.416232 Tokens per Sec: 7411, Lr: 0.000300\n", "2019-11-27 13:40:14,692 Epoch 36: total training loss 1894.89\n", "2019-11-27 13:40:14,693 EPOCH 37\n", "2019-11-27 13:40:40,077 Epoch 37 Step: 47600 Batch Loss: 1.591334 Tokens per Sec: 7451, Lr: 0.000300\n", "2019-11-27 13:41:12,204 Epoch 37 Step: 47700 Batch Loss: 1.182363 Tokens per Sec: 7692, Lr: 0.000300\n", "2019-11-27 13:41:43,825 Epoch 37 Step: 47800 Batch Loss: 1.332414 Tokens per Sec: 7616, Lr: 0.000300\n", "2019-11-27 13:42:15,846 Epoch 37 Step: 47900 Batch Loss: 1.557815 Tokens per Sec: 7606, Lr: 0.000300\n", "2019-11-27 13:42:47,840 Epoch 37 Step: 48000 Batch Loss: 1.459852 Tokens per Sec: 7548, Lr: 0.000300\n", "2019-11-27 13:44:30,454 Hooray! New best validation result [ppl]!\n", "2019-11-27 13:44:30,454 Saving new checkpoint.\n", "2019-11-27 13:44:30,773 Example #0\n", "2019-11-27 13:44:30,774 \tSource: The services are free in South Africa\n", "2019-11-27 13:44:30,774 \tReference: Mo Aforikaborwa , ditirelo di neelwa kwa ntle ga go duelelwa .\n", "2019-11-27 13:44:30,775 \tHypothesis: Ditirelo di gololesegile mo Aforikaborwa\n", "2019-11-27 13:44:30,775 Example #1\n", "2019-11-27 13:44:30,775 \tSource: ( 7 ) A person or community dispossessed of property after 19 June 1913 as a result of past racially discriminatory laws or practices is entitled , to the extent provided by an Act of Parliament , either to restitution of that property or to equitable redress .\n", "2019-11-27 13:44:30,775 \tReference: ( 7 ) Motho kgotsa setlhopha sa magae se se amogilweng dithoto morago ga la 19 Seetebosigo 1913 ka ntlha ya melao ya kgale ya tlhotlhololo kgotsa ditlwaelo , se tshwanelwa , go ya ka thulaganyetso ya Molao wa Palamente go busediwa dithoto tseo , kgotsa tshiamisetso e e e kalekanang .\n", "2019-11-27 13:44:30,775 \tHypothesis: ( 7 ) Motho kgotsa baagi ba ba nang le dithoto morago ga 19 Seetebosigo 1913 ka ntlha ya melao ya kgethololo kgotsa mekgwa e e sa siamang , go ya ka Molao wa Palamente , e ka nna go ikobela thoto eo kgotsa go tlhoka kgatelelo e e lekaneng .\n", "2019-11-27 13:44:30,775 Example #2\n", "2019-11-27 13:44:30,776 \tSource: ruling\n", "2019-11-27 13:44:30,776 \tReference: tshwetsotaolo\n", "2019-11-27 13:44:30,776 \tHypothesis: go ikhutsa\n", "2019-11-27 13:44:30,776 Example #3\n", "2019-11-27 13:44:30,776 \tSource: Skills Development course .\n", "2019-11-27 13:44:30,776 \tReference: Khoso ya Tlhabololo ya Bokgoni .\n", "2019-11-27 13:44:30,776 \tHypothesis: Khoso ya Tlhabololo ya Bokgoni .\n", "2019-11-27 13:44:30,777 Validation result (greedy) at epoch 37, step 48000: bleu: 27.78, loss: 40279.3750, ppl: 4.2185, duration: 102.9363s\n", "2019-11-27 13:45:02,476 Epoch 37 Step: 48100 Batch Loss: 1.532448 Tokens per Sec: 7582, Lr: 0.000300\n", "2019-11-27 13:45:34,528 Epoch 37 Step: 48200 Batch Loss: 1.499903 Tokens per Sec: 7497, Lr: 0.000300\n", "2019-11-27 13:46:06,465 Epoch 37 Step: 48300 Batch Loss: 1.453014 Tokens per Sec: 7532, Lr: 0.000300\n", "2019-11-27 13:46:38,151 Epoch 37 Step: 48400 Batch Loss: 1.277184 Tokens per Sec: 7458, Lr: 0.000300\n", "2019-11-27 13:47:09,948 Epoch 37 Step: 48500 Batch Loss: 1.479676 Tokens per Sec: 7457, Lr: 0.000300\n", "2019-11-27 13:47:42,004 Epoch 37 Step: 48600 Batch Loss: 1.644649 Tokens per Sec: 7593, Lr: 0.000300\n", "2019-11-27 13:48:13,657 Epoch 37 Step: 48700 Batch Loss: 1.477106 Tokens per Sec: 7542, Lr: 0.000300\n", "2019-11-27 13:48:45,607 Epoch 37 Step: 48800 Batch Loss: 1.428840 Tokens per Sec: 7599, Lr: 0.000300\n", "2019-11-27 13:48:57,972 Epoch 37: total training loss 1878.21\n", "2019-11-27 13:48:57,973 EPOCH 38\n", "2019-11-27 13:49:17,589 Epoch 38 Step: 48900 Batch Loss: 1.252699 Tokens per Sec: 7577, Lr: 0.000300\n", "2019-11-27 13:49:49,518 Epoch 38 Step: 49000 Batch Loss: 1.543510 Tokens per Sec: 7480, Lr: 0.000300\n", "2019-11-27 13:50:21,728 Epoch 38 Step: 49100 Batch Loss: 1.510459 Tokens per Sec: 7522, Lr: 0.000300\n", "2019-11-27 13:50:53,576 Epoch 38 Step: 49200 Batch Loss: 1.610879 Tokens per Sec: 7561, Lr: 0.000300\n", "2019-11-27 13:51:26,090 Epoch 38 Step: 49300 Batch Loss: 1.520711 Tokens per Sec: 7568, Lr: 0.000300\n", "2019-11-27 13:51:58,201 Epoch 38 Step: 49400 Batch Loss: 1.545545 Tokens per Sec: 7415, Lr: 0.000300\n", "2019-11-27 13:52:30,457 Epoch 38 Step: 49500 Batch Loss: 1.256113 Tokens per Sec: 7400, Lr: 0.000300\n", "2019-11-27 13:53:02,437 Epoch 38 Step: 49600 Batch Loss: 1.646463 Tokens per Sec: 7623, Lr: 0.000300\n", "2019-11-27 13:53:34,613 Epoch 38 Step: 49700 Batch Loss: 1.575490 Tokens per Sec: 7558, Lr: 0.000300\n", "2019-11-27 13:54:06,660 Epoch 38 Step: 49800 Batch Loss: 1.518013 Tokens per Sec: 7540, Lr: 0.000300\n", "2019-11-27 13:54:38,687 Epoch 38 Step: 49900 Batch Loss: 1.188792 Tokens per Sec: 7553, Lr: 0.000300\n", "2019-11-27 13:55:10,403 Epoch 38 Step: 50000 Batch Loss: 1.344512 Tokens per Sec: 7569, Lr: 0.000300\n", "2019-11-27 13:56:52,987 Hooray! New best validation result [ppl]!\n", "2019-11-27 13:56:52,987 Saving new checkpoint.\n", "2019-11-27 13:56:53,274 Example #0\n", "2019-11-27 13:56:53,274 \tSource: The services are free in South Africa\n", "2019-11-27 13:56:53,274 \tReference: Mo Aforikaborwa , ditirelo di neelwa kwa ntle ga go duelelwa .\n", "2019-11-27 13:56:53,275 \tHypothesis: Ditirelo di gololesegile mo Aforikaborwa\n", "2019-11-27 13:56:53,275 Example #1\n", "2019-11-27 13:56:53,275 \tSource: ( 7 ) A person or community dispossessed of property after 19 June 1913 as a result of past racially discriminatory laws or practices is entitled , to the extent provided by an Act of Parliament , either to restitution of that property or to equitable redress .\n", "2019-11-27 13:56:53,275 \tReference: ( 7 ) Motho kgotsa setlhopha sa magae se se amogilweng dithoto morago ga la 19 Seetebosigo 1913 ka ntlha ya melao ya kgale ya tlhotlhololo kgotsa ditlwaelo , se tshwanelwa , go ya ka thulaganyetso ya Molao wa Palamente go busediwa dithoto tseo , kgotsa tshiamisetso e e e kalekanang .\n", "2019-11-27 13:56:53,275 \tHypothesis: ( 7 ) Motho kgotsa baagi ba ba neng ba sa amogele thoto morago ga 19 Seetebosigo 1913 ka ntlha ya melao ya kgethololo e e fetileng kgotsa mekgwatiriso e tshwanetse go latelwa , go fitlha ka Molao wa Palamente , o ka nna wa ipoeletsa go tlosa dithoto tseo kgotsa go siamisa go se kgone go siamisa .\n", "2019-11-27 13:56:53,275 Example #2\n", "2019-11-27 13:56:53,276 \tSource: ruling\n", "2019-11-27 13:56:53,276 \tReference: tshwetsotaolo\n", "2019-11-27 13:56:53,276 \tHypothesis: tshutiso\n", "2019-11-27 13:56:53,276 Example #3\n", "2019-11-27 13:56:53,276 \tSource: Skills Development course .\n", "2019-11-27 13:56:53,277 \tReference: Khoso ya Tlhabololo ya Bokgoni .\n", "2019-11-27 13:56:53,277 \tHypothesis: Khoso ya Tlhabololo ya Bokgoni .\n", "2019-11-27 13:56:53,277 Validation result (greedy) at epoch 38, step 50000: bleu: 28.53, loss: 39917.3906, ppl: 4.1643, duration: 102.8731s\n", "2019-11-27 13:57:25,053 Epoch 38 Step: 50100 Batch Loss: 1.306150 Tokens per Sec: 7506, Lr: 0.000300\n", "2019-11-27 13:57:43,024 Epoch 38: total training loss 1870.86\n", "2019-11-27 13:57:43,024 EPOCH 39\n", "2019-11-27 13:57:56,491 Epoch 39 Step: 50200 Batch Loss: 1.433909 Tokens per Sec: 7344, Lr: 0.000300\n", "2019-11-27 13:58:28,426 Epoch 39 Step: 50300 Batch Loss: 1.220430 Tokens per Sec: 7513, Lr: 0.000300\n", "2019-11-27 13:59:00,098 Epoch 39 Step: 50400 Batch Loss: 1.406719 Tokens per Sec: 7456, Lr: 0.000300\n", "2019-11-27 13:59:32,045 Epoch 39 Step: 50500 Batch Loss: 1.456174 Tokens per Sec: 7622, Lr: 0.000300\n", "2019-11-27 14:00:03,609 Epoch 39 Step: 50600 Batch Loss: 1.315114 Tokens per Sec: 7646, Lr: 0.000300\n", "2019-11-27 14:00:35,493 Epoch 39 Step: 50700 Batch Loss: 1.319806 Tokens per Sec: 7573, Lr: 0.000300\n", "2019-11-27 14:01:06,837 Epoch 39 Step: 50800 Batch Loss: 1.559555 Tokens per Sec: 7477, Lr: 0.000300\n", "2019-11-27 14:01:38,992 Epoch 39 Step: 50900 Batch Loss: 1.534078 Tokens per Sec: 7540, Lr: 0.000300\n", "2019-11-27 14:02:10,776 Epoch 39 Step: 51000 Batch Loss: 1.467141 Tokens per Sec: 7481, Lr: 0.000300\n", "2019-11-27 14:02:42,506 Epoch 39 Step: 51100 Batch Loss: 1.304167 Tokens per Sec: 7452, Lr: 0.000300\n", "2019-11-27 14:03:14,421 Epoch 39 Step: 51200 Batch Loss: 1.567845 Tokens per Sec: 7642, Lr: 0.000300\n", "2019-11-27 14:03:46,405 Epoch 39 Step: 51300 Batch Loss: 1.505763 Tokens per Sec: 7637, Lr: 0.000300\n", "2019-11-27 14:04:18,626 Epoch 39 Step: 51400 Batch Loss: 1.442390 Tokens per Sec: 7617, Lr: 0.000300\n", "2019-11-27 14:04:43,544 Epoch 39: total training loss 1859.15\n", "2019-11-27 14:04:43,544 EPOCH 40\n", "2019-11-27 14:04:50,926 Epoch 40 Step: 51500 Batch Loss: 1.323330 Tokens per Sec: 7276, Lr: 0.000300\n", "2019-11-27 14:05:22,495 Epoch 40 Step: 51600 Batch Loss: 1.177066 Tokens per Sec: 7610, Lr: 0.000300\n", "2019-11-27 14:05:53,923 Epoch 40 Step: 51700 Batch Loss: 1.331736 Tokens per Sec: 7402, Lr: 0.000300\n", "2019-11-27 14:06:25,688 Epoch 40 Step: 51800 Batch Loss: 1.405469 Tokens per Sec: 7450, Lr: 0.000300\n", "2019-11-27 14:06:57,861 Epoch 40 Step: 51900 Batch Loss: 1.220772 Tokens per Sec: 7603, Lr: 0.000300\n", "2019-11-27 14:07:29,730 Epoch 40 Step: 52000 Batch Loss: 1.356715 Tokens per Sec: 7507, Lr: 0.000300\n", "2019-11-27 14:09:12,386 Hooray! New best validation result [ppl]!\n", "2019-11-27 14:09:12,387 Saving new checkpoint.\n", "2019-11-27 14:09:12,723 Example #0\n", "2019-11-27 14:09:12,724 \tSource: The services are free in South Africa\n", "2019-11-27 14:09:12,724 \tReference: Mo Aforikaborwa , ditirelo di neelwa kwa ntle ga go duelelwa .\n", "2019-11-27 14:09:12,724 \tHypothesis: Ditirelo di gololesegile mo Aforikaborwa\n", "2019-11-27 14:09:12,724 Example #1\n", "2019-11-27 14:09:12,725 \tSource: ( 7 ) A person or community dispossessed of property after 19 June 1913 as a result of past racially discriminatory laws or practices is entitled , to the extent provided by an Act of Parliament , either to restitution of that property or to equitable redress .\n", "2019-11-27 14:09:12,725 \tReference: ( 7 ) Motho kgotsa setlhopha sa magae se se amogilweng dithoto morago ga la 19 Seetebosigo 1913 ka ntlha ya melao ya kgale ya tlhotlhololo kgotsa ditlwaelo , se tshwanelwa , go ya ka thulaganyetso ya Molao wa Palamente go busediwa dithoto tseo , kgotsa tshiamisetso e e e kalekanang .\n", "2019-11-27 14:09:12,725 \tHypothesis: ( 7 ) Motho kgotsa baagi ba ba nang le dithoto morago ga 19 Seetebosigo 1913 ka ntlha ya melao ya kgethololo e e fetileng kgotsa mekgwa e e tshwanetseng , go ya ka Molao wa Palamente , e ka nna go tlosa maemo a go dira dithoto kgotsa go siamisa go siamisa maemo a a lekaneng .\n", "2019-11-27 14:09:12,725 Example #2\n", "2019-11-27 14:09:12,726 \tSource: ruling\n", "2019-11-27 14:09:12,726 \tReference: tshwetsotaolo\n", "2019-11-27 14:09:12,726 \tHypothesis: pusetso\n", "2019-11-27 14:09:12,726 Example #3\n", "2019-11-27 14:09:12,726 \tSource: Skills Development course .\n", "2019-11-27 14:09:12,726 \tReference: Khoso ya Tlhabololo ya Bokgoni .\n", "2019-11-27 14:09:12,727 \tHypothesis: Khoso ya Tlhabololo ya Bokgoni .\n", "2019-11-27 14:09:12,727 Validation result (greedy) at epoch 40, step 52000: bleu: 28.70, loss: 39685.9570, ppl: 4.1300, duration: 102.9963s\n", "2019-11-27 14:09:44,956 Epoch 40 Step: 52100 Batch Loss: 1.336505 Tokens per Sec: 7686, Lr: 0.000300\n", "2019-11-27 14:10:16,482 Epoch 40 Step: 52200 Batch Loss: 1.214933 Tokens per Sec: 7452, Lr: 0.000300\n", "2019-11-27 14:10:48,067 Epoch 40 Step: 52300 Batch Loss: 1.454934 Tokens per Sec: 7437, Lr: 0.000300\n", "2019-11-27 14:11:19,752 Epoch 40 Step: 52400 Batch Loss: 1.252046 Tokens per Sec: 7543, Lr: 0.000300\n", "2019-11-27 14:11:51,691 Epoch 40 Step: 52500 Batch Loss: 1.565211 Tokens per Sec: 7666, Lr: 0.000300\n", "2019-11-27 14:12:23,552 Epoch 40 Step: 52600 Batch Loss: 1.238301 Tokens per Sec: 7596, Lr: 0.000300\n", "2019-11-27 14:12:55,292 Epoch 40 Step: 52700 Batch Loss: 1.727185 Tokens per Sec: 7664, Lr: 0.000300\n", "2019-11-27 14:13:27,242 Epoch 40: total training loss 1850.72\n", "2019-11-27 14:13:27,243 EPOCH 41\n", "2019-11-27 14:13:27,763 Epoch 41 Step: 52800 Batch Loss: 1.492553 Tokens per Sec: 5856, Lr: 0.000300\n", "2019-11-27 14:13:59,629 Epoch 41 Step: 52900 Batch Loss: 1.433213 Tokens per Sec: 7487, Lr: 0.000300\n", "2019-11-27 14:14:31,455 Epoch 41 Step: 53000 Batch Loss: 1.173835 Tokens per Sec: 7591, Lr: 0.000300\n", "2019-11-27 14:15:03,650 Epoch 41 Step: 53100 Batch Loss: 1.501592 Tokens per Sec: 7739, Lr: 0.000300\n", "2019-11-27 14:15:35,359 Epoch 41 Step: 53200 Batch Loss: 1.411483 Tokens per Sec: 7540, Lr: 0.000300\n", "2019-11-27 14:16:06,938 Epoch 41 Step: 53300 Batch Loss: 1.261787 Tokens per Sec: 7602, Lr: 0.000300\n", "2019-11-27 14:16:39,321 Epoch 41 Step: 53400 Batch Loss: 1.386172 Tokens per Sec: 7711, Lr: 0.000300\n", "2019-11-27 14:17:11,307 Epoch 41 Step: 53500 Batch Loss: 1.516077 Tokens per Sec: 7580, Lr: 0.000300\n", "2019-11-27 14:17:42,784 Epoch 41 Step: 53600 Batch Loss: 1.425349 Tokens per Sec: 7482, Lr: 0.000300\n", "2019-11-27 14:18:14,506 Epoch 41 Step: 53700 Batch Loss: 1.195588 Tokens per Sec: 7431, Lr: 0.000300\n", "2019-11-27 14:18:46,435 Epoch 41 Step: 53800 Batch Loss: 1.348306 Tokens per Sec: 7684, Lr: 0.000300\n", "2019-11-27 14:19:18,501 Epoch 41 Step: 53900 Batch Loss: 1.602806 Tokens per Sec: 7561, Lr: 0.000300\n", "2019-11-27 14:19:50,073 Epoch 41 Step: 54000 Batch Loss: 1.110930 Tokens per Sec: 7564, Lr: 0.000300\n", "2019-11-27 14:21:32,867 Hooray! New best validation result [ppl]!\n", "2019-11-27 14:21:32,867 Saving new checkpoint.\n", "2019-11-27 14:21:33,183 Example #0\n", "2019-11-27 14:21:33,184 \tSource: The services are free in South Africa\n", "2019-11-27 14:21:33,184 \tReference: Mo Aforikaborwa , ditirelo di neelwa kwa ntle ga go duelelwa .\n", "2019-11-27 14:21:33,184 \tHypothesis: Ditirelo di gololosegile mo Aforikaborwa\n", "2019-11-27 14:21:33,184 Example #1\n", "2019-11-27 14:21:33,185 \tSource: ( 7 ) A person or community dispossessed of property after 19 June 1913 as a result of past racially discriminatory laws or practices is entitled , to the extent provided by an Act of Parliament , either to restitution of that property or to equitable redress .\n", "2019-11-27 14:21:33,185 \tReference: ( 7 ) Motho kgotsa setlhopha sa magae se se amogilweng dithoto morago ga la 19 Seetebosigo 1913 ka ntlha ya melao ya kgale ya tlhotlhololo kgotsa ditlwaelo , se tshwanelwa , go ya ka thulaganyetso ya Molao wa Palamente go busediwa dithoto tseo , kgotsa tshiamisetso e e e kalekanang .\n", "2019-11-27 14:21:33,185 \tHypothesis: ( 7 ) Motho kgotsa baagi ba ba nang le dithoto morago ga 19 Seetebosigo 1913 ka ntlha ya melao ya kgethololo e e fetileng kgotsa mekgwa ya go kgetholola , go ya ka Molao wa Palamente , e ka nna go tlosa dithoto kgotsa go siamisa go siamisa .\n", "2019-11-27 14:21:33,185 Example #2\n", "2019-11-27 14:21:33,185 \tSource: ruling\n", "2019-11-27 14:21:33,186 \tReference: tshwetsotaolo\n", "2019-11-27 14:21:33,186 \tHypothesis: kgobalo\n", "2019-11-27 14:21:33,186 Example #3\n", "2019-11-27 14:21:33,186 \tSource: Skills Development course .\n", "2019-11-27 14:21:33,186 \tReference: Khoso ya Tlhabololo ya Bokgoni .\n", "2019-11-27 14:21:33,186 \tHypothesis: Khoso ya Tlhabololo ya Bokgoni .\n", "2019-11-27 14:21:33,186 Validation result (greedy) at epoch 41, step 54000: bleu: 28.64, loss: 39321.5234, ppl: 4.0765, duration: 103.1132s\n", "2019-11-27 14:22:04,784 Epoch 41 Step: 54100 Batch Loss: 1.139061 Tokens per Sec: 7452, Lr: 0.000300\n", "2019-11-27 14:22:10,098 Epoch 41: total training loss 1836.27\n", "2019-11-27 14:22:10,098 EPOCH 42\n", "2019-11-27 14:22:36,882 Epoch 42 Step: 54200 Batch Loss: 1.289777 Tokens per Sec: 7467, Lr: 0.000300\n", "2019-11-27 14:23:08,259 Epoch 42 Step: 54300 Batch Loss: 1.276387 Tokens per Sec: 7413, Lr: 0.000300\n", "2019-11-27 14:23:39,887 Epoch 42 Step: 54400 Batch Loss: 1.205678 Tokens per Sec: 7501, Lr: 0.000300\n", "2019-11-27 14:24:11,323 Epoch 42 Step: 54500 Batch Loss: 1.614185 Tokens per Sec: 7514, Lr: 0.000300\n", "2019-11-27 14:24:43,407 Epoch 42 Step: 54600 Batch Loss: 1.463781 Tokens per Sec: 7664, Lr: 0.000300\n", "2019-11-27 14:25:15,417 Epoch 42 Step: 54700 Batch Loss: 1.189779 Tokens per Sec: 7665, Lr: 0.000300\n", "2019-11-27 14:25:47,187 Epoch 42 Step: 54800 Batch Loss: 1.459378 Tokens per Sec: 7632, Lr: 0.000300\n", "2019-11-27 14:26:18,661 Epoch 42 Step: 54900 Batch Loss: 1.482843 Tokens per Sec: 7594, Lr: 0.000300\n", "2019-11-27 14:26:50,473 Epoch 42 Step: 55000 Batch Loss: 1.503228 Tokens per Sec: 7503, Lr: 0.000300\n", "2019-11-27 14:27:22,732 Epoch 42 Step: 55100 Batch Loss: 1.516768 Tokens per Sec: 7726, Lr: 0.000300\n", "2019-11-27 14:27:54,430 Epoch 42 Step: 55200 Batch Loss: 1.263331 Tokens per Sec: 7436, Lr: 0.000300\n", "2019-11-27 14:28:26,212 Epoch 42 Step: 55300 Batch Loss: 1.488422 Tokens per Sec: 7567, Lr: 0.000300\n", "2019-11-27 14:28:58,134 Epoch 42 Step: 55400 Batch Loss: 1.516925 Tokens per Sec: 7468, Lr: 0.000300\n", "2019-11-27 14:29:11,095 Epoch 42: total training loss 1834.94\n", "2019-11-27 14:29:11,095 EPOCH 43\n", "2019-11-27 14:29:30,178 Epoch 43 Step: 55500 Batch Loss: 1.459202 Tokens per Sec: 7637, Lr: 0.000300\n", "2019-11-27 14:30:01,974 Epoch 43 Step: 55600 Batch Loss: 1.101120 Tokens per Sec: 7441, Lr: 0.000300\n", "2019-11-27 14:30:33,892 Epoch 43 Step: 55700 Batch Loss: 1.412910 Tokens per Sec: 7634, Lr: 0.000300\n", "2019-11-27 14:31:05,893 Epoch 43 Step: 55800 Batch Loss: 1.497028 Tokens per Sec: 7590, Lr: 0.000300\n", "2019-11-27 14:31:37,877 Epoch 43 Step: 55900 Batch Loss: 1.098422 Tokens per Sec: 7591, Lr: 0.000300\n", "2019-11-27 14:32:09,589 Epoch 43 Step: 56000 Batch Loss: 1.410753 Tokens per Sec: 7439, Lr: 0.000300\n", "2019-11-27 14:33:52,418 Example #0\n", "2019-11-27 14:33:52,419 \tSource: The services are free in South Africa\n", "2019-11-27 14:33:52,419 \tReference: Mo Aforikaborwa , ditirelo di neelwa kwa ntle ga go duelelwa .\n", "2019-11-27 14:33:52,419 \tHypothesis: Ditirelo di gololesegile mo Aforikaborwa\n", "2019-11-27 14:33:52,419 Example #1\n", "2019-11-27 14:33:52,419 \tSource: ( 7 ) A person or community dispossessed of property after 19 June 1913 as a result of past racially discriminatory laws or practices is entitled , to the extent provided by an Act of Parliament , either to restitution of that property or to equitable redress .\n", "2019-11-27 14:33:52,419 \tReference: ( 7 ) Motho kgotsa setlhopha sa magae se se amogilweng dithoto morago ga la 19 Seetebosigo 1913 ka ntlha ya melao ya kgale ya tlhotlhololo kgotsa ditlwaelo , se tshwanelwa , go ya ka thulaganyetso ya Molao wa Palamente go busediwa dithoto tseo , kgotsa tshiamisetso e e e kalekanang .\n", "2019-11-27 14:33:52,420 \tHypothesis: ( 7 ) Motho kgotsa setšhaba se se neng sa sa amogelwa thoto morago ga 19 Seetebosigo 1913 ka ntlha ya melao ya kgethololo e e fetileng kgotsa mekgwatiriso e tshwanelwa ke go nna teng , go ya ka Molao wa Palamente , e ka nna ya go dira gore dithoto kgotsa go siame go siamisa maemo a a siameng a go dira gore go nne le tekatekano .\n", "2019-11-27 14:33:52,420 Example #2\n", "2019-11-27 14:33:52,420 \tSource: ruling\n", "2019-11-27 14:33:52,420 \tReference: tshwetsotaolo\n", "2019-11-27 14:33:52,420 \tHypothesis: go ikuela\n", "2019-11-27 14:33:52,420 Example #3\n", "2019-11-27 14:33:52,421 \tSource: Skills Development course .\n", "2019-11-27 14:33:52,421 \tReference: Khoso ya Tlhabololo ya Bokgoni .\n", "2019-11-27 14:33:52,421 \tHypothesis: Khoso ya Tlhabololo ya Bokgoni .\n", "2019-11-27 14:33:52,421 Validation result (greedy) at epoch 43, step 56000: bleu: 28.71, loss: 39422.4219, ppl: 4.0912, duration: 102.8313s\n", "2019-11-27 14:34:24,213 Epoch 43 Step: 56100 Batch Loss: 1.174836 Tokens per Sec: 7592, Lr: 0.000300\n", "2019-11-27 14:34:56,059 Epoch 43 Step: 56200 Batch Loss: 1.482411 Tokens per Sec: 7514, Lr: 0.000300\n", "2019-11-27 14:35:27,747 Epoch 43 Step: 56300 Batch Loss: 1.527804 Tokens per Sec: 7469, Lr: 0.000300\n", "2019-11-27 14:35:59,766 Epoch 43 Step: 56400 Batch Loss: 1.318311 Tokens per Sec: 7484, Lr: 0.000300\n", "2019-11-27 14:36:31,424 Epoch 43 Step: 56500 Batch Loss: 1.182300 Tokens per Sec: 7471, Lr: 0.000300\n", "2019-11-27 14:37:03,129 Epoch 43 Step: 56600 Batch Loss: 1.571154 Tokens per Sec: 7491, Lr: 0.000300\n", "2019-11-27 14:37:35,349 Epoch 43 Step: 56700 Batch Loss: 1.582292 Tokens per Sec: 7663, Lr: 0.000300\n", "2019-11-27 14:37:55,380 Epoch 43: total training loss 1824.74\n", "2019-11-27 14:37:55,380 EPOCH 44\n", "2019-11-27 14:38:07,118 Epoch 44 Step: 56800 Batch Loss: 1.270024 Tokens per Sec: 7508, Lr: 0.000300\n", "2019-11-27 14:38:39,021 Epoch 44 Step: 56900 Batch Loss: 1.422598 Tokens per Sec: 7425, Lr: 0.000300\n", "2019-11-27 14:39:10,636 Epoch 44 Step: 57000 Batch Loss: 1.060679 Tokens per Sec: 7442, Lr: 0.000300\n", "2019-11-27 14:39:42,470 Epoch 44 Step: 57100 Batch Loss: 1.481881 Tokens per Sec: 7510, Lr: 0.000300\n", "2019-11-27 14:40:14,394 Epoch 44 Step: 57200 Batch Loss: 1.496193 Tokens per Sec: 7645, Lr: 0.000300\n", "2019-11-27 14:40:46,161 Epoch 44 Step: 57300 Batch Loss: 1.265997 Tokens per Sec: 7547, Lr: 0.000300\n", "2019-11-27 14:41:17,849 Epoch 44 Step: 57400 Batch Loss: 1.439737 Tokens per Sec: 7622, Lr: 0.000300\n", "2019-11-27 14:41:49,929 Epoch 44 Step: 57500 Batch Loss: 1.501039 Tokens per Sec: 7666, Lr: 0.000300\n", "2019-11-27 14:42:22,032 Epoch 44 Step: 57600 Batch Loss: 1.420997 Tokens per Sec: 7612, Lr: 0.000300\n", "2019-11-27 14:42:53,858 Epoch 44 Step: 57700 Batch Loss: 1.113715 Tokens per Sec: 7489, Lr: 0.000300\n", "2019-11-27 14:43:25,833 Epoch 44 Step: 57800 Batch Loss: 1.511494 Tokens per Sec: 7419, Lr: 0.000300\n", "2019-11-27 14:43:57,631 Epoch 44 Step: 57900 Batch Loss: 1.443857 Tokens per Sec: 7460, Lr: 0.000300\n", "2019-11-27 14:44:30,149 Epoch 44 Step: 58000 Batch Loss: 1.358672 Tokens per Sec: 7589, Lr: 0.000300\n", "2019-11-27 14:46:12,796 Hooray! New best validation result [ppl]!\n", "2019-11-27 14:46:12,796 Saving new checkpoint.\n", "2019-11-27 14:46:13,112 Example #0\n", "2019-11-27 14:46:13,112 \tSource: The services are free in South Africa\n", "2019-11-27 14:46:13,112 \tReference: Mo Aforikaborwa , ditirelo di neelwa kwa ntle ga go duelelwa .\n", "2019-11-27 14:46:13,113 \tHypothesis: Ditirelo di gololesegile mo Aforikaborwa\n", "2019-11-27 14:46:13,113 Example #1\n", "2019-11-27 14:46:13,113 \tSource: ( 7 ) A person or community dispossessed of property after 19 June 1913 as a result of past racially discriminatory laws or practices is entitled , to the extent provided by an Act of Parliament , either to restitution of that property or to equitable redress .\n", "2019-11-27 14:46:13,113 \tReference: ( 7 ) Motho kgotsa setlhopha sa magae se se amogilweng dithoto morago ga la 19 Seetebosigo 1913 ka ntlha ya melao ya kgale ya tlhotlhololo kgotsa ditlwaelo , se tshwanelwa , go ya ka thulaganyetso ya Molao wa Palamente go busediwa dithoto tseo , kgotsa tshiamisetso e e e kalekanang .\n", "2019-11-27 14:46:13,113 \tHypothesis: ( 7 ) Motho kgotsa baagi ba ba neng ba sa amogele thoto morago ga 19 Seetebosigo 1913 ka ntlha ya melao ya kgethololo kgotsa mekgwa e e farologaneng e tshwanelwa ke go nna ka fa tlase ga Molao wa Palamente , e ka nna go iphitlhela ka gone kgotsa go siamisa kgatelelo eo .\n", "2019-11-27 14:46:13,113 Example #2\n", "2019-11-27 14:46:13,114 \tSource: ruling\n", "2019-11-27 14:46:13,114 \tReference: tshwetsotaolo\n", "2019-11-27 14:46:13,114 \tHypothesis: kgogelo\n", "2019-11-27 14:46:13,114 Example #3\n", "2019-11-27 14:46:13,114 \tSource: Skills Development course .\n", "2019-11-27 14:46:13,114 \tReference: Khoso ya Tlhabololo ya Bokgoni .\n", "2019-11-27 14:46:13,115 \tHypothesis: Khoso ya Tlhabololo ya Bokgoni .\n", "2019-11-27 14:46:13,115 Validation result (greedy) at epoch 44, step 58000: bleu: 29.13, loss: 39122.0547, ppl: 4.0476, duration: 102.9657s\n", "2019-11-27 14:46:39,360 Epoch 44: total training loss 1813.26\n", "2019-11-27 14:46:39,360 EPOCH 45\n", "2019-11-27 14:46:45,053 Epoch 45 Step: 58100 Batch Loss: 1.115102 Tokens per Sec: 7484, Lr: 0.000300\n", "2019-11-27 14:47:16,970 Epoch 45 Step: 58200 Batch Loss: 1.222566 Tokens per Sec: 7611, Lr: 0.000300\n", "2019-11-27 14:47:48,597 Epoch 45 Step: 58300 Batch Loss: 1.266979 Tokens per Sec: 7431, Lr: 0.000300\n", "2019-11-27 14:48:20,315 Epoch 45 Step: 58400 Batch Loss: 1.313282 Tokens per Sec: 7438, Lr: 0.000300\n", "2019-11-27 14:48:52,255 Epoch 45 Step: 58500 Batch Loss: 1.546566 Tokens per Sec: 7619, Lr: 0.000300\n", "2019-11-27 14:49:24,119 Epoch 45 Step: 58600 Batch Loss: 1.222806 Tokens per Sec: 7580, Lr: 0.000300\n", "2019-11-27 14:49:55,912 Epoch 45 Step: 58700 Batch Loss: 1.368399 Tokens per Sec: 7611, Lr: 0.000300\n", "2019-11-27 14:50:27,371 Epoch 45 Step: 58800 Batch Loss: 1.442108 Tokens per Sec: 7652, Lr: 0.000300\n", "2019-11-27 14:50:59,027 Epoch 45 Step: 58900 Batch Loss: 1.419443 Tokens per Sec: 7560, Lr: 0.000300\n", "2019-11-27 14:51:31,098 Epoch 45 Step: 59000 Batch Loss: 1.513233 Tokens per Sec: 7513, Lr: 0.000300\n", "2019-11-27 14:52:03,251 Epoch 45 Step: 59100 Batch Loss: 1.199212 Tokens per Sec: 7597, Lr: 0.000300\n", "2019-11-27 14:52:35,107 Epoch 45 Step: 59200 Batch Loss: 1.367877 Tokens per Sec: 7643, Lr: 0.000300\n", "2019-11-27 14:53:06,869 Epoch 45 Step: 59300 Batch Loss: 1.271107 Tokens per Sec: 7632, Lr: 0.000300\n", "2019-11-27 14:53:38,389 Epoch 45 Step: 59400 Batch Loss: 1.328347 Tokens per Sec: 7533, Lr: 0.000300\n", "2019-11-27 14:53:39,050 Epoch 45: total training loss 1803.34\n", "2019-11-27 14:53:39,050 EPOCH 46\n", "2019-11-27 14:54:10,583 Epoch 46 Step: 59500 Batch Loss: 1.449388 Tokens per Sec: 7627, Lr: 0.000300\n", "2019-11-27 14:54:42,535 Epoch 46 Step: 59600 Batch Loss: 1.366818 Tokens per Sec: 7498, Lr: 0.000300\n", "2019-11-27 14:55:14,420 Epoch 46 Step: 59700 Batch Loss: 1.140736 Tokens per Sec: 7558, Lr: 0.000300\n", "2019-11-27 14:55:46,239 Epoch 46 Step: 59800 Batch Loss: 1.372903 Tokens per Sec: 7568, Lr: 0.000300\n", "2019-11-27 14:56:17,754 Epoch 46 Step: 59900 Batch Loss: 1.226605 Tokens per Sec: 7396, Lr: 0.000300\n", "2019-11-27 14:56:49,923 Epoch 46 Step: 60000 Batch Loss: 1.306621 Tokens per Sec: 7804, Lr: 0.000300\n", "2019-11-27 14:58:32,402 Hooray! New best validation result [ppl]!\n", "2019-11-27 14:58:32,402 Saving new checkpoint.\n", "2019-11-27 14:58:32,715 Example #0\n", "2019-11-27 14:58:32,716 \tSource: The services are free in South Africa\n", "2019-11-27 14:58:32,716 \tReference: Mo Aforikaborwa , ditirelo di neelwa kwa ntle ga go duelelwa .\n", "2019-11-27 14:58:32,716 \tHypothesis: Ditirelo di gololesegile mo Aforikaborwa\n", "2019-11-27 14:58:32,716 Example #1\n", "2019-11-27 14:58:32,716 \tSource: ( 7 ) A person or community dispossessed of property after 19 June 1913 as a result of past racially discriminatory laws or practices is entitled , to the extent provided by an Act of Parliament , either to restitution of that property or to equitable redress .\n", "2019-11-27 14:58:32,716 \tReference: ( 7 ) Motho kgotsa setlhopha sa magae se se amogilweng dithoto morago ga la 19 Seetebosigo 1913 ka ntlha ya melao ya kgale ya tlhotlhololo kgotsa ditlwaelo , se tshwanelwa , go ya ka thulaganyetso ya Molao wa Palamente go busediwa dithoto tseo , kgotsa tshiamisetso e e e kalekanang .\n", "2019-11-27 14:58:32,716 \tHypothesis: ( 7 ) Motho kgotsa baagi ba ba nang le dithoto morago ga 19 Seetebosigo 1913 ka ntlha ya melao ya kgethololo e e fetileng kgotsa mekgwa e e nang le taolo , go ya ka Molao wa Palamente , e ka nna go ikobela thoto eo kgotsa go siamisa kgatelelo e e lekanang .\n", "2019-11-27 14:58:32,716 Example #2\n", "2019-11-27 14:58:32,717 \tSource: ruling\n", "2019-11-27 14:58:32,717 \tReference: tshwetsotaolo\n", "2019-11-27 14:58:32,717 \tHypothesis: katlholo\n", "2019-11-27 14:58:32,717 Example #3\n", "2019-11-27 14:58:32,717 \tSource: Skills Development course .\n", "2019-11-27 14:58:32,717 \tReference: Khoso ya Tlhabololo ya Bokgoni .\n", "2019-11-27 14:58:32,717 \tHypothesis: Khoso ya Tlhabololo ya Bokgoni .\n", "2019-11-27 14:58:32,717 Validation result (greedy) at epoch 46, step 60000: bleu: 28.96, loss: 38971.5664, ppl: 4.0259, duration: 102.7937s\n", "2019-11-27 14:59:04,306 Epoch 46 Step: 60100 Batch Loss: 1.176928 Tokens per Sec: 7480, Lr: 0.000300\n", "2019-11-27 14:59:36,039 Epoch 46 Step: 60200 Batch Loss: 1.490799 Tokens per Sec: 7574, Lr: 0.000300\n", "2019-11-27 15:00:07,879 Epoch 46 Step: 60300 Batch Loss: 1.391736 Tokens per Sec: 7589, Lr: 0.000300\n", "2019-11-27 15:00:39,585 Epoch 46 Step: 60400 Batch Loss: 1.466212 Tokens per Sec: 7550, Lr: 0.000300\n", "2019-11-27 15:01:11,601 Epoch 46 Step: 60500 Batch Loss: 1.276558 Tokens per Sec: 7603, Lr: 0.000300\n", "2019-11-27 15:01:43,580 Epoch 46 Step: 60600 Batch Loss: 1.222775 Tokens per Sec: 7712, Lr: 0.000300\n", "2019-11-27 15:02:14,963 Epoch 46 Step: 60700 Batch Loss: 1.283468 Tokens per Sec: 7465, Lr: 0.000300\n", "2019-11-27 15:02:21,302 Epoch 46: total training loss 1787.46\n", "2019-11-27 15:02:21,302 EPOCH 47\n", "2019-11-27 15:02:46,690 Epoch 47 Step: 60800 Batch Loss: 1.527889 Tokens per Sec: 7574, Lr: 0.000300\n", "2019-11-27 15:03:18,179 Epoch 47 Step: 60900 Batch Loss: 1.266240 Tokens per Sec: 7498, Lr: 0.000300\n", "2019-11-27 15:03:49,947 Epoch 47 Step: 61000 Batch Loss: 1.301882 Tokens per Sec: 7544, Lr: 0.000300\n", "2019-11-27 15:04:21,597 Epoch 47 Step: 61100 Batch Loss: 1.362390 Tokens per Sec: 7585, Lr: 0.000300\n", "2019-11-27 15:04:53,539 Epoch 47 Step: 61200 Batch Loss: 1.445703 Tokens per Sec: 7490, Lr: 0.000300\n", "2019-11-27 15:05:25,733 Epoch 47 Step: 61300 Batch Loss: 1.230418 Tokens per Sec: 7478, Lr: 0.000300\n", "2019-11-27 15:05:57,382 Epoch 47 Step: 61400 Batch Loss: 1.068748 Tokens per Sec: 7476, Lr: 0.000300\n", "2019-11-27 15:06:29,570 Epoch 47 Step: 61500 Batch Loss: 1.411093 Tokens per Sec: 7545, Lr: 0.000300\n", "2019-11-27 15:07:01,545 Epoch 47 Step: 61600 Batch Loss: 1.512949 Tokens per Sec: 7503, Lr: 0.000300\n", "2019-11-27 15:07:33,790 Epoch 47 Step: 61700 Batch Loss: 1.429171 Tokens per Sec: 7613, Lr: 0.000300\n", "2019-11-27 15:08:06,066 Epoch 47 Step: 61800 Batch Loss: 1.377141 Tokens per Sec: 7638, Lr: 0.000300\n", "2019-11-27 15:08:37,831 Epoch 47 Step: 61900 Batch Loss: 1.432819 Tokens per Sec: 7549, Lr: 0.000300\n", "2019-11-27 15:09:09,783 Epoch 47 Step: 62000 Batch Loss: 1.529558 Tokens per Sec: 7603, Lr: 0.000300\n", "2019-11-27 15:10:52,271 Hooray! New best validation result [ppl]!\n", "2019-11-27 15:10:52,271 Saving new checkpoint.\n", "2019-11-27 15:10:52,558 Example #0\n", "2019-11-27 15:10:52,558 \tSource: The services are free in South Africa\n", "2019-11-27 15:10:52,558 \tReference: Mo Aforikaborwa , ditirelo di neelwa kwa ntle ga go duelelwa .\n", "2019-11-27 15:10:52,558 \tHypothesis: Ditirelo di gololesegile mo Aforikaborwa\n", "2019-11-27 15:10:52,559 Example #1\n", "2019-11-27 15:10:52,559 \tSource: ( 7 ) A person or community dispossessed of property after 19 June 1913 as a result of past racially discriminatory laws or practices is entitled , to the extent provided by an Act of Parliament , either to restitution of that property or to equitable redress .\n", "2019-11-27 15:10:52,559 \tReference: ( 7 ) Motho kgotsa setlhopha sa magae se se amogilweng dithoto morago ga la 19 Seetebosigo 1913 ka ntlha ya melao ya kgale ya tlhotlhololo kgotsa ditlwaelo , se tshwanelwa , go ya ka thulaganyetso ya Molao wa Palamente go busediwa dithoto tseo , kgotsa tshiamisetso e e e kalekanang .\n", "2019-11-27 15:10:52,559 \tHypothesis: ( 7 ) Motho kgotsa baagi ba ba nang le dithoto morago ga 19 Seetebosigo 1913 ka ntlha ya melao ya kgethololo e e fetileng kgotsa mekgwa e e nang le tshwanelo ya go ya ka Molao wa Palamente , e ka nna go busetsa morago ga go bona dithoto tseo kgotsa go lekanya kgatelelo e e lekanang .\n", "2019-11-27 15:10:52,559 Example #2\n", "2019-11-27 15:10:52,559 \tSource: ruling\n", "2019-11-27 15:10:52,560 \tReference: tshwetsotaolo\n", "2019-11-27 15:10:52,560 \tHypothesis: puduso\n", "2019-11-27 15:10:52,560 Example #3\n", "2019-11-27 15:10:52,560 \tSource: Skills Development course .\n", "2019-11-27 15:10:52,560 \tReference: Khoso ya Tlhabololo ya Bokgoni .\n", "2019-11-27 15:10:52,560 \tHypothesis: Khoso ya Tlhabololo ya Bokgoni .\n", "2019-11-27 15:10:52,560 Validation result (greedy) at epoch 47, step 62000: bleu: 29.27, loss: 38802.1055, ppl: 4.0015, duration: 102.7768s\n", "2019-11-27 15:11:05,329 Epoch 47: total training loss 1782.25\n", "2019-11-27 15:11:05,329 EPOCH 48\n", "2019-11-27 15:11:24,279 Epoch 48 Step: 62100 Batch Loss: 1.375691 Tokens per Sec: 7572, Lr: 0.000300\n", "2019-11-27 15:11:56,103 Epoch 48 Step: 62200 Batch Loss: 1.506102 Tokens per Sec: 7467, Lr: 0.000300\n", "2019-11-27 15:12:27,756 Epoch 48 Step: 62300 Batch Loss: 1.045380 Tokens per Sec: 7606, Lr: 0.000300\n", "2019-11-27 15:12:58,859 Epoch 48 Step: 62400 Batch Loss: 1.267679 Tokens per Sec: 7406, Lr: 0.000300\n", "2019-11-27 15:13:30,361 Epoch 48 Step: 62500 Batch Loss: 1.342866 Tokens per Sec: 7602, Lr: 0.000300\n", "2019-11-27 15:14:02,310 Epoch 48 Step: 62600 Batch Loss: 1.169056 Tokens per Sec: 7658, Lr: 0.000300\n", "2019-11-27 15:14:33,732 Epoch 48 Step: 62700 Batch Loss: 1.044773 Tokens per Sec: 7556, Lr: 0.000300\n", "2019-11-27 15:15:05,920 Epoch 48 Step: 62800 Batch Loss: 1.212111 Tokens per Sec: 7639, Lr: 0.000300\n", "2019-11-27 15:15:37,591 Epoch 48 Step: 62900 Batch Loss: 1.547888 Tokens per Sec: 7587, Lr: 0.000300\n", "2019-11-27 15:16:09,009 Epoch 48 Step: 63000 Batch Loss: 1.432867 Tokens per Sec: 7472, Lr: 0.000300\n", "2019-11-27 15:16:40,776 Epoch 48 Step: 63100 Batch Loss: 1.465696 Tokens per Sec: 7674, Lr: 0.000300\n", "2019-11-27 15:17:12,548 Epoch 48 Step: 63200 Batch Loss: 1.291032 Tokens per Sec: 7549, Lr: 0.000300\n", "2019-11-27 15:17:44,185 Epoch 48 Step: 63300 Batch Loss: 1.418721 Tokens per Sec: 7517, Lr: 0.000300\n", "2019-11-27 15:18:04,957 Epoch 48: total training loss 1781.67\n", "2019-11-27 15:18:04,957 EPOCH 49\n", "2019-11-27 15:18:16,208 Epoch 49 Step: 63400 Batch Loss: 1.401556 Tokens per Sec: 7504, Lr: 0.000300\n", "2019-11-27 15:18:48,136 Epoch 49 Step: 63500 Batch Loss: 1.087506 Tokens per Sec: 7707, Lr: 0.000300\n", "2019-11-27 15:19:20,033 Epoch 49 Step: 63600 Batch Loss: 1.170051 Tokens per Sec: 7632, Lr: 0.000300\n", "2019-11-27 15:19:51,564 Epoch 49 Step: 63700 Batch Loss: 1.378256 Tokens per Sec: 7569, Lr: 0.000300\n", "2019-11-27 15:20:23,197 Epoch 49 Step: 63800 Batch Loss: 1.342472 Tokens per Sec: 7521, Lr: 0.000300\n", "2019-11-27 15:20:54,823 Epoch 49 Step: 63900 Batch Loss: 1.270379 Tokens per Sec: 7494, Lr: 0.000300\n", "2019-11-27 15:21:26,367 Epoch 49 Step: 64000 Batch Loss: 1.572159 Tokens per Sec: 7618, Lr: 0.000300\n", "2019-11-27 15:23:08,526 Hooray! New best validation result [ppl]!\n", "2019-11-27 15:23:08,527 Saving new checkpoint.\n", "2019-11-27 15:23:08,824 Example #0\n", "2019-11-27 15:23:08,825 \tSource: The services are free in South Africa\n", "2019-11-27 15:23:08,825 \tReference: Mo Aforikaborwa , ditirelo di neelwa kwa ntle ga go duelelwa .\n", "2019-11-27 15:23:08,825 \tHypothesis: Ditirelo di gololesegile mo Aforikaborwa\n", "2019-11-27 15:23:08,825 Example #1\n", "2019-11-27 15:23:08,825 \tSource: ( 7 ) A person or community dispossessed of property after 19 June 1913 as a result of past racially discriminatory laws or practices is entitled , to the extent provided by an Act of Parliament , either to restitution of that property or to equitable redress .\n", "2019-11-27 15:23:08,826 \tReference: ( 7 ) Motho kgotsa setlhopha sa magae se se amogilweng dithoto morago ga la 19 Seetebosigo 1913 ka ntlha ya melao ya kgale ya tlhotlhololo kgotsa ditlwaelo , se tshwanelwa , go ya ka thulaganyetso ya Molao wa Palamente go busediwa dithoto tseo , kgotsa tshiamisetso e e e kalekanang .\n", "2019-11-27 15:23:08,826 \tHypothesis: ( 7 ) Motho kgotsa baagi ba ba nang le thoto morago ga 19 Seetebosigo 1913 ka ntlha ya melao ya kgethololo e e fetileng kgotsa mekgwatiriso e e nang le tetla , go ya ka Molao wa Palamente , e ka nna go phimola thoto eo kgotsa go siamisa kgatelelo e e lekanang .\n", "2019-11-27 15:23:08,826 Example #2\n", "2019-11-27 15:23:08,826 \tSource: ruling\n", "2019-11-27 15:23:08,826 \tReference: tshwetsotaolo\n", "2019-11-27 15:23:08,826 \tHypothesis: tshutiso\n", "2019-11-27 15:23:08,826 Example #3\n", "2019-11-27 15:23:08,827 \tSource: Skills Development course .\n", "2019-11-27 15:23:08,827 \tReference: Khoso ya Tlhabololo ya Bokgoni .\n", "2019-11-27 15:23:08,827 \tHypothesis: Khoso ya Tlhabololo ya Bokgoni .\n", "2019-11-27 15:23:08,827 Validation result (greedy) at epoch 49, step 64000: bleu: 29.14, loss: 38765.7656, ppl: 3.9964, duration: 102.4590s\n", "2019-11-27 15:23:40,476 Epoch 49 Step: 64100 Batch Loss: 1.333125 Tokens per Sec: 7636, Lr: 0.000300\n", "2019-11-27 15:24:12,186 Epoch 49 Step: 64200 Batch Loss: 1.603204 Tokens per Sec: 7556, Lr: 0.000300\n", "2019-11-27 15:24:43,963 Epoch 49 Step: 64300 Batch Loss: 1.200348 Tokens per Sec: 7623, Lr: 0.000300\n", "2019-11-27 15:25:16,123 Epoch 49 Step: 64400 Batch Loss: 1.436096 Tokens per Sec: 7711, Lr: 0.000300\n", "2019-11-27 15:25:47,695 Epoch 49 Step: 64500 Batch Loss: 1.390680 Tokens per Sec: 7515, Lr: 0.000300\n", "2019-11-27 15:26:19,493 Epoch 49 Step: 64600 Batch Loss: 1.131863 Tokens per Sec: 7533, Lr: 0.000300\n", "2019-11-27 15:26:45,444 Epoch 49: total training loss 1762.71\n", "2019-11-27 15:26:45,444 EPOCH 50\n", "2019-11-27 15:26:51,202 Epoch 50 Step: 64700 Batch Loss: 1.237172 Tokens per Sec: 7418, Lr: 0.000300\n", "2019-11-27 15:27:22,721 Epoch 50 Step: 64800 Batch Loss: 1.419024 Tokens per Sec: 7457, Lr: 0.000300\n", "2019-11-27 15:27:54,467 Epoch 50 Step: 64900 Batch Loss: 1.270912 Tokens per Sec: 7589, Lr: 0.000300\n", "2019-11-27 15:28:26,507 Epoch 50 Step: 65000 Batch Loss: 1.447750 Tokens per Sec: 7537, Lr: 0.000300\n", "2019-11-27 15:28:58,252 Epoch 50 Step: 65100 Batch Loss: 1.215533 Tokens per Sec: 7581, Lr: 0.000300\n", "2019-11-27 15:29:29,970 Epoch 50 Step: 65200 Batch Loss: 1.446372 Tokens per Sec: 7659, Lr: 0.000300\n", "2019-11-27 15:30:01,724 Epoch 50 Step: 65300 Batch Loss: 1.085642 Tokens per Sec: 7454, Lr: 0.000300\n", "2019-11-27 15:30:33,595 Epoch 50 Step: 65400 Batch Loss: 1.320897 Tokens per Sec: 7537, Lr: 0.000300\n", "2019-11-27 15:31:05,445 Epoch 50 Step: 65500 Batch Loss: 1.199773 Tokens per Sec: 7674, Lr: 0.000300\n", "2019-11-27 15:31:37,076 Epoch 50 Step: 65600 Batch Loss: 1.413032 Tokens per Sec: 7527, Lr: 0.000300\n", "2019-11-27 15:32:08,711 Epoch 50 Step: 65700 Batch Loss: 1.416163 Tokens per Sec: 7710, Lr: 0.000300\n", "2019-11-27 15:32:40,508 Epoch 50 Step: 65800 Batch Loss: 1.442357 Tokens per Sec: 7553, Lr: 0.000300\n", "2019-11-27 15:33:12,063 Epoch 50 Step: 65900 Batch Loss: 1.497608 Tokens per Sec: 7647, Lr: 0.000300\n", "2019-11-27 15:33:43,856 Epoch 50 Step: 66000 Batch Loss: 1.553019 Tokens per Sec: 7686, Lr: 0.000300\n", "2019-11-27 15:35:26,483 Hooray! New best validation result [ppl]!\n", "2019-11-27 15:35:26,483 Saving new checkpoint.\n", "2019-11-27 15:35:26,784 Example #0\n", "2019-11-27 15:35:26,784 \tSource: The services are free in South Africa\n", "2019-11-27 15:35:26,784 \tReference: Mo Aforikaborwa , ditirelo di neelwa kwa ntle ga go duelelwa .\n", "2019-11-27 15:35:26,784 \tHypothesis: Ditirelo di gololesegile mo Aforikaborwa\n", "2019-11-27 15:35:26,784 Example #1\n", "2019-11-27 15:35:26,785 \tSource: ( 7 ) A person or community dispossessed of property after 19 June 1913 as a result of past racially discriminatory laws or practices is entitled , to the extent provided by an Act of Parliament , either to restitution of that property or to equitable redress .\n", "2019-11-27 15:35:26,785 \tReference: ( 7 ) Motho kgotsa setlhopha sa magae se se amogilweng dithoto morago ga la 19 Seetebosigo 1913 ka ntlha ya melao ya kgale ya tlhotlhololo kgotsa ditlwaelo , se tshwanelwa , go ya ka thulaganyetso ya Molao wa Palamente go busediwa dithoto tseo , kgotsa tshiamisetso e e e kalekanang .\n", "2019-11-27 15:35:26,785 \tHypothesis: ( 7 ) Motho kgotsa baagi ba ba neng ba sa amogele thoto morago ga 19 Seetebosigo 1913 ka ntlha ya melao ya kgethololo e e fetileng kgotsa mekgwatiriso e tshwanelwa ke go nna teng , go ya ka Molao wa Palamente , e ka nna go busediwa ga dithoto kgotsa go siamisa go siamisa .\n", "2019-11-27 15:35:26,785 Example #2\n", "2019-11-27 15:35:26,786 \tSource: ruling\n", "2019-11-27 15:35:26,786 \tReference: tshwetsotaolo\n", "2019-11-27 15:35:26,786 \tHypothesis: katlholo\n", "2019-11-27 15:35:26,786 Example #3\n", "2019-11-27 15:35:26,787 \tSource: Skills Development course .\n", "2019-11-27 15:35:26,787 \tReference: Khoso ya Tlhabololo ya Bokgoni .\n", "2019-11-27 15:35:26,787 \tHypothesis: Khoso ya Tlhabololo ya Bokgoni .\n", "2019-11-27 15:35:26,787 Validation result (greedy) at epoch 50, step 66000: bleu: 28.72, loss: 38558.0391, ppl: 3.9668, duration: 102.9308s\n", "2019-11-27 15:35:27,351 Epoch 50: total training loss 1759.20\n", "2019-11-27 15:35:27,352 EPOCH 51\n", "2019-11-27 15:35:58,633 Epoch 51 Step: 66100 Batch Loss: 1.501051 Tokens per Sec: 7509, Lr: 0.000300\n", "2019-11-27 15:36:30,332 Epoch 51 Step: 66200 Batch Loss: 1.150266 Tokens per Sec: 7544, Lr: 0.000300\n", "2019-11-27 15:37:02,685 Epoch 51 Step: 66300 Batch Loss: 1.043230 Tokens per Sec: 7408, Lr: 0.000300\n", "2019-11-27 15:37:34,359 Epoch 51 Step: 66400 Batch Loss: 1.087331 Tokens per Sec: 7536, Lr: 0.000300\n", "2019-11-27 15:38:06,159 Epoch 51 Step: 66500 Batch Loss: 1.371544 Tokens per Sec: 7626, Lr: 0.000300\n", "2019-11-27 15:38:38,052 Epoch 51 Step: 66600 Batch Loss: 1.057289 Tokens per Sec: 7626, Lr: 0.000300\n", "2019-11-27 15:39:09,900 Epoch 51 Step: 66700 Batch Loss: 1.366356 Tokens per Sec: 7517, Lr: 0.000300\n", "2019-11-27 15:39:41,457 Epoch 51 Step: 66800 Batch Loss: 1.169267 Tokens per Sec: 7399, Lr: 0.000300\n", "2019-11-27 15:40:13,193 Epoch 51 Step: 66900 Batch Loss: 1.186088 Tokens per Sec: 7616, Lr: 0.000300\n", "2019-11-27 15:40:44,998 Epoch 51 Step: 67000 Batch Loss: 1.123199 Tokens per Sec: 7742, Lr: 0.000300\n", "2019-11-27 15:41:16,925 Epoch 51 Step: 67100 Batch Loss: 1.286072 Tokens per Sec: 7517, Lr: 0.000300\n", "2019-11-27 15:41:48,346 Epoch 51 Step: 67200 Batch Loss: 1.324816 Tokens per Sec: 7672, Lr: 0.000300\n", "2019-11-27 15:42:20,074 Epoch 51 Step: 67300 Batch Loss: 0.985288 Tokens per Sec: 7655, Lr: 0.000300\n", "2019-11-27 15:42:27,081 Epoch 51: total training loss 1753.98\n", "2019-11-27 15:42:27,082 EPOCH 52\n", "2019-11-27 15:42:52,204 Epoch 52 Step: 67400 Batch Loss: 1.164200 Tokens per Sec: 7591, Lr: 0.000300\n", "2019-11-27 15:43:23,906 Epoch 52 Step: 67500 Batch Loss: 1.069906 Tokens per Sec: 7667, Lr: 0.000300\n", "2019-11-27 15:43:55,689 Epoch 52 Step: 67600 Batch Loss: 1.382591 Tokens per Sec: 7560, Lr: 0.000300\n", "2019-11-27 15:44:27,642 Epoch 52 Step: 67700 Batch Loss: 1.403090 Tokens per Sec: 7624, Lr: 0.000300\n", "2019-11-27 15:44:58,876 Epoch 52 Step: 67800 Batch Loss: 1.441465 Tokens per Sec: 7479, Lr: 0.000300\n", "2019-11-27 15:45:30,621 Epoch 52 Step: 67900 Batch Loss: 1.230926 Tokens per Sec: 7468, Lr: 0.000300\n", "2019-11-27 15:46:02,527 Epoch 52 Step: 68000 Batch Loss: 1.219287 Tokens per Sec: 7679, Lr: 0.000300\n", "2019-11-27 15:47:45,168 Hooray! New best validation result [ppl]!\n", "2019-11-27 15:47:45,169 Saving new checkpoint.\n", "2019-11-27 15:47:45,473 Example #0\n", "2019-11-27 15:47:45,473 \tSource: The services are free in South Africa\n", "2019-11-27 15:47:45,473 \tReference: Mo Aforikaborwa , ditirelo di neelwa kwa ntle ga go duelelwa .\n", "2019-11-27 15:47:45,474 \tHypothesis: Ditirelo di gololesegile mo Aforikaborwa\n", "2019-11-27 15:47:45,474 Example #1\n", "2019-11-27 15:47:45,474 \tSource: ( 7 ) A person or community dispossessed of property after 19 June 1913 as a result of past racially discriminatory laws or practices is entitled , to the extent provided by an Act of Parliament , either to restitution of that property or to equitable redress .\n", "2019-11-27 15:47:45,474 \tReference: ( 7 ) Motho kgotsa setlhopha sa magae se se amogilweng dithoto morago ga la 19 Seetebosigo 1913 ka ntlha ya melao ya kgale ya tlhotlhololo kgotsa ditlwaelo , se tshwanelwa , go ya ka thulaganyetso ya Molao wa Palamente go busediwa dithoto tseo , kgotsa tshiamisetso e e e kalekanang .\n", "2019-11-27 15:47:45,474 \tHypothesis: ( 7 ) Motho kgotsa baagi ba ba neng ba ikgatholositswe ka thoto morago ga 19 Seetebosigo 1913 ka ntlha ya melao ya kgethololo kgotsa mekgwatiriso e e fetileng e tshwanelwa ke go nna ka fa tlase ga Molao wa Palamente , e ka nna go ipuelela ga dithoto tseo kgotsa go siamisa go siamisa ga gagwe ka tekatekano .\n", "2019-11-27 15:47:45,474 Example #2\n", "2019-11-27 15:47:45,474 \tSource: ruling\n", "2019-11-27 15:47:45,475 \tReference: tshwetsotaolo\n", "2019-11-27 15:47:45,475 \tHypothesis: tshutiso\n", "2019-11-27 15:47:45,475 Example #3\n", "2019-11-27 15:47:45,475 \tSource: Skills Development course .\n", "2019-11-27 15:47:45,475 \tReference: Khoso ya Tlhabololo ya Bokgoni .\n", "2019-11-27 15:47:45,475 \tHypothesis: Khoso ya Tlhabololo ya Bokgoni .\n", "2019-11-27 15:47:45,475 Validation result (greedy) at epoch 52, step 68000: bleu: 28.78, loss: 38503.2031, ppl: 3.9590, duration: 102.9480s\n", "2019-11-27 15:48:17,109 Epoch 52 Step: 68100 Batch Loss: 1.531469 Tokens per Sec: 7544, Lr: 0.000300\n", "2019-11-27 15:48:48,827 Epoch 52 Step: 68200 Batch Loss: 1.517711 Tokens per Sec: 7665, Lr: 0.000300\n", "2019-11-27 15:49:20,781 Epoch 52 Step: 68300 Batch Loss: 0.983914 Tokens per Sec: 7684, Lr: 0.000300\n", "2019-11-27 15:49:52,253 Epoch 52 Step: 68400 Batch Loss: 1.396192 Tokens per Sec: 7486, Lr: 0.000300\n", "2019-11-27 15:50:23,806 Epoch 52 Step: 68500 Batch Loss: 1.204386 Tokens per Sec: 7471, Lr: 0.000300\n", "2019-11-27 15:50:55,657 Epoch 52 Step: 68600 Batch Loss: 1.363155 Tokens per Sec: 7530, Lr: 0.000300\n", "2019-11-27 15:51:09,350 Epoch 52: total training loss 1745.94\n", "2019-11-27 15:51:09,351 EPOCH 53\n", "2019-11-27 15:51:27,668 Epoch 53 Step: 68700 Batch Loss: 1.288749 Tokens per Sec: 7326, Lr: 0.000300\n", "2019-11-27 15:51:59,524 Epoch 53 Step: 68800 Batch Loss: 1.217283 Tokens per Sec: 7602, Lr: 0.000300\n", "2019-11-27 15:52:30,834 Epoch 53 Step: 68900 Batch Loss: 1.013817 Tokens per Sec: 7446, Lr: 0.000300\n", "2019-11-27 15:53:03,143 Epoch 53 Step: 69000 Batch Loss: 1.093165 Tokens per Sec: 7637, Lr: 0.000300\n", "2019-11-27 15:53:34,726 Epoch 53 Step: 69100 Batch Loss: 1.191089 Tokens per Sec: 7576, Lr: 0.000300\n", "2019-11-27 15:54:06,225 Epoch 53 Step: 69200 Batch Loss: 1.440638 Tokens per Sec: 7574, Lr: 0.000300\n", "2019-11-27 15:54:37,789 Epoch 53 Step: 69300 Batch Loss: 1.200159 Tokens per Sec: 7725, Lr: 0.000300\n", "2019-11-27 15:55:09,532 Epoch 53 Step: 69400 Batch Loss: 1.494443 Tokens per Sec: 7750, Lr: 0.000300\n", "2019-11-27 15:55:41,164 Epoch 53 Step: 69500 Batch Loss: 1.491504 Tokens per Sec: 7630, Lr: 0.000300\n", "2019-11-27 15:56:12,879 Epoch 53 Step: 69600 Batch Loss: 1.157951 Tokens per Sec: 7675, Lr: 0.000300\n", "2019-11-27 15:56:44,569 Epoch 53 Step: 69700 Batch Loss: 1.201052 Tokens per Sec: 7570, Lr: 0.000300\n", "2019-11-27 15:57:16,229 Epoch 53 Step: 69800 Batch Loss: 1.137955 Tokens per Sec: 7541, Lr: 0.000300\n", "2019-11-27 15:57:47,693 Epoch 53 Step: 69900 Batch Loss: 1.172042 Tokens per Sec: 7560, Lr: 0.000300\n", "2019-11-27 15:58:07,465 Epoch 53: total training loss 1739.46\n", "2019-11-27 15:58:07,465 EPOCH 54\n", "2019-11-27 15:58:19,115 Epoch 54 Step: 70000 Batch Loss: 1.397516 Tokens per Sec: 7437, Lr: 0.000300\n", "2019-11-27 16:00:01,272 Hooray! New best validation result [ppl]!\n", "2019-11-27 16:00:01,272 Saving new checkpoint.\n", "2019-11-27 16:00:01,558 Example #0\n", "2019-11-27 16:00:01,558 \tSource: The services are free in South Africa\n", "2019-11-27 16:00:01,558 \tReference: Mo Aforikaborwa , ditirelo di neelwa kwa ntle ga go duelelwa .\n", "2019-11-27 16:00:01,559 \tHypothesis: Ditirelo di gololesegile mo Aforikaborwa\n", "2019-11-27 16:00:01,559 Example #1\n", "2019-11-27 16:00:01,559 \tSource: ( 7 ) A person or community dispossessed of property after 19 June 1913 as a result of past racially discriminatory laws or practices is entitled , to the extent provided by an Act of Parliament , either to restitution of that property or to equitable redress .\n", "2019-11-27 16:00:01,559 \tReference: ( 7 ) Motho kgotsa setlhopha sa magae se se amogilweng dithoto morago ga la 19 Seetebosigo 1913 ka ntlha ya melao ya kgale ya tlhotlhololo kgotsa ditlwaelo , se tshwanelwa , go ya ka thulaganyetso ya Molao wa Palamente go busediwa dithoto tseo , kgotsa tshiamisetso e e e kalekanang .\n", "2019-11-27 16:00:01,559 \tHypothesis: ( 7 ) Motho kgotsa baagi ba ba nang le thoto morago ga 19 Seetebosigo 1913 ka ntlha ya melao ya kgethololo e e fetileng kgotsa mekgwatiriso e e nang le tlhaolele , go ya ka Molao wa Palamente , e ka nna go busediwa ga yona kgotsa go siamisa tshobokanyo ya ntlo eo e e lekaneng .\n", "2019-11-27 16:00:01,559 Example #2\n", "2019-11-27 16:00:01,560 \tSource: ruling\n", "2019-11-27 16:00:01,560 \tReference: tshwetsotaolo\n", "2019-11-27 16:00:01,560 \tHypothesis: tshutiso\n", "2019-11-27 16:00:01,560 Example #3\n", "2019-11-27 16:00:01,560 \tSource: Skills Development course .\n", "2019-11-27 16:00:01,560 \tReference: Khoso ya Tlhabololo ya Bokgoni .\n", "2019-11-27 16:00:01,560 \tHypothesis: Khoso ya Tlhabololo ya Bokgoni .\n", "2019-11-27 16:00:01,561 Validation result (greedy) at epoch 54, step 70000: bleu: 29.45, loss: 38476.7227, ppl: 3.9553, duration: 102.4451s\n", "2019-11-27 16:00:33,510 Epoch 54 Step: 70100 Batch Loss: 1.397223 Tokens per Sec: 7799, Lr: 0.000300\n", "2019-11-27 16:01:05,165 Epoch 54 Step: 70200 Batch Loss: 1.252394 Tokens per Sec: 7542, Lr: 0.000300\n", "2019-11-27 16:01:36,826 Epoch 54 Step: 70300 Batch Loss: 1.101417 Tokens per Sec: 7643, Lr: 0.000300\n", "2019-11-27 16:02:08,278 Epoch 54 Step: 70400 Batch Loss: 1.360441 Tokens per Sec: 7482, Lr: 0.000300\n", "2019-11-27 16:02:40,271 Epoch 54 Step: 70500 Batch Loss: 1.351215 Tokens per Sec: 7699, Lr: 0.000300\n", "2019-11-27 16:03:12,074 Epoch 54 Step: 70600 Batch Loss: 1.525992 Tokens per Sec: 7536, Lr: 0.000300\n", "2019-11-27 16:03:43,897 Epoch 54 Step: 70700 Batch Loss: 1.468056 Tokens per Sec: 7621, Lr: 0.000300\n", "2019-11-27 16:04:15,410 Epoch 54 Step: 70800 Batch Loss: 1.198517 Tokens per Sec: 7433, Lr: 0.000300\n", "2019-11-27 16:04:47,682 Epoch 54 Step: 70900 Batch Loss: 1.466859 Tokens per Sec: 7818, Lr: 0.000300\n", "2019-11-27 16:05:19,588 Epoch 54 Step: 71000 Batch Loss: 1.221587 Tokens per Sec: 7585, Lr: 0.000300\n", "2019-11-27 16:05:51,760 Epoch 54 Step: 71100 Batch Loss: 1.351900 Tokens per Sec: 7616, Lr: 0.000300\n", "2019-11-27 16:06:23,588 Epoch 54 Step: 71200 Batch Loss: 1.424720 Tokens per Sec: 7479, Lr: 0.000300\n", "2019-11-27 16:06:47,265 Epoch 54: total training loss 1717.19\n", "2019-11-27 16:06:47,265 EPOCH 55\n", "2019-11-27 16:06:55,770 Epoch 55 Step: 71300 Batch Loss: 1.293056 Tokens per Sec: 7753, Lr: 0.000300\n", "2019-11-27 16:07:27,192 Epoch 55 Step: 71400 Batch Loss: 1.138071 Tokens per Sec: 7460, Lr: 0.000300\n", "2019-11-27 16:07:59,095 Epoch 55 Step: 71500 Batch Loss: 1.302631 Tokens per Sec: 7515, Lr: 0.000300\n", "2019-11-27 16:08:31,121 Epoch 55 Step: 71600 Batch Loss: 1.385544 Tokens per Sec: 7598, Lr: 0.000300\n", "2019-11-27 16:09:02,647 Epoch 55 Step: 71700 Batch Loss: 1.266120 Tokens per Sec: 7430, Lr: 0.000300\n", "2019-11-27 16:09:34,897 Epoch 55 Step: 71800 Batch Loss: 1.434524 Tokens per Sec: 7554, Lr: 0.000300\n", "2019-11-27 16:10:06,539 Epoch 55 Step: 71900 Batch Loss: 1.319663 Tokens per Sec: 7625, Lr: 0.000300\n", "2019-11-27 16:10:38,424 Epoch 55 Step: 72000 Batch Loss: 1.328212 Tokens per Sec: 7676, Lr: 0.000300\n", "2019-11-27 16:12:20,749 Hooray! New best validation result [ppl]!\n", "2019-11-27 16:12:20,749 Saving new checkpoint.\n", "2019-11-27 16:12:21,042 Example #0\n", "2019-11-27 16:12:21,043 \tSource: The services are free in South Africa\n", "2019-11-27 16:12:21,043 \tReference: Mo Aforikaborwa , ditirelo di neelwa kwa ntle ga go duelelwa .\n", "2019-11-27 16:12:21,043 \tHypothesis: Ditirelo di gololesegile mo Aforikaborwa\n", "2019-11-27 16:12:21,043 Example #1\n", "2019-11-27 16:12:21,043 \tSource: ( 7 ) A person or community dispossessed of property after 19 June 1913 as a result of past racially discriminatory laws or practices is entitled , to the extent provided by an Act of Parliament , either to restitution of that property or to equitable redress .\n", "2019-11-27 16:12:21,043 \tReference: ( 7 ) Motho kgotsa setlhopha sa magae se se amogilweng dithoto morago ga la 19 Seetebosigo 1913 ka ntlha ya melao ya kgale ya tlhotlhololo kgotsa ditlwaelo , se tshwanelwa , go ya ka thulaganyetso ya Molao wa Palamente go busediwa dithoto tseo , kgotsa tshiamisetso e e e kalekanang .\n", "2019-11-27 16:12:21,044 \tHypothesis: ( 7 ) Motho kgotsa setšhaba se se nang le thoto morago ga 19 Seetebosigo 1913 ka ntlha ya melao ya kgethololo kgotsa mekgwatiriso e e kgetholololololwang , go fitlha ka Molao wa Palamente , e ka nna go busediwa ga thoto eo kgotsa go siamisa go siamisa ga gagwe ka tekatekano .\n", "2019-11-27 16:12:21,044 Example #2\n", "2019-11-27 16:12:21,044 \tSource: ruling\n", "2019-11-27 16:12:21,044 \tReference: tshwetsotaolo\n", "2019-11-27 16:12:21,044 \tHypothesis: tshutiso\n", "2019-11-27 16:12:21,044 Example #3\n", "2019-11-27 16:12:21,045 \tSource: Skills Development course .\n", "2019-11-27 16:12:21,045 \tReference: Khoso ya Tlhabololo ya Bokgoni .\n", "2019-11-27 16:12:21,045 \tHypothesis: Khoso ya Tlhabololo ya Bokgoni .\n", "2019-11-27 16:12:21,045 Validation result (greedy) at epoch 55, step 72000: bleu: 29.53, loss: 38025.3594, ppl: 3.8920, duration: 102.6204s\n", "2019-11-27 16:12:52,939 Epoch 55 Step: 72100 Batch Loss: 1.509905 Tokens per Sec: 7646, Lr: 0.000300\n", "2019-11-27 16:13:24,581 Epoch 55 Step: 72200 Batch Loss: 1.067100 Tokens per Sec: 7610, Lr: 0.000300\n", "2019-11-27 16:13:56,190 Epoch 55 Step: 72300 Batch Loss: 1.468176 Tokens per Sec: 7446, Lr: 0.000300\n", "2019-11-27 16:14:28,258 Epoch 55 Step: 72400 Batch Loss: 1.398689 Tokens per Sec: 7675, Lr: 0.000300\n", "2019-11-27 16:15:00,146 Epoch 55 Step: 72500 Batch Loss: 1.143301 Tokens per Sec: 7531, Lr: 0.000300\n", "2019-11-27 16:15:29,125 Epoch 55: total training loss 1721.52\n", "2019-11-27 16:15:29,125 EPOCH 56\n", "2019-11-27 16:15:32,529 Epoch 56 Step: 72600 Batch Loss: 1.244477 Tokens per Sec: 7242, Lr: 0.000300\n", "2019-11-27 16:16:04,374 Epoch 56 Step: 72700 Batch Loss: 1.207769 Tokens per Sec: 7599, Lr: 0.000300\n", "2019-11-27 16:16:36,343 Epoch 56 Step: 72800 Batch Loss: 1.066515 Tokens per Sec: 7668, Lr: 0.000300\n", "2019-11-27 16:17:08,422 Epoch 56 Step: 72900 Batch Loss: 1.414924 Tokens per Sec: 7534, Lr: 0.000300\n", "2019-11-27 16:17:40,210 Epoch 56 Step: 73000 Batch Loss: 1.281375 Tokens per Sec: 7502, Lr: 0.000300\n", "2019-11-27 16:18:11,871 Epoch 56 Step: 73100 Batch Loss: 1.378401 Tokens per Sec: 7680, Lr: 0.000300\n", "2019-11-27 16:18:43,885 Epoch 56 Step: 73200 Batch Loss: 1.399378 Tokens per Sec: 7556, Lr: 0.000300\n", "2019-11-27 16:19:16,103 Epoch 56 Step: 73300 Batch Loss: 1.392573 Tokens per Sec: 7572, Lr: 0.000300\n", "2019-11-27 16:19:47,814 Epoch 56 Step: 73400 Batch Loss: 1.227925 Tokens per Sec: 7397, Lr: 0.000300\n", "2019-11-27 16:20:19,832 Epoch 56 Step: 73500 Batch Loss: 1.476291 Tokens per Sec: 7445, Lr: 0.000300\n", "2019-11-27 16:20:51,702 Epoch 56 Step: 73600 Batch Loss: 1.267695 Tokens per Sec: 7422, Lr: 0.000300\n", "2019-11-27 16:21:23,660 Epoch 56 Step: 73700 Batch Loss: 1.381181 Tokens per Sec: 7538, Lr: 0.000300\n", "2019-11-27 16:21:55,890 Epoch 56 Step: 73800 Batch Loss: 1.348701 Tokens per Sec: 7477, Lr: 0.000300\n", "2019-11-27 16:22:27,833 Epoch 56 Step: 73900 Batch Loss: 1.477889 Tokens per Sec: 7442, Lr: 0.000300\n", "2019-11-27 16:22:31,129 Epoch 56: total training loss 1718.55\n", "2019-11-27 16:22:31,130 EPOCH 57\n", "Traceback (most recent call last):\n", " File \"/usr/lib/python3.6/runpy.py\", line 193, in _run_module_as_main\n", " \"__main__\", mod_spec)\n", " File \"/usr/lib/python3.6/runpy.py\", line 85, in _run_code\n", " exec(code, run_globals)\n", " File \"/content/joeynmt/joeynmt/__main__.py\", line 41, in \n", " main()\n", " File \"/content/joeynmt/joeynmt/__main__.py\", line 29, in main\n", " train(cfg_file=args.config_path)\n", " File \"/content/joeynmt/joeynmt/training.py\", line 596, in train\n", " trainer.train_and_validate(train_data=train_data, valid_data=dev_data)\n", " File \"/content/joeynmt/joeynmt/training.py\", line 296, in train_and_validate\n", " batch_loss = self._train_batch(batch, update=update)\n", " File \"/content/joeynmt/joeynmt/training.py\", line 436, in _train_batch\n", " batch=batch, loss_function=self.loss)\n", " File \"/content/joeynmt/joeynmt/model.py\", line 133, in get_loss_for_batch\n", " trg_mask=batch.trg_mask)\n", " File \"/content/joeynmt/joeynmt/model.py\", line 80, in forward\n", " trg_mask=trg_mask)\n", " File \"/content/joeynmt/joeynmt/model.py\", line 117, in decode\n", " trg_mask=trg_mask)\n", " File \"/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py\", line 541, in __call__\n", " result = self.forward(*input, **kwargs)\n", " File \"/content/joeynmt/joeynmt/decoders.py\", line 518, in forward\n", " src_mask=src_mask, trg_mask=trg_mask)\n", " File \"/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py\", line 541, in __call__\n", " result = self.forward(*input, **kwargs)\n", " File \"/content/joeynmt/joeynmt/transformer_layers.py\", line 267, in forward\n", " h2 = self.src_trg_att(memory, memory, h1_norm, mask=src_mask)\n", " File \"/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py\", line 541, in __call__\n", " result = self.forward(*input, **kwargs)\n", " File \"/content/joeynmt/joeynmt/transformer_layers.py\", line 76, in forward\n", " attention = self.softmax(scores)\n", " File \"/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py\", line 541, in __call__\n", " result = self.forward(*input, **kwargs)\n", " File \"/usr/local/lib/python3.6/dist-packages/torch/nn/modules/activation.py\", line 982, in forward\n", " return F.softmax(input, self.dim, _stacklevel=5)\n", " File \"/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py\", line 1231, in softmax\n", " ret = input.softmax(dim)\n", "KeyboardInterrupt\n" ], "name": "stdout" } ] }, { "cell_type": "code", "metadata": { "colab_type": "code", "id": "MBoDS09JM807", "outputId": "6dfd74e0-78f4-4ed9-cbee-9dbb288571df", "colab": { "base_uri": "https://localhost:8080/", "height": 54 } }, "source": [ "# Copy the created models from the notebook storage to google drive for persistant storage \n", "!cp -r joeynmt/models/${src}${tgt}_transformer/* \"/content/drive/My Drive/masakhane/en-tn-baseline/models/entn_transformer\"" ], "execution_count": 18, "outputs": [ { "output_type": "stream", "text": [ "cp: cannot create symbolic link '/content/drive/My Drive/masakhane/en-tn-baseline/models/entn_transformer/best.ckpt': Function not implemented\n" ], "name": "stdout" } ] }, { "cell_type": "code", "metadata": { "colab_type": "code", "id": "n94wlrCjVc17", "outputId": "51120724-a83e-483e-ba1e-80f5d71177cd", "colab": { "base_uri": "https://localhost:8080/", "height": 622 } }, "source": [ "# Output our validation accuracy\n", "! cat \"$gdrive_path/models/${src}${tgt}_transformer/validations.txt\"" ], "execution_count": 19, "outputs": [ { "output_type": "stream", "text": [ "Steps: 2000\tLoss: 93091.50781\tPPL: 27.85007\tbleu: 1.28343\tLR: 0.00030000\t*\n", "Steps: 4000\tLoss: 77095.53125\tPPL: 15.72392\tbleu: 5.86352\tLR: 0.00030000\t*\n", "Steps: 6000\tLoss: 68491.06250\tPPL: 11.56153\tbleu: 9.37769\tLR: 0.00030000\t*\n", "Steps: 8000\tLoss: 62278.05078\tPPL: 9.25947\tbleu: 12.98628\tLR: 0.00030000\t*\n", "Steps: 10000\tLoss: 57997.64453\tPPL: 7.94606\tbleu: 15.44574\tLR: 0.00030000\t*\n", "Steps: 12000\tLoss: 54562.86719\tPPL: 7.02818\tbleu: 18.12086\tLR: 0.00030000\t*\n", "Steps: 14000\tLoss: 51923.93359\tPPL: 6.39566\tbleu: 19.94620\tLR: 0.00030000\t*\n", "Steps: 16000\tLoss: 50232.12109\tPPL: 6.02043\tbleu: 20.93697\tLR: 0.00030000\t*\n", "Steps: 18000\tLoss: 48756.14453\tPPL: 5.71110\tbleu: 22.27166\tLR: 0.00030000\t*\n", "Steps: 20000\tLoss: 47692.87109\tPPL: 5.49815\tbleu: 22.94679\tLR: 0.00030000\t*\n", "Steps: 22000\tLoss: 46253.86719\tPPL: 5.22255\tbleu: 23.78359\tLR: 0.00030000\t*\n", "Steps: 24000\tLoss: 45594.04297\tPPL: 5.10084\tbleu: 24.47695\tLR: 0.00030000\t*\n", "Steps: 26000\tLoss: 44392.13672\tPPL: 4.88639\tbleu: 25.22885\tLR: 0.00030000\t*\n", "Steps: 28000\tLoss: 44094.18359\tPPL: 4.83463\tbleu: 25.46200\tLR: 0.00030000\t*\n", "Steps: 30000\tLoss: 43571.17578\tPPL: 4.74511\tbleu: 25.58564\tLR: 0.00030000\t*\n", "Steps: 32000\tLoss: 42917.80078\tPPL: 4.63560\tbleu: 26.52136\tLR: 0.00030000\t*\n", "Steps: 34000\tLoss: 42369.94922\tPPL: 4.54572\tbleu: 26.15194\tLR: 0.00030000\t*\n", "Steps: 36000\tLoss: 42141.73438\tPPL: 4.50880\tbleu: 26.98753\tLR: 0.00030000\t*\n", "Steps: 38000\tLoss: 41620.48438\tPPL: 4.42558\tbleu: 26.72346\tLR: 0.00030000\t*\n", "Steps: 40000\tLoss: 41505.07812\tPPL: 4.40737\tbleu: 27.23369\tLR: 0.00030000\t*\n", "Steps: 42000\tLoss: 41047.19531\tPPL: 4.33584\tbleu: 27.17121\tLR: 0.00030000\t*\n", "Steps: 44000\tLoss: 40879.41797\tPPL: 4.30992\tbleu: 27.85087\tLR: 0.00030000\t*\n", "Steps: 46000\tLoss: 40705.35156\tPPL: 4.28319\tbleu: 27.94499\tLR: 0.00030000\t*\n", "Steps: 48000\tLoss: 40279.37500\tPPL: 4.21848\tbleu: 27.78448\tLR: 0.00030000\t*\n", "Steps: 50000\tLoss: 39917.39062\tPPL: 4.16426\tbleu: 28.53349\tLR: 0.00030000\t*\n", "Steps: 52000\tLoss: 39685.95703\tPPL: 4.12996\tbleu: 28.70320\tLR: 0.00030000\t*\n", "Steps: 54000\tLoss: 39321.52344\tPPL: 4.07652\tbleu: 28.64323\tLR: 0.00030000\t*\n", "Steps: 56000\tLoss: 39422.42188\tPPL: 4.09125\tbleu: 28.70637\tLR: 0.00030000\t\n", "Steps: 58000\tLoss: 39122.05469\tPPL: 4.04756\tbleu: 29.12577\tLR: 0.00030000\t*\n", "Steps: 60000\tLoss: 38971.56641\tPPL: 4.02585\tbleu: 28.95924\tLR: 0.00030000\t*\n", "Steps: 62000\tLoss: 38802.10547\tPPL: 4.00155\tbleu: 29.26524\tLR: 0.00030000\t*\n", "Steps: 64000\tLoss: 38765.76562\tPPL: 3.99635\tbleu: 29.13887\tLR: 0.00030000\t*\n", "Steps: 66000\tLoss: 38558.03906\tPPL: 3.96680\tbleu: 28.72364\tLR: 0.00030000\t*\n", "Steps: 68000\tLoss: 38503.20312\tPPL: 3.95903\tbleu: 28.77827\tLR: 0.00030000\t*\n", "Steps: 70000\tLoss: 38476.72266\tPPL: 3.95528\tbleu: 29.45178\tLR: 0.00030000\t*\n", "Steps: 72000\tLoss: 38025.35938\tPPL: 3.89200\tbleu: 29.52991\tLR: 0.00030000\t*\n" ], "name": "stdout" } ] }, { "cell_type": "code", "metadata": { "colab_type": "code", "id": "66WhRE9lIhoD", "outputId": "ec518141-6065-4438-eb38-fff924e37c13", "colab": { "base_uri": "https://localhost:8080/", "height": 67 } }, "source": [ "# Test our model\n", "! cd joeynmt; python3 -m joeynmt test \"$gdrive_path/models/${src}${tgt}_transformer/config.yaml\"" ], "execution_count": 20, "outputs": [ { "output_type": "stream", "text": [ "2019-11-27 16:25:27,547 Hello! This is Joey-NMT.\n", "2019-11-27 16:26:40,836 dev bleu: 29.78 [Beam search decoding with beam size = 5 and alpha = 1.0]\n", "2019-11-27 16:29:09,465 test bleu: 19.66 [Beam search decoding with beam size = 5 and alpha = 1.0]\n" ], "name": "stdout" } ] }, { "cell_type": "code", "metadata": { "id": "k9QbsG8rCJBX", "colab_type": "code", "colab": {} }, "source": [ "" ], "execution_count": 0, "outputs": [] } ] }