Repository: dmlc/gluon-nlp

Issue:
Transformer BLEU of the current code base is worse than previous one
I recently retrained the transformer, and found that it will diverge with the default setting. The divergence issue is caused by use_layer_norm_before_dropout=True and is fixed in #543. However, the final testing BLEU is still worse than the previous result. Compare to the version in #413, the testing BLEU is about 0.8 worse.


Discussion:
@eric-haibin-lin 
Any idea where the difference is coming from?
I checked the part of code of model and didn't find error except the one mentioned in #543. Comparing the logs, the outputs of bucket sampler are different. But I haven't got time looking into it. This may be caused by the change to the bucket sampler or the data processing. I paste the output of sampler as follows:

Current code base:
FixedBucketSampler:
  sample_num=4493328, batch_num=59321
  key=[(4, 5), (7, 8), (11, 13), (16, 18), (22, 24), (29, 31), (38, 40), (48, 50), (60, 63), (74, 78), (92, 96), (113, 118), (138, 144), (168, 175), (205, 213), (248, 258), (301, 312), (363, 377), (439, 
455), (529, 548)]
  cnt=[15338, 57067, 248232, 511919, 747010, 820181, 816207, 571262, 380721, 193794, 90070, 32452, 6865, 1445, 517, 171, 57, 11, 7, 2]
  batch_size=[629, 362, 233, 166, 123, 94, 72, 57, 45, 36, 29, 24, 19, 16, 13, 10, 8, 7, 6, 4]
2019-01-05 16:04:49,638 - root - Valid Batch Sampler:
FixedBucketSampler:
  sample_num=3000, batch_num=376
  key=[3, 4, 5, 6, 7, 9, 11, 14, 17, 20, 25, 30, 36, 44, 53, 64, 77, 93, 111, 134]
  cnt=[13, 12, 14, 24, 53, 110, 141, 254, 253, 275, 411, 377, 321, 328, 207, 109, 61, 24, 7, 6]
  batch_size=[80, 64, 51, 42, 36, 28, 23, 18, 15, 12, 10, 8, 7, 6, 5, 4, 3, 2, 2, 1]
2019-01-05 16:04:49,647 - root - Test Batch Sampler:
FixedBucketSampler:
  sample_num=3003, batch_num=394
  key=[6, 7, 8, 9, 10, 12, 13, 15, 18, 21, 25, 29, 35, 41, 49, 58, 70, 83, 99]
  cnt=[16, 20, 35, 43, 46, 141, 86, 167, 261, 257, 355, 308, 422, 306, 223, 192, 93, 27, 5]
  batch_size=[41, 36, 32, 28, 25, 21, 19, 17, 14, 12, 10, 8, 7, 6, 5, 4, 3, 3, 2]

Previous one:
FixedBucketSampler:
  sample_num=4493328, batch_num=61430
  key=[(5, 6), (8, 9), (12, 14), (17, 19), (23, 25), (30, 32), (39, 41), (49, 51), (61, 64), (75, 79), (93, 97), (114, 119), (139, 145), (169, 176), (206, 214), (249, 259), (302, 313), (364, 378), (440, 456), (530, 549)]
  cnt=[15338, 57067, 248232, 511919, 747010, 820181, 816207, 571262, 380721, 193794, 90070, 32452, 6865, 1445, 517, 171, 57, 11, 7, 2]
  batch_size=[510, 319, 214, 156, 117, 90, 70, 55, 44, 35, 29, 23, 19, 16, 13, 10, 8, 7, 6, 4]
2019-01-09 14:13:53,198 - root - Valid Batch Sampler:
FixedBucketSampler:
  sample_num=3000, batch_num=400
  key=[4, 5, 6, 7, 8, 10, 12, 15, 18, 21, 26, 31, 37, 45, 54, 65, 78, 94, 112, 135]
  cnt=[13, 12, 14, 24, 53, 110, 141, 254, 253, 275, 411, 377, 321, 328, 207, 109, 61, 24, 7, 6]
  batch_size=[61, 51, 42, 36, 32, 25, 21, 17, 14, 12, 10, 8, 7, 5, 4, 4, 3, 2, 2, 1]
2019-01-09 14:13:53,204 - root - Test Batch Sampler:
FixedBucketSampler:
  sample_num=3003, batch_num=400
  key=[7, 8, 9, 10, 11, 13, 14, 16, 19, 22, 26, 30, 36, 42, 50, 59, 71, 84, 100]
  cnt=[16, 20, 35, 43, 46, 141, 86, 167, 261, 257, 355, 308, 422, 306, 223, 192, 93, 27, 5]
  batch_size=[36, 32, 28, 25, 23, 19, 18, 16, 13, 11, 10, 8, 7, 6, 5, 4, 3, 3, 2]
Does it mean there's also some changes in the data processing pipeline?
Comparing the keys, it seems each sequence lose one token.

Resolving patch:
diff --git a/scripts/machine_translation/dataprocessor.py b/scripts/machine_translation/dataprocessor.py
--- a/scripts/machine_translation/dataprocessor.py
+++ b/scripts/machine_translation/dataprocessor.py
@@ -86,11 +86,13 @@ def __init__(self, src_vocab, tgt_vocab, src_max_len=None, tgt_max_len=None):
         self._tgt_max_len = tgt_max_len
 
     def __call__(self, src, tgt):
-        if self._src_max_len:
+        # For src_max_len < 0, we do not clip the sequence
+        if self._src_max_len >= 0:
             src_sentence = self._src_vocab[src.split()[:self._src_max_len]]
         else:
             src_sentence = self._src_vocab[src.split()]
-        if self._tgt_max_len:
+        # For tgt_max_len < 0, we do not clip the sequence
+        if self._tgt_max_len >= 0:
             tgt_sentence = self._tgt_vocab[tgt.split()[:self._tgt_max_len]]
         else:
             tgt_sentence = self._tgt_vocab[tgt.split()]
@@ -162,10 +164,14 @@ def load_translation_data(dataset, bleu, args):
     if not data_val_processed:
         data_val_processed = process_dataset(data_val, src_vocab, tgt_vocab)
         _cache_dataset(data_val_processed, common_prefix + '_val')
-    data_test_processed = _load_cached_dataset(common_prefix + '_' + str(False) + '_test')
+    if dataset == 'WMT2014BPE':
+        filename = common_prefix + '_' + str(args.full) + '_test'
+    else:
+        filename = common_prefix + '_test'
+    data_test_processed = _load_cached_dataset(filename)
     if not data_test_processed:
         data_test_processed = process_dataset(data_test, src_vocab, tgt_vocab)
-        _cache_dataset(data_test_processed, common_prefix + '_' + str(False) + '_test')
+        _cache_dataset(data_test_processed, filename)
     if bleu == 'tweaked':
         fetch_tgt_sentence = lambda src, tgt: tgt.split()
         val_tgt_sentences = list(data_val.transform(fetch_tgt_sentence))


Test patch:
diff --git a/scripts/tests/test_dataprocessor.py b/scripts/tests/test_dataprocessor.py
new file mode 100644
--- /dev/null
+++ b/scripts/tests/test_dataprocessor.py
@@ -0,0 +1,52 @@
+# coding: utf-8
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Test DataProcessor."""
+
+from __future__ import print_function
+
+import sys
+import os
+
+sys.path.append(os.path.join(os.path.dirname(os.path.dirname(__file__)), 'machine_translation'))
+
+from ..machine_translation.dataprocessor import process_dataset
+from ..machine_translation.dataset import TOY
+
+
+def test_translation_preprocess():
+    src_lang = 'en'
+    tgt_lang = 'de'
+    max_lens = ((10, 10), (0, 0), (-1, -1))
+    for (src_max_len, tgt_max_len) in max_lens:
+        data_train = TOY('train', src_lang=src_lang, tgt_lang=tgt_lang)
+        data_val = TOY('val', src_lang=src_lang, tgt_lang=tgt_lang)
+        src_vocab, tgt_vocab = data_train.src_vocab, data_train.tgt_vocab
+        data_val_processed = process_dataset(data_val, src_vocab, tgt_vocab,
+                                             src_max_len, tgt_max_len)
+        for (src, tgt), (preprocessed_src, preprocessed_tgt) in zip(data_val, data_val_processed):
+            if src_max_len >= 0:
+                assert len(preprocessed_src) == min(len(src.split()), src_max_len) + 1
+            else:
+                assert len(preprocessed_src) == len(src.split()) + 1
+            if tgt_max_len >= 0:
+                assert len(preprocessed_tgt) == min(len(tgt.split()), tgt_max_len) + 2
+            else:
+                assert len(preprocessed_tgt) == len(tgt.split()) + 2
+