Repository: OpenNMT/OpenNMT-tf (python)

Issue:
Subword tokenisation spacer can mark the beginning of word
Certain sequence noising operations need to retrieve a list of words from the raw list of subword tokens. For example:

* Decoding with word removal/reordering to produce noisy back-translations as in [Scaling BT paper](https://arxiv.org/abs/1808.09381)

* Word omission to support the new contrastive learning feature as in the [contrastive learning paper](https://www.aclweb.org/anthology/P19-1623.pdf)

* Presumably more features relying on word level noise might come up in the future

In these cases the user should specify some details for the sub-tokenisation process: 
1. What subword tokens was used? (`decoding_subword_token`)
2. Was that token a joiner or a spacer? (`decoding_subword_token_is_spacer`)

When the user specifies (explicitly or implicitly) a spacer, the framework assumes that the spacer symbol appears at the beginning of each word, similar to what SentencePiece does. However this does not have to be the case, the spacer could also appear at the end of each word - for example [this one does](https://github.com/kovalevfm/SubTokenizer). If that extra sub-tokenisation flexibility is desired, we can add this configuration parameter. A sample implementation could look like [this](https://github.com/steremma/OpenNMT-tf/commit/d109af49911431e424b28def575fb94f07bfec47).

I realise that most user's rely on standard tools that are covered by the current implementation. If there is a user base for which the extra flexibility is desired, I can submit a PR that reads this option from the YAML.

Pull request:
Support arbitrary spacer placement in tokens_to_words
Closes #514.

Patch:
diff --git a/opennmt/data/text.py b/opennmt/data/text.py
index 91b1012a..42a80bab 100644
--- a/opennmt/data/text.py
+++ b/opennmt/data/text.py
@@ -40,13 +40,18 @@ def tokens_to_words(tokens, subword_token="￭", is_spacer=None):
   if is_spacer is None:
     is_spacer = subword_token == "▁"
   if is_spacer:
-    subword = tf.strings.regex_full_match(tokens, "[^%s].*" % subword_token)
+    # First token implicitly starts with a spacer.
+    left_and_single = tf.logical_or(
+        tf.strings.regex_full_match(tokens, "%s.*" % subword_token),
+        tf.one_hot(0, tf.shape(tokens)[0], on_value=True, off_value=False))
+    right = tf.strings.regex_full_match(tokens, ".+%s" % subword_token)
+    word_start = tf.logical_or(tf.roll(right, shift=1, axis=0), left_and_single)
   else:
     right = tf.strings.regex_full_match(tokens, ".*%s" % subword_token)
     left = tf.strings.regex_full_match(tokens, "%s.*" % subword_token)
     subword = tf.logical_or(tf.roll(right, shift=1, axis=0), left)
-  start = tf.logical_not(subword)
-  start_indices = tf.squeeze(tf.where(start), -1)
+    word_start = tf.logical_not(subword)
+  start_indices = tf.squeeze(tf.where(word_start), -1)
   return tf.RaggedTensor.from_row_starts(tokens, start_indices)
 
 def alignment_matrix_from_pharaoh(alignment_line,


Test patch:
diff --git a/opennmt/tests/text_test.py b/opennmt/tests/text_test.py
index 5d943906..4d403e35 100644
--- a/opennmt/tests/text_test.py
+++ b/opennmt/tests/text_test.py
@@ -44,6 +44,8 @@ class TextTest(tf.test.TestCase):
   @parameterized.expand([
     [["▁a", "b", "▁c", "d", "e"], [["▁a", "b", ""], ["▁c", "d", "e"]]],
     [["▁", "a", "b", "▁", "c", "d", "e"], [["▁", "a", "b", ""], ["▁", "c", "d", "e"]]],
+    [["a▁", "b", "c▁", "d", "e"], [["a▁", ""], ["b", "c▁"], ["d", "e"]]],
+    [["a", "▁b▁", "c", "d", "▁", "e"], [["a", ""], ["▁b▁", ""], ["c", "d"], ["▁", "e"]]],
   ])
   def testToWordsWithSpacer(self, tokens, expected):
     tokens = tf.constant(tokens)