Repository: pytorch/text

Issue:
Include Moses Tokenizer



Discussion:
I know NLTK has a Moses tokenizer, but unsure if it's a good port / I've always used tokenize.pl anyway

For the reference, for large corpora (e.g MT) I've found it  faster to pretokenize your data and feed it into TorchText with the str.split tokenizer, versus having to tokenize on every run.
@nelson-liu Is there an option to include a bash script to tokenize your data as part of the library?

This issue mentions that Moses in NLTK was fixed https://github.com/nltk/nltk/issues/1214
Hmm, not directly. Ostensibly you could write a function that takes an input string and runs the bash script on it (e.g with subprocess) and parse the output for tokenization, but that feels like a lot of overhead
Edited my comment above. Looks like NLTK contributors fixed Moses in this merged pull request: https://github.com/nltk/nltk/pull/1553
I think including the NLTK version of the Moses tokenizer is a good idea, though it shouldn’t be difficult to use the existing API to call it for now
+1, slighty unrelated  but wonder if it's worth including the other NLTK tokenizers (e.g punkt or PTB) / having some sort of public facing API for using the spacy tokenizers in other languages. This seems like a slippery slope API design-wise, though...
Yeah, I think I’d rather show in the docs etc how easy it is to call them yourself. But lots of people (including me until a few minutes ago!) don’t know that NLTK now has a Moses-compatible tokenizer, so it might be worth including that one so more people know they can move away from the perl script preprocessing approach.
Fair enough, I can open a PR for this in a bit / add a custom tokenizer
example.

On Sun, Jul 9, 2017 at 1:30 PM jekbradbury <notifications@github.com> wrote:

> Yeah, I think I’d rather show in the docs etc how easy it is to call them
> yourself. But lots of people (including me until a few minutes ago!) don’t
> know that NLTK now has a Moses-compatible tokenizer, so it might be worth
> including that one so more people know they can move away from the perl
> script preprocessing approach.
>
> —
> You are receiving this because you were mentioned.
>
>
> Reply to this email directly, view it on GitHub
> <https://github.com/pytorch/text/issues/53#issuecomment-313952582>, or mute
> the thread
> <https://github.com/notifications/unsubscribe-auth/AG72X-odRaXKqFDmR7OocaMGSfxQYN0Nks5sMThygaJpZM4OSKBj>
> .
>


Resolving patch:
diff --git a/torchtext/data/example.py b/torchtext/data/example.py
--- a/torchtext/data/example.py
+++ b/torchtext/data/example.py
@@ -49,8 +49,8 @@ def fromtree(cls, data, fields, subtrees=False):
         try:
             from nltk.tree import Tree
         except ImportError:
-            print('''Please install NLTK:
-    $ pip install nltk''')
+            print("Please install NLTK. "
+                  "See the docs at http://nltk.org for more information.")
             raise
         tree = Tree.fromstring(data)
         if subtrees:
diff --git a/torchtext/data/utils.py b/torchtext/data/utils.py
--- a/torchtext/data/utils.py
+++ b/torchtext/data/utils.py
@@ -1,7 +1,7 @@
 def get_tokenizer(tokenizer):
     if callable(tokenizer):
         return tokenizer
-    if tokenizer == 'spacy':
+    if tokenizer == "spacy":
         try:
             import spacy
             spacy_en = spacy.load('en')
@@ -14,10 +14,24 @@ def get_tokenizer(tokenizer):
             print("Please install SpaCy and the SpaCy English tokenizer. "
                   "See the docs at https://spacy.io for more information.")
             raise
+    elif tokenizer == "moses":
+        try:
+            from nltk.tokenize.moses import MosesTokenizer
+            moses_tokenizer = MosesTokenizer()
+            return moses_tokenizer.tokenize
+        except ImportError:
+            print("Please install NLTK. "
+                  "See the docs at http://nltk.org for more information.")
+            raise
+        except LookupError:
+            print("Please install the necessary NLTK corpora. "
+                  "See the docs at http://nltk.org for more information.")
+            raise
     raise ValueError("Requested tokenizer {}, valid choices are a "
-                     "callable that takes a single string as input "
-                     "and \"spacy\" for the SpaCy English "
-                     "tokenizer.".format(tokenizer))
+                     "callable that takes a single string as input, "
+                     "\"spacy\" for the SpaCy English tokenizer, or "
+                     "\"moses\" for the NLTK port of the Moses tokenization "
+                     "script.".format(tokenizer))
 
 
 def interleave_keys(a, b):


Test patch:
diff --git a/test/data/test_field.py b/test/data/test_field.py
--- a/test/data/test_field.py
+++ b/test/data/test_field.py
@@ -1,6 +1,5 @@
 from unittest import TestCase
 
-import six
 import torchtext.data as data
 
 
@@ -88,20 +87,3 @@ def test_pad(self):
         field = data.Field(init_token="<bos>", eos_token="<eos>",
                            sequential=False, include_lengths=True)
         assert field.pad(minibatch) == minibatch
-
-    def test_get_tokenizer(self):
-        # Test the default case with str.split
-        assert data.get_tokenizer(str.split) == str.split
-        test_str = "A string, particularly one with slightly complex punctuation."
-        assert data.get_tokenizer(str.split)(test_str) == str.split(test_str)
-
-        # Test SpaCy option, and verify it properly handles punctuation.
-        assert data.get_tokenizer("spacy")(six.text_type(test_str)) == [
-            "A", "string", ",", "particularly", "one", "with", "slightly",
-            "complex", "punctuation", "."]
-
-        # Test that errors are raised for invalid input arguments.
-        with self.assertRaises(ValueError):
-            data.get_tokenizer(1)
-        with self.assertRaises(ValueError):
-            data.get_tokenizer("some other string")
diff --git a/test/data/test_utils.py b/test/data/test_utils.py
new file mode 100644
--- /dev/null
+++ b/test/data/test_utils.py
@@ -0,0 +1,33 @@
+from unittest import TestCase
+
+import six
+import torchtext.data as data
+
+
+class TestUtils(TestCase):
+    def test_get_tokenizer(self):
+        # Test the default case with str.split
+        assert data.get_tokenizer(str.split) == str.split
+        test_str = "A string, particularly one with slightly complex punctuation."
+        assert data.get_tokenizer(str.split)(test_str) == str.split(test_str)
+
+        # Test SpaCy option, and verify it properly handles punctuation.
+        assert data.get_tokenizer("spacy")(six.text_type(test_str)) == [
+            "A", "string", ",", "particularly", "one", "with", "slightly",
+            "complex", "punctuation", "."]
+
+        # Test Moses option. Test strings taken from NLTK doctests.
+        # Note that internally, MosesTokenizer converts to unicode if applicable
+        moses_tokenizer = data.get_tokenizer("moses")
+        assert moses_tokenizer(test_str) == [
+            "A", "string", ",", "particularly", "one", "with", "slightly",
+            "complex", "punctuation", "."]
+
+        # Nonbreaking prefixes should tokenize the final period.
+        assert moses_tokenizer(six.text_type("abc def.")) == ["abc", "def", "."]
+
+        # Test that errors are raised for invalid input arguments.
+        with self.assertRaises(ValueError):
+            data.get_tokenizer(1)
+        with self.assertRaises(ValueError):
+            data.get_tokenizer("some other string")