Repository: pytorch/text

Issue:
MosesTokenizer has been moved out of NLTK due to licensing issues
@jekbradbury great work here!

Due to https://github.com/nltk/nltk/issues/2000, we had to remove MosesTokenizer out of NLTK but now it's hosted on https://github.com/alvations/sacremoses 

```
pip install sacremoses
```

The silver lining is that the package comes with the data needed for tokenization so there's no need to keep the `nltk_data` directory =)

----

I would propose adding `sacremoses` on top of `nltk` because NLTK has another port of a nice tokenizer (by @jonsafari) that people overlook, https://github.com/nltk/nltk/blob/develop/nltk/tokenize/toktok.py (I think it's fast too)


Discussion:
I'm not sure which branch is the dev branch to add tokenizers, so I'll rely on the github search:

`text/torchtext/data/utils.py`

```python

    elif tokenizer == "moses":
        try:
            from sacremoses import MosesTokenizer
            moses_tokenizer = MosesTokenizer()
            return moses_tokenizer.tokenize
        except ImportError:
            print("Please install SacreMoses. "
                  "See the docs at https://github.com/alvations/sacremoses for more information.")
            raise

    elif tokenizer == "toktok":
        try:
            from nltk.tokenize.toktok import ToktokTokenizer
            toktok = ToktokTokenizer()
            return toktok.tokenize
        except ImportError:
            print("Please install NLTK. "
                  "See the docs at https://nltk.org  for more information.")
            raise

```

Hope it helps!
@alvations Thanks for the repo! We're coming up on the same problem: https://github.com/PetrochukM/PyTorch-NLP 

I'll update it to SacreMoses.

Resolving patch:
diff --git a/torchtext/data/utils.py b/torchtext/data/utils.py
--- a/torchtext/data/utils.py
+++ b/torchtext/data/utils.py
@@ -21,16 +21,22 @@ def get_tokenizer(tokenizer):
             raise
     elif tokenizer == "moses":
         try:
-            from nltk.tokenize.moses import MosesTokenizer
+            from sacremoses import MosesTokenizer
             moses_tokenizer = MosesTokenizer()
             return moses_tokenizer.tokenize
         except ImportError:
-            print("Please install NLTK. "
-                  "See the docs at http://nltk.org for more information.")
+            print("Please install SacreMoses. "
+                  "See the docs at https://github.com/alvations/sacremoses "
+                  "for more information.")
             raise
-        except LookupError:
-            print("Please install the necessary NLTK corpora. "
-                  "See the docs at http://nltk.org for more information.")
+    elif tokenizer == "toktok":
+        try:
+            from nltk.tokenize.toktok import ToktokTokenizer
+            toktok = ToktokTokenizer()
+            return toktok.tokenize
+        except ImportError:
+            print("Please install NLTK. "
+                  "See the docs at https://nltk.org  for more information.")
             raise
     elif tokenizer == 'revtok':
         try:


Test patch:
diff --git a/test/data/test_utils.py b/test/data/test_utils.py
--- a/test/data/test_utils.py
+++ b/test/data/test_utils.py
@@ -16,7 +16,7 @@ def test_get_tokenizer(self):
             "A", "string", ",", "particularly", "one", "with", "slightly",
             "complex", "punctuation", "."]
 
-        # Test Moses option. Test strings taken from NLTK doctests.
+        # Test Moses option.
         # Note that internally, MosesTokenizer converts to unicode if applicable
         moses_tokenizer = data.get_tokenizer("moses")
         assert moses_tokenizer(test_str) == [
@@ -26,6 +26,13 @@ def test_get_tokenizer(self):
         # Nonbreaking prefixes should tokenize the final period.
         assert moses_tokenizer(six.text_type("abc def.")) == ["abc", "def", "."]
 
+        # Test Toktok option. Test strings taken from NLTK doctests.
+        # Note that internally, MosesTokenizer converts to unicode if applicable
+        toktok_tokenizer = data.get_tokenizer("toktok")
+        assert toktok_tokenizer(test_str) == [
+            "A", "string", ",", "particularly", "one", "with", "slightly",
+            "complex", "punctuation", "."]
+
         # Test that errors are raised for invalid input arguments.
         with self.assertRaises(ValueError):
             data.get_tokenizer(1)