Repository: huggingface/transformers (Python) — Bug Fix

Issue:
Transformer-XL tokenizer cannot properly tokenize brackets
# 🐛 Bug

## Information
The `TransfoXLTokenizer` is not able to tokenize words with surrounding brackets correctly. I compared it with the `BertTokenizer` from `bert-base-uncased` which gives the expected result. Example text is: `"Hello (bracket)"`

Model I am using: **Transformer-XL**

Language I am using the model on: **English**

The problem arises when using:
* [x] my own modified scripts

## To reproduce

Steps to reproduce the behavior:
```python
from transformers import BertTokenizer, TransfoXLTokenizer

bert = BertTokenizer.from_pretrained('bert-base-uncased')
transfoxl = TransfoXLTokenizer.from_pretrained('transfo-xl-wt103')

def test_bracket(tokenizer):
    enc = tokenizer.encode("Hello (bracket)")
    dec = tokenizer.decode(enc)
    print(f"ORG: Hello (bracket)\nENC: {enc}\nDEC: {dec}")
```
Results:  
`test_bracket(bert)` gives the following output:
```
ORG: Hello (bracket)
ENC: [101, 7592, 1006, 21605, 1007, 102]
DEC: [CLS] hello ( bracket ) [SEP]
```

`test_bracket(transfoxl)` gives the following output:
```
ORG: Hello (bracket)
ENC: [14049, 24]
DEC: Hello <unk>
```
If the parameter `add_space_before_punct_symbol=True` is passed, then the result is:
```
ORG: Hello (bracket)
ENC: [14049, 24, 21]
DEC: Hello <unk> )
```

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior
The `TransfoXLTokenizer` should detect the punctuation symbols, e.g. `(`, separately and thus give the same result as the `BertTokenizer` (except the special tokens of course): `hello ( bracket )`


## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 2.11.0
- Platform: Windows-10-10.0.18362-SP0
- Python version: 3.6.10
- PyTorch version (GPU?): 1.4.0 (False)
- Tensorflow version (GPU?): 2.1.0 (False)



Discussion:
**UPDATE**
I've done some further research and discovered that the tokenization of strings containing either

1. any opening bracket, e.g. `( [ {`
2. words with dashes, e.g. `10-year-old`
3. other symbols with no space afterwards, e.g. (`km/h` or `$3`)
4. numbers, either floating point, e.g. `3.23`, or large comma separated, e.g. `5,000`

result in tokenization errors. See the following example:
Example string: 
```
"Hello (bracket) and side-scrolled [and] Henry's $5,000 km/h with 3.34 m. What's up!?"
```
Encoded and decoded again with `TransfoXLTokenizer`: 
```
Hello <unk> ) and side <unk> <unk> ] <unk> <unk> <unk> km <unk> with 3 <unk> m . <unk> up ! ?
```

In the [Transformer-XL paper](http://arxiv.org/abs/1901.02860) they used the WikiText-103 dataset. The authors of the [WikiText-103 paper](http://arxiv.org/abs/1609.07843) stated that they used the *Moses tokenizer* for tokenization of the wikipedia articles which can deal with the errors stated above (except for 4. but I implemented a custom solution for it - the authors did this too for WikiText-103). This tokenizer replaces dashes with `@-@`, e.g. `10-year-old` gets `10 @-@ year @-@ old`, and dots or commas in number the same way, e.g. `3.5` gets `3 @.@ 5` or `5,000` gets `5 @,@ 000`.

Since the pretrained Transformer-XL model is trained with the tokenization above it would make sense to use the same rules for the `TransfoXLTokenizer`, in my opinion. I have found a python package for the *Moses tokenizer* (see [link](https://github.com/alvations/sacremoses)) but I would understand if you do not prefer using it here.

Otherwise some logic of the `BertTokenizer` could be used to because it does perfectly fine with the string above: 
```
[CLS] hello ( bracket ) and side - scrolled [ and ] henry ' s $ 5 , 000 km / h with 3 . 34 m . what ' s up ! ? [SEP]
```
Then the only thing to add would be the replacements with the `@` character, from my point of view.

What do you think?
Hi, yes we already have a [dependency on sacremoses](https://github.com/huggingface/transformers/blob/master/setup.py#L128) for XLM so you can use it.

Do you want to try to propose a PR fixing this issue?
Ah ok, I didn't know that.

Sure, but could you please give me a hint where it's best to implement this piece of code? I lost track a bit since the tokenization refactoring and I'm not sure what method I would have to overwrite in `TransfoXLTokenizer`.
Yes of course, so the new API didn't touch any model-specific behavior, it was all about the user-facing up-stream methods.

In your case, I think you'll probably want to update the `_tokenize()` method of Transfo-XL tokenizer here: https://github.com/huggingface/transformers/blob/master/src/transformers/tokenization_transfo_xl.py#L339-L356

This is the method in charge of splitting words in token strings.

You can have a look at the XLM tokenizer if you want to see how people have been using sacremoses:
https://github.com/huggingface/transformers/blob/master/src/transformers/tokenization_xlm.py

Resolving patch:
diff --git a/src/transformers/tokenization_transfo_xl.py b/src/transformers/tokenization_transfo_xl.py
--- a/src/transformers/tokenization_transfo_xl.py
+++ b/src/transformers/tokenization_transfo_xl.py
@@ -22,11 +22,13 @@
 import os
 import pickle
 import re
+import warnings
 from collections import Counter, OrderedDict
-from typing import Optional
+from typing import List, Optional
 
 import numpy as np
 
+import sacremoses as sm
 from tokenizers import Tokenizer
 from tokenizers.implementations import BaseTokenizer
 from tokenizers.models import WordLevel
@@ -70,6 +72,47 @@
 }
 CORPUS_NAME = "corpus.bin"
 
+MATCH_NUMBERS = r"(?<=\d)[,.](?=\d)", r" @\g<0>@ "
+DETOKENIZE_NUMBERS = [(r" @\,@ ", r","), (r" @\.@ ", r".")]
+
+
+def tokenize_numbers(text_array: List[str]) -> List[str]:
+    """
+    Splits large comma-separated numbers and floating point values.
+    This is done by replacing commas with ' @,@ ' and dots with ' @.@ '.
+    Args:
+        text_array: An already tokenized text as list
+    Returns:
+        A list of strings with tokenized numbers
+    Example::
+        >>> tokenize_numbers(["$", "5,000", "1.73", "m"])
+        ["$", "5", "@,@", "000", "1", "@.@", "73", "m"]
+    """
+    tokenized = []
+    for i in range(len(text_array)):
+        reg, sub = MATCH_NUMBERS
+        replaced = re.sub(reg, sub, text_array[i]).split()
+        tokenized.extend(replaced)
+
+    return tokenized
+
+
+def detokenize_numbers(text: str) -> str:
+    """
+    Inverts the operation of `tokenize_numbers`.
+    This is replacing ' @,@ ' and ' @.@' by ',' and '.'.
+    Args:
+        text: A string where the number should be detokenized
+    Returns:
+        A detokenized string
+    Example::
+        >>> detokenize_numbers("$ 5 @,@ 000 1 @.@ 73 m")
+        "$ 5,000 1.73 m"
+    """
+    for reg, sub in DETOKENIZE_NUMBERS:
+        text = re.sub(reg, sub, text)
+    return text
+
 
 class TransfoXLTokenizer(PreTrainedTokenizer):
     """
@@ -97,6 +140,7 @@ def __init__(
         unk_token="<unk>",
         eos_token="<eos>",
         additional_special_tokens=["<formula>"],
+        language="en",
         **kwargs
     ):
         super().__init__(
@@ -118,6 +162,10 @@ def __init__(
         self.punctuation_symbols = '!"#$%&()*+,-./\\:;<=>?@[\\]^_`{|}~'
         self.punction_without_space_before_pattern = re.compile(r"[^\s][{}]".format(self.punctuation_symbols))
         self.punctuation_with_space_around_pattern = self._compile_space_around_punctuation_pattern()
+        self.language = language
+        self.moses_punct_normalizer = sm.MosesPunctNormalizer(language)
+        self.moses_tokenizer = sm.MosesTokenizer(language)
+        self.moses_detokenizer = sm.MosesDetokenizer(language)
 
         try:
             if pretrained_vocab_file is not None:
@@ -300,6 +348,34 @@ def move_added_token(self, token: str, target_idx: int):
         del self.added_tokens_decoder[old_index]
         del self.added_tokens_encoder[token]
 
+    def moses_punct_norm(self, text):
+        return self.moses_punct_normalizer.normalize(text)
+
+    def moses_tokenize(self, text):
+        return self.moses_tokenizer.tokenize(
+            text, aggressive_dash_splits=True, return_str=False, escape=False, protected_patterns=self.never_split
+        )
+
+    def moses_pipeline(self, text: str) -> List[str]:
+        """
+        Does basic tokenization using :class:`sacremoses.MosesPunctNormalizer` and :class:`sacremoses.MosesTokenizer`
+        with `aggressive_dash_splits=True` (see :func:`sacremoses.tokenize.MosesTokenizer.tokenize`).
+        Additionally, large comma-separated numbers and floating point values are split.
+        E.g. "23,000 people are 1.80m tall" -> "23 @,@ 000 people are 1 @.@ 80m tall".
+        Args:
+            text: Text to be tokenized
+        Returns:
+            A list of tokenized strings
+        Example::
+            >>> tokenizer = TransfoXLTokenizer.from_pretrained("transfo-xl-wt103")
+            >>> tokenizer.moses_pipeline("23,000 people are 1.80 m tall")
+            ['23', '@,@', '000', 'people', 'are', '1', '@.@', '80', 'm', 'tall']
+        """
+        text = self.moses_punct_norm(text)
+        text = self.moses_tokenize(text)
+        text = tokenize_numbers(text)
+        return text
+
     def _convert_id_to_token(self, idx):
         """Converts an id in a token (BPE) using the vocab."""
         assert 0 <= idx < len(self), "Index {} out of vocabulary range".format(idx)
@@ -323,9 +399,12 @@ def _convert_token_to_id(self, sym):
                 raise ValueError("Token not in vocabulary and no <unk> token in vocabulary for replacement")
 
     def convert_tokens_to_string(self, tokens):
-        """ Converts a sequence of tokens (string) in a single string. """
-        out_string = " ".join(tokens).strip()
-        return out_string
+        """
+        Converts a sequence of tokens (string) in a single string.
+        Additionally, the split numbers are converted back into it's original form.
+        """
+        out_string = self.moses_detokenizer.detokenize(tokens)
+        return detokenize_numbers(out_string).strip()
 
     def convert_to_tensor(self, symbols):
         return torch.LongTensor(self.convert_tokens_to_ids(symbols))
@@ -347,7 +426,7 @@ def _tokenize(self, line, add_eos=False, add_double_eos=False):
         if self.delimiter == "":
             symbols = line
         else:
-            symbols = line.split(self.delimiter)
+            symbols = self.moses_pipeline(line)
 
         if add_double_eos:  # lm1b
             return ["<S>"] + symbols + ["<S>"]
@@ -356,19 +435,6 @@ def _tokenize(self, line, add_eos=False, add_double_eos=False):
         else:
             return symbols
 
-    def prepare_for_tokenization(self, text, is_pretokenized=False, **kwargs):
-        # add spaces before punctuation symbols as should be done in transfo-xl
-        add_space_before_punct_symbol = kwargs.pop("add_space_before_punct_symbol", False)
-        if add_space_before_punct_symbol:
-            text = self.punctuation_with_space_around_pattern.sub(r" ", text)
-        elif self.punction_without_space_before_pattern.search(text):
-            # searches until the first occurence of a punctuation symbol without surrounding spaces
-            logger.warning(
-                "You might want to consider setting `add_space_before_punct_symbol=True` as an argument to the `tokenizer.encode()` to avoid tokenizing words with punctuation symbols to the `<unk>` token"
-            )
-
-        return (text, kwargs)
-
 
 class _TransfoXLDelimiterLookupTokenizer(BaseTokenizer):
     def __init__(
@@ -484,6 +550,11 @@ def __init__(
             **kwargs,
         )
 
+        warnings.warn(
+            "The class `TransfoXLTokenizerFast` is deprecated and will be removed in a future version. Please use `TransfoXLTokenizer` with it's enhanced tokenization instead.",
+            FutureWarning,
+        )
+
     def save_pretrained(self, save_directory):
         logger.warning(
             "Please note you will not be able to load the vocabulary in"


Test patch:
diff --git a/tests/test_tokenization_fast.py b/tests/test_tokenization_fast.py
--- a/tests/test_tokenization_fast.py
+++ b/tests/test_tokenization_fast.py
@@ -12,14 +12,12 @@
     OpenAIGPTTokenizer,
     PreTrainedTokenizer,
     RobertaTokenizer,
-    TransfoXLTokenizer,
     is_torch_available,
 )
 from transformers.testing_utils import get_tests_dir, require_torch
 from transformers.tokenization_distilbert import DistilBertTokenizerFast
 from transformers.tokenization_openai import OpenAIGPTTokenizerFast
 from transformers.tokenization_roberta import RobertaTokenizerFast
-from transformers.tokenization_transfo_xl import TransfoXLTokenizerFast
 
 
 logger = logging.getLogger(__name__)
@@ -895,17 +893,3 @@ def assert_padding(self, tokenizer_r, tokenizer_p, max_length=15):
             max_length=max_length,
             padding="max_length",
         )
-
-
-class TransfoXLFastTokenizerTest(NoPaddingTokenFastTokenizerMatchingTest):
-    TOKENIZERS_CLASSES = frozenset(
-        [Tokenizer("TransfoXL", TransfoXLTokenizerFast, TransfoXLTokenizer, "pretrained_vocab_file", None, None)]
-    )
-
-    @require_torch
-    def test_all_tokenizers(self):
-        super().test_all_tokenizers()
-
-    @require_torch
-    def test_pretokenized_tokenizers(self):
-        super().test_pretokenized_tokenizers()
diff --git a/tests/test_tokenization_transfo_xl.py b/tests/test_tokenization_transfo_xl.py
--- a/tests/test_tokenization_transfo_xl.py
+++ b/tests/test_tokenization_transfo_xl.py
@@ -83,6 +83,44 @@ def test_full_tokenizer_no_lower(self):
             tokenizer.tokenize(" \tHeLLo ! how  \n Are yoU ?  "), ["HeLLo", "!", "how", "Are", "yoU", "?"]
         )
 
+    def test_full_tokenizer_moses_numbers(self):
+        tokenizer = TransfoXLTokenizer(lower_case=False)
+        text_in = "Hello (bracket) and side-scrolled [and] Henry's $5,000 with 3.34 m. What's up!?"
+        tokens_out = [
+            "Hello",
+            "(",
+            "bracket",
+            ")",
+            "and",
+            "side",
+            "@-@",
+            "scrolled",
+            "[",
+            "and",
+            "]",
+            "Henry",
+            "'s",
+            "$",
+            "5",
+            "@,@",
+            "000",
+            "with",
+            "3",
+            "@.@",
+            "34",
+            "m",
+            ".",
+            "What",
+            "'s",
+            "up",
+            "!",
+            "?",
+        ]
+
+        self.assertListEqual(tokenizer.tokenize(text_in), tokens_out)
+
+        self.assertEqual(tokenizer.convert_tokens_to_string(tokens_out), text_in)
+
     def test_move_added_token(self):
         tokenizer = self.get_tokenizer()
         original_len = len(tokenizer)