Repository: huggingface/transformers

Issue:
01_how-to-train.ipynb broken
# 🐛 Bug

## To reproduce

Steps to reproduce the behavior:

1. Go to https://github.com/huggingface/transformers/tree/master/examples
2. Click the colab for `language-modeling`: https://colab.research.google.com/github/huggingface/blog/blob/master/notebooks/01_how_to_train.ipynb
3. Run notebook

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

The notebook finishes succesfuly

What I get is:

```
---------------------------------------------------------------------------
Exception                                 Traceback (most recent call last)
<ipython-input-5-52625a7c86e5> in <module>()
      1 get_ipython().system('mkdir EsperBERTo')
----> 2 tokenizer.save("EsperBERTo")

/usr/local/lib/python3.6/dist-packages/tokenizers/implementations/base_tokenizer.py in save(self, path, pretty)
    330                 A path to the destination Tokenizer file
    331         """
--> 332         return self._tokenizer.save(path, pretty)
    333 
    334     def to_str(self, pretty: bool = False):

Exception: Is a directory (os error 21)
```

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 2.11.0
- Platform: Linux-4.19.104+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.5.0+cu101 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: NA
- Using distributed or parallel set-up in script?: NA

01_how-to-train.ipynb broken
# 🐛 Bug

## To reproduce

Steps to reproduce the behavior:

1. Go to https://github.com/huggingface/transformers/tree/master/examples
2. Click the colab for `language-modeling`: https://colab.research.google.com/github/huggingface/blog/blob/master/notebooks/01_how_to_train.ipynb
3. Run notebook

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

The notebook finishes succesfuly

What I get is:

```
---------------------------------------------------------------------------
Exception                                 Traceback (most recent call last)
<ipython-input-5-52625a7c86e5> in <module>()
      1 get_ipython().system('mkdir EsperBERTo')
----> 2 tokenizer.save("EsperBERTo")

/usr/local/lib/python3.6/dist-packages/tokenizers/implementations/base_tokenizer.py in save(self, path, pretty)
    330                 A path to the destination Tokenizer file
    331         """
--> 332         return self._tokenizer.save(path, pretty)
    333 
    334     def to_str(self, pretty: bool = False):

Exception: Is a directory (os error 21)
```

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 2.11.0
- Platform: Linux-4.19.104+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.5.0+cu101 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: NA
- Using distributed or parallel set-up in script?: NA



Discussion:
@orestisfl thanks for raising this, I also was scratching my head as the same tokenizer.save("EsperBERTo") worked for me a few days ago and not anymore. I could save the tokenizer using tokenizer.save("EsperBERTo/vocab.txt"), but then I can't load it.  If I try to load I get:
```
TypeError: sep_token not found in the vocabulary
```
I'm using BertWordPieceTokenizer (not ByteLevelBPETokenizer used in your example) 

It worked with tokenizers version 0.7.0, I just checked - I got version 0.8.0rc1 currently installed.
I'll downgrade to 0.7.0 for now. 

Was this BC intended @n1t0?
Yes, `tokenizers` `0.8.0` introduces the full tokenizer serialization, whereas before it saved the "model" only (vocab.json + merges.txt for BPE). So the save method should be used like that: `.save("tokenizer.json")` and it saves the entire tokenizer to a JSON file.
We need to update the Notebook to use this new serialization method, but in the meantime, the only thing needed to make it work exactly like before is to replace:
```python
!mkdir EsperBERTo
tokenizer.save("EsperBERTo")
```
by
```python
!mkdir EsperBERTo
tokenizer.save_model("EsperBERTo")
```
mind updating it before we forget? Thanks!
Sure, updated it with the quick change I mentioned. Will do a better update later.
Hey there, thanks for the quick fix!
The notebook now crashes for me during training, however:
```
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-19-0c647bc3a8b8> in <module>()
----> 1 get_ipython().run_cell_magic('time', '', 'trainer.train()')

11 frames
<decorator-gen-60> in time(self, line, cell, local_ns)

<timed eval> in <module>()

/usr/local/lib/python3.6/dist-packages/transformers/data/data_collator.py in <listcomp>(.0)
    112         probability_matrix = torch.full(labels.shape, self.mlm_probability)
    113         special_tokens_mask = [
--> 114             self.tokenizer.get_special_tokens_mask(val, already_has_special_tokens=True) for val in labels.tolist()
    115         ]
    116         probability_matrix.masked_fill_(torch.tensor(special_tokens_mask, dtype=torch.bool), value=0.0)

AttributeError: 'RobertaTokenizerFast' object has no attribute 'get_special_tokens_mask'
```

Let me know if I should make a separate issue 
@orestisfl thanks for raising this, I also was scratching my head as the same tokenizer.save("EsperBERTo") worked for me a few days ago and not anymore. I could save the tokenizer using tokenizer.save("EsperBERTo/vocab.txt"), but then I can't load it.  If I try to load I get:
```
TypeError: sep_token not found in the vocabulary
```
I'm using BertWordPieceTokenizer (not ByteLevelBPETokenizer used in your example) 

It worked with tokenizers version 0.7.0, I just checked - I got version 0.8.0rc1 currently installed.
I'll downgrade to 0.7.0 for now. 

Was this BC intended @n1t0?
Yes, `tokenizers` `0.8.0` introduces the full tokenizer serialization, whereas before it saved the "model" only (vocab.json + merges.txt for BPE). So the save method should be used like that: `.save("tokenizer.json")` and it saves the entire tokenizer to a JSON file.
We need to update the Notebook to use this new serialization method, but in the meantime, the only thing needed to make it work exactly like before is to replace:
```python
!mkdir EsperBERTo
tokenizer.save("EsperBERTo")
```
by
```python
!mkdir EsperBERTo
tokenizer.save_model("EsperBERTo")
```
mind updating it before we forget? Thanks!
Sure, updated it with the quick change I mentioned. Will do a better update later.
Hey there, thanks for the quick fix!
The notebook now crashes for me during training, however:
```
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-19-0c647bc3a8b8> in <module>()
----> 1 get_ipython().run_cell_magic('time', '', 'trainer.train()')

11 frames
<decorator-gen-60> in time(self, line, cell, local_ns)

<timed eval> in <module>()

/usr/local/lib/python3.6/dist-packages/transformers/data/data_collator.py in <listcomp>(.0)
    112         probability_matrix = torch.full(labels.shape, self.mlm_probability)
    113         special_tokens_mask = [
--> 114             self.tokenizer.get_special_tokens_mask(val, already_has_special_tokens=True) for val in labels.tolist()
    115         ]
    116         probability_matrix.masked_fill_(torch.tensor(special_tokens_mask, dtype=torch.bool), value=0.0)

AttributeError: 'RobertaTokenizerFast' object has no attribute 'get_special_tokens_mask'
```

Let me know if I should make a separate issue 

Resolving patch:
diff --git a/src/transformers/tokenization_utils.py b/src/transformers/tokenization_utils.py
--- a/src/transformers/tokenization_utils.py
+++ b/src/transformers/tokenization_utils.py
@@ -755,16 +755,6 @@ def convert_tokens_to_string(self, tokens: List[str]) -> str:
     def decode(
         self, token_ids: List[int], skip_special_tokens: bool = False, clean_up_tokenization_spaces: bool = True
     ) -> str:
-        """
-        Converts a sequence of ids (integer) in a string, using the tokenizer and vocabulary
-        with options to remove special tokens and clean up tokenization spaces.
-        Similar to doing ``self.convert_tokens_to_string(self.convert_ids_to_tokens(token_ids))``.
-
-        Args:
-            token_ids: list of tokenized input ids. Can be obtained using the `encode` or `encode_plus` methods.
-            skip_special_tokens: if set to True, will replace special tokens.
-            clean_up_tokenization_spaces: if set to True, will clean up the tokenization spaces.
-        """
         filtered_tokens = self.convert_ids_to_tokens(token_ids, skip_special_tokens=skip_special_tokens)
 
         # To avoid mixing byte-level and unicode for byte-level BPT
diff --git a/src/transformers/tokenization_utils_base.py b/src/transformers/tokenization_utils_base.py
--- a/src/transformers/tokenization_utils_base.py
+++ b/src/transformers/tokenization_utils_base.py
@@ -1774,6 +1774,51 @@ def _pad(
     def batch_decode(self, sequences: List[List[int]], **kwargs) -> List[str]:
         return [self.decode(seq, **kwargs) for seq in sequences]
 
+    def decode(
+        self, token_ids: List[int], skip_special_tokens: bool = False, clean_up_tokenization_spaces: bool = True
+    ) -> str:
+        """
+        Converts a sequence of ids (integer) in a string, using the tokenizer and vocabulary
+        with options to remove special tokens and clean up tokenization spaces.
+        Similar to doing ``self.convert_tokens_to_string(self.convert_ids_to_tokens(token_ids))``.
+
+        Args:
+            token_ids: list of tokenized input ids. Can be obtained using the `encode` or `encode_plus` methods.
+            skip_special_tokens: if set to True, will replace special tokens.
+            clean_up_tokenization_spaces: if set to True, will clean up the tokenization spaces.
+        """
+        raise NotImplementedError
+
+    def get_special_tokens_mask(
+        self, token_ids_0: List, token_ids_1: Optional[List] = None, already_has_special_tokens: bool = False
+    ) -> List[int]:
+        """
+        Retrieves sequence ids from a token list that has no special tokens added. This method is called when adding
+        special tokens using the tokenizer ``prepare_for_model`` or ``encode_plus`` methods.
+
+        Args:
+            token_ids_0: list of ids (must not contain special tokens)
+            token_ids_1: Optional list of ids (must not contain special tokens), necessary when fetching sequence ids
+                for sequence pairs
+            already_has_special_tokens: (default False) Set to True if the token list is already formated with
+                special tokens for the model
+
+        Returns:
+            A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token.
+        """
+        assert already_has_special_tokens and token_ids_1 is None, (
+            "You cannot use ``already_has_special_tokens=False`` with this tokenizer. "
+            "Please use a slow (full python) tokenizer to activate this argument."
+            "Or set `return_special_token_mask=True` when calling the encoding method "
+            "to get the special tokens mask in any tokenizer. "
+        )
+
+        all_special_ids = self.all_special_ids  # cache the property
+
+        special_tokens_mask = [1 if token in all_special_ids else 0 for token in token_ids_0]
+
+        return special_tokens_mask
+
     @staticmethod
     def clean_up_tokenization(out_string: str) -> str:
         """ Clean up a list of simple English tokenization artifacts like spaces before punctuations and abreviated forms.


Test patch:
diff --git a/tests/test_tokenization_common.py b/tests/test_tokenization_common.py
--- a/tests/test_tokenization_common.py
+++ b/tests/test_tokenization_common.py
@@ -672,29 +672,6 @@ def test_special_tokens_mask_input_pairs(self):
                 filtered_sequence = [x for x in filtered_sequence if x is not None]
                 self.assertEqual(encoded_sequence, filtered_sequence)
 
-    def test_special_tokens_mask_already_has_special_tokens(self):
-        tokenizers = self.get_tokenizers(do_lower_case=False)
-        for tokenizer in tokenizers:
-            if not hasattr(tokenizer, "get_special_tokens_mask") or tokenizer.get_special_tokens_mask(
-                [0, 1, 2, 3]
-            ) == [0, 0, 0, 0]:
-                continue
-            with self.subTest(f"{tokenizer.__class__.__name__}"):
-                sequence_0 = "Encode this."
-                if (
-                    tokenizer.cls_token_id == tokenizer.unk_token_id
-                    and tokenizer.cls_token_id == tokenizer.unk_token_id
-                ):
-                    tokenizer.add_special_tokens({"cls_token": "
", "sep_token": "
"})
-                encoded_sequence_dict = tokenizer.encode_plus(
-                    sequence_0, add_special_tokens=True, return_special_tokens_mask=True
-                )
-                # encoded_sequence_w_special = encoded_sequence_dict["input_ids"]
-                special_tokens_mask_orig = encoded_sequence_dict["special_tokens_mask"]
-                min_val = min(special_tokens_mask_orig)
-                max_val = max(special_tokens_mask_orig)
-                self.assertNotEqual(min_val, max_val)
-
     def test_right_and_left_padding(self):
         tokenizers = self.get_tokenizers(do_lower_case=False)
         for tokenizer in tokenizers: