Repository: huggingface/transformers (Python) — Bug Fix

Issue:
ByT5: problem with tokenizer.decode()
## Environment info

- transformers version: 4.11.0
- Platform:  Google Colab
- Python version: 3.7.12
- Using GPU in script?: NO
- Using distributed or parallel set-up in script?: NO

### Who can help

ByT5: @patrickvonplaten 
Documentation: @sgugger 

## Information

Model I am using: `google/byt5-small` (the problem is the same with `google/byt5-base`).

## To reproduce

See this [notebook ](https://colab.research.google.com/drive/1ZS_zPF_ShLU0SKVLt5zYNHqoPOenBkEN?usp=sharing&authuser=1#scrollTo=PiKc6U3atGoh) that shows the problem when using `google/byt5-small` from the model hub of Hugging Face and the `tokenizer.decode()` method, when the `transformers `version is 4.11.0.

The problem does not appear with the `transformers `version 4.9.2 for example.

```
from transformers import T5ForConditionalGeneration, ByT5Tokenizer

model_checkpoint = 'google/byt5-small'
model = T5ForConditionalGeneration.from_pretrained(model_checkpoint)
tokenizer = ByT5Tokenizer.from_pretrained(model_checkpoint)

texts = ["Life is like a box of chocolates.", "Today is Monday."]

for text in texts:
  inputs = tokenizer(text, padding="longest", return_tensors="pt")
  output = model.generate(**inputs)
  print(tokenizer.decode(output[0], skip_special_tokens=True, clean_up_tokenization_spaces=True))
```

Error:

```
---------------------------------------------------------------------------
UnicodeDecodeError                        Traceback (most recent call last)
<ipython-input-8-6f8451a23561> in <module>()
      6       output[0],
      7       skip_special_tokens=True,
----> 8       clean_up_tokenization_spaces=True
      9       )
     10   )

2 frames
/usr/local/lib/python3.7/dist-packages/transformers/models/byt5/tokenization_byt5.py in convert_tokens_to_string(self, tokens)
    238                 tok_string = bytes([ord(token)])
    239             bstring += tok_string
--> 240         string = bstring.decode("utf-8")
    241         return string
    242 

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte
```

## Expected behavior

2 strings as outputs of the ByT5.



Discussion:
Hey :)

for faster debugging this can be break-downed to:

```python
from transformers import T5ForConditionalGeneration, ByT5Tokenizer

model_checkpoint = 'google/byt5-small'
tokenizer = ByT5Tokenizer.from_pretrained(model_checkpoint)

print(tokenizer.decode([258], skip_special_tokens=True, clean_up_tokenization_spaces=True))
```

The "official" ByT5 tokenizer is used from `seqio` and their implementation would return:

```python
from seqio import ByteVocabulary

tokenizer = ByteVocabulary()

tokenizer._decode([258])

# Returns:
# ''

# Better test:
tokenizer._decode([258]) == ''
# Return True
```
But as seen in the `ByteVocabulary()` implementation:

https://github.com/google/seqio/blob/main/seqio/vocabularies.py#L399

they use `errors="ignore"` as attribute of the `.decode()` function. Maybe this kind of error handling should also be applied here:

https://github.com/huggingface/transformers/blob/83d3dc0f6f8ae03e01aa5acacf88e79b2c1ecd06/src/transformers/models/byt5/tokenization_byt5.py#L240

:thinking: 
Update: this seems to be intended:

https://github.com/huggingface/transformers/commit/5c7789d4167064f7464b8801c7488a9a2878480a

Pinging @Narsil :)
Hello @stefan-it.

Thank you very much for taking the time to verify the problem.

Now I understand that `string = bstring.decode(" utf-8 ", error =" ignore ")` has been replaced by `string = bstring.decode(" utf-8 ")` by @Narsil (see [5c7789d](https://github.com/huggingface/transformers/commit/5c7789d4167064f7464b8801c7488a9a2878480a)) but because of this, it is not possible anymore:
- to use for example `model.generate()` with a `ByT5` model (because it will fail)
- and it is not possible to finetune a `ByT5` model (because when evaluating metrics it will use `tokenizer.decode()` that will fail).

We must find a solution. Do you have a proposal?
@Narsil - could you take a look once you're back?
Hi @piegu @stefan-it @patrickvonplaten ,

Do you know what you would expect to see instead ?

IMHO, failing here is perfectly correct as there is no correct way to represent byte (255) on its own.

If ByT5 generates invalid bytes, then the part that is supposed to recover a string should fail just like regular Python IMO, that's a model fail (it failed to generate bytes that correspond to a real string). Ignoring the error will just hide the problem under the rug and not really solve the problem. If you really don't care about failed generated bytes, then having a way to opt-in a different way of decoding even with malformed bytes makes sense. For the library, I am not sure it's a desired behavior by default as really we're ignoring a real model error.

If I take a less "simple" error where the model would generate `b'\x80\x81\x82\x83\x84\x85\x86\x87\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f\x90\x91\x92\x93\x94\x95\x96\x97\x98\x99\x9a\x9b\x9c\x9d\x9e\x9f\xa0\xa1\xa2\xa3\xa4\xa5\xa6\xa7\xa8\xa9\xaa\xab\xac\xad\xae\xaf\xb0\xb1\xb2\xb3\xb4\xb5\xb6\xb7\xb8\xb9\xba\xbb\xbc\xbd\xbe\xbf\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd7\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7\xe8\xe9\xea\xeb\xec\xed\xee\xef\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff'` then if we had errors=ignore by default, this would be `''` which would be very surprising to say the least as the model actually generated 128 tokens here...

Proposed modifications to make things easier for users:

- Add a new method specific to bytes tokenizers that would return raw `bytes` instead of `str`. This would not fail, and it would really be a user responsability to use it. (`decode_bytes` ?). Here it would return `b'\xff chocolates chocol'` leaving the user in charge to do something meaningful with it.
- Add a way to override to `errors=ignore` somehow directly from `decode`. It would add non trivial complexity to tokenizer code as `decode` is a generic method though.
- Add a way to override the `errors=ignore` through some attribute of the tokenizer. A bit better as it does not spill into generic code, but probably less discoverable (would need to add this to the doc in a very clear way).
- Switching back to `errors=ignore` but I really think it's a mistake in that case (just like having `errors=ignore` within Python itself would be pretty bad).

My personal take on this is that the first solution seems the better, but happy to hear counterarguments or if I overlooked something.
Hi @Narsil.

I would start with your question `Do you know what you would expect to see instead ?`
My answer: just a model (ByT5 here) with an output that can be always decoded in order to use the model in production and to finetune it with new data.... as this is the case for all models of the HF model hub.

In fact, I read and understood your arguments about not using `errors=ignore` in the `.decode()` method (`bstring.decode("utf-8")` instead of `bstring.decode("utf-8", errors="ignore")`) but the problem is in fact about ByT5 outputs, not about its `.decode()` method.

In my opinion, the true question is: **Why the model ByT5 in the Transformers library outputs tokens that can not be decoded?**

When I use the model BERT or T5 to generate outputs, I do not have the generation of such tokens (for example, I do not have the generation of a token id that is out of the tokenizer ids list).

And if you have a loot at the HF model hub, there are the Google ByT5 models and the finetuned ones:
- **Google ByT5 models**: how did Google train their ByT5 models? Indeed, at the end of each epoch, it is necessary to use  a method `.decode()` in order to obtain the generated texts and thus compare them to the targeted ones. Google used `errors=ignore`?
- **ByT5 finetuned models**: which version of Transformers the ByT5 models finetuned in the HF model hub used? Certainly the version 4.9.2 as this [notebook](https://colab.research.google.com/drive/1syXmhEQ5s7C59zU8RtHVru0wAvMXTSQ8) but not the actual one. What does it mean about the **quality** of these finetuned ByT5 models (which were finetuned with `errors=ignore` ) and how to finetune now a ByT5 model with the actual Transformers version?

What do you think? Should we focus on the method `.decode()` or on debugging ByT5?
> Hi @Narsil.
> 
> I would start with your question `Do you know what you would expect to see instead ?` My answer: just a model (ByT5 here) with an output that can be always decoded in order to use the model in production and to finetune it with new data.... as this is the case for all models of the HF model hub.

ByT5, unlike ALL other models (afaik), uses raw bytes, so it has no guarantee whatsoever to output a `string`.
It will however always produce `bytes`. (hence `decode_bytes` proposal).

> 
> In fact, I read and understood your arguments about not using `errors=ignore` in the `.decode()` method (`bstring.decode("utf-8")` instead of `bstring.decode("utf-8", errors="ignore")`) but the problem is in fact about ByT5 outputs, not about its `.decode()` method.
> 
> In my opinion, the true question is: **Why the model ByT5 in the Transformers library outputs tokens that can not be decoded?**

I have no idea, but it's expected that if it can produce non string data, it will (at some point at least). 

> 
> When I use the model BERT or T5 to generate outputs, I do not have the generation of such tokens (for example, I do not have the generation of a token id that is out of the tokenizer ids list).
> 
> And if you have a loot at the HF model hub, there are the Google ByT5 models and the finetuned ones:
> 
>     * **Google ByT5 models**: how did Google train their ByT5 models? Indeed, at the end of each epoch, it is necessary to use  a method `.decode()` in order to obtain the generated texts and thus compare them to the targeted ones. Google used `errors=ignore`?

Probably differently. All `string` can be casted to `bytes` (but not the other way around). So checking two `bytes` objects was probably the way it was done as this is always possible. Take the generated output, convert to bytes, take expected string, convert to bytes and compare the.

> 
>     * **ByT5 finetuned models**: which version of Transformers the ByT5 models finetuned in the HF model hub used? Certainly the version 4.9.2 as this [notebook](https://colab.research.google.com/drive/1syXmhEQ5s7C59zU8RtHVru0wAvMXTSQ8) but not the actual one. What does it mean about the **quality** of these finetuned ByT5 models (which were finetuned with `errors=ignore` ) and how to finetune now a ByT5 model with the actual Transformers version?
> 
> 
> What do you think? Should we focus on the method `.decode()` or on debugging ByT5?

I think we should accept that `ByT5` is different from other models, propose `decode_bytes` method and let users try to do things with it.

We could also break standard API and `decode` would return `bytes` instead of `string` but that would break many things, the automated tests at the very least.
To weigh in on this discussion, I wanted to reiterate the points raised by @piegu:
>  it is not possible anymore:
>
>  * to use for example `model.generate()` with a `ByT5` model (because it will fail)
>  * and it is not possible to finetune a `ByT5` model (because when evaluating metrics it will use `tokenizer.decode()` that will fail).

This means that it would always be required to overwrite the `evaluate` function when using `Seq2SeqTrainer` in combination with `predict_with_generate`, unless the `decode_bytes` option is directly addressed in the `Trainer`/`generate` implementation as well (creating additional overhead).

Since it is required to pass a `Tokenizer` in any case, I would prefer the option to choose directly through the tokenizer whether to ignore errors or not. I agree that it would have to be quite visible, but even for the T5 repository's implementation, this behavior is not very obvious ([reference issue](https://github.com/google-research/byt5/issues/11)), but implemented with ignoring the errors by default.

As to this point: 

> So checking two bytes objects was probably the way it was done as this is always possible. Take the generated output, convert to bytes, take expected string, convert to bytes and compare the.

I don't see any indication of the evaluation on `bytes` objects instead of `string`, as there seem to be no modifications on top of the vanilla T5 modeling from their own repository.
Thanks a lot for the nice repro @stefan-it! 
To be honest, I think we should just add `errors="ignore"` for the following reasons:

- One of the philosophies of `transformers` is to stay as close as possible to the original code
- If google added `errors="ignore"`, it was probably intended and is therefore not a bug IMO
- We broke backwards compatibility between 4.9 and 4.11 

What do you think @Narsil ? 

Also cc @LysandreJik here 
If google did it, then let's do it.

Resolving patch:
diff --git a/src/transformers/models/byt5/tokenization_byt5.py b/src/transformers/models/byt5/tokenization_byt5.py
--- a/src/transformers/models/byt5/tokenization_byt5.py
+++ b/src/transformers/models/byt5/tokenization_byt5.py
@@ -237,7 +237,7 @@ def convert_tokens_to_string(self, tokens):
             else:
                 tok_string = bytes([ord(token)])
             bstring += tok_string
-        string = bstring.decode("utf-8")
+        string = bstring.decode("utf-8", errors="ignore")
         return string
 
     # ByT5Tokenizer has no vocab file


Test patch:
diff --git a/tests/test_tokenization_byt5.py b/tests/test_tokenization_byt5.py
--- a/tests/test_tokenization_byt5.py
+++ b/tests/test_tokenization_byt5.py
@@ -290,6 +290,22 @@ def test_special_tokens_initialization_with_non_empty_additional_special_tokens(
                     ),
                 )
 
+    def test_decode_single_bytes(self):
+        tokenizer_list = []
+        if self.test_slow_tokenizer:
+            tokenizer_list.append((self.tokenizer_class, self.get_tokenizer()))
+
+        if self.test_rust_tokenizer:
+            tokenizer_list.append((self.rust_tokenizer_class, self.get_rust_tokenizer()))
+
+        for tokenizer_class, tokenizer_utils in tokenizer_list:
+            with tempfile.TemporaryDirectory() as tmp_dir:
+                tokenizer_utils.save_pretrained(tmp_dir)
+
+                tokenizer = tokenizer_class.from_pretrained(tmp_dir)
+
+                self.assertTrue(tokenizer.decode([255]) == "")
+
     # tokenizer can be instantiated without any pretrained files, so no need for pretrained tokenizer list
     def test_pretrained_model_lists(self):
         pass