Repository: huggingface/tokenizers (language: rust)

Issue:
added_tokens with bytemap charaters in ByteLevel could not be decoded correctly
I just found that if added tokens contain some characters that exist in the byte map for ByteLevel preprocessor could not be decoded correctly.
This is a script to reproduce the problem with version 0.14.1
```python
from tokenizers import Tokenizer
from tokenizers import normalizers
from tokenizers.pre_tokenizers import (
    ByteLevel,
)
from tokenizers.models import BPE
from tokenizers import decoders
tokenizer = Tokenizer(BPE())
tokenizer.normalizer = normalizers.Sequence([])

tokenizer.pre_tokenizer = Sequence(
    [
        ByteLevel(add_prefix_space=False, use_regex=False),
    ])
tokenizer.add_tokens(["ilÖveyou"])
# Ö is the character representing for 0xf6
tokenizer.decoder = decoders.ByteLevel()
encode_result = tokenizer.encode("ilÖveyou")
print(encode_result.ids)
print(tokenizer.decode(encode_result.ids))
```
the output wil be 
```
[0]
il�veyou
```

I believe the problem comes from 
https://github.com/huggingface/tokenizers/blob/main/tokenizers/src/tokenizer/mod.rs#L832-L836
I don't think added token should be sent to bytelevel decoder for it is extacted before pretokenize.


Discussion / hints:
Hey! Thanks for the report, the output is the same if you use `tokenizers==0.13.3` so unrelated to the `0.14.1` release. 
I think one solution would be to have a normalizer that does ByteLevel, and set normalized to false for the tokens. Not sure we have an other solution for now. 

AddedTokens usually have to be sent to the decoder, because the pre-tokenizatio is applied to them. 
is it possible to let added_tokens_map_r of AddedVocabulary store the mapping of id to tokens after pre-tokenization? so that it can generate the same output after decoder?
It's not possible no, normalizers are use for that purpose however. And you can also add the token like this:
Since the text is first splitted and normalized, and then pre_tokenized, adding the pre-tokenized version of the token to the `added_tokens_map_r` will still not work. 
In this case you could use a `StripAccents` normalizer to make sure accents are stripped, but this will also affect all the other tokens. 
Ah I see. Thanks for your explaination. Is there any planning on solving this bug?
This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.
A pr for a fix that is backward compatible is welcome! Otherwise I won't have time to dive in this 🤗 
This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.
https://github.com/meta-llama/llama3/issues/67#issuecomment-2072952721 TLDR this should help: 
```python3
>>> from tokenizers import AddedToken, pre_tokenizers
>>> from transformers import AutoTokenizer
>>> pre_tokenizers.ByteLevel(False,False).pre_tokenize_str("Bác")
[('BÃ¡c', (0, 3))]
>>> tokenizer = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3-8B")
>>> tokenizer.add_tokens(AddedToken("BÃ¡c", normalized=False,special=False))
>>> tokenizer.decode(tokenizer.encode("Bác"))
'
Bác'
```
Re-opening as the merge on main will be reverted for a better fix soon

Fix patch:
diff --git a/bindings/python/py_src/tokenizers/normalizers/__init__.py b/bindings/python/py_src/tokenizers/normalizers/__init__.py
index 15a16f1e2..86d233bd2 100644
--- a/bindings/python/py_src/tokenizers/normalizers/__init__.py
+++ b/bindings/python/py_src/tokenizers/normalizers/__init__.py
@@ -15,7 +15,7 @@
 Nmt = normalizers.Nmt
 Precompiled = normalizers.Precompiled
 Replace = normalizers.Replace
-
+ByteLevel = normalizers.ByteLevel
 
 NORMALIZERS = {"nfc": NFC, "nfd": NFD, "nfkc": NFKC, "nfkd": NFKD}
 
diff --git a/bindings/python/py_src/tokenizers/normalizers/__init__.pyi b/bindings/python/py_src/tokenizers/normalizers/__init__.pyi
index 507d44731..8c4e744d1 100644
--- a/bindings/python/py_src/tokenizers/normalizers/__init__.pyi
+++ b/bindings/python/py_src/tokenizers/normalizers/__init__.pyi
@@ -99,6 +99,47 @@ class BertNormalizer(Normalizer):
         """
         pass
 
+class ByteLevel(Normalizer):
+    """
+    Bytelevel Normalizer
+    """
+    def __init__(self):
+        pass
+
+    def normalize(self, normalized):
+        """
+        Normalize a :class:`~tokenizers.NormalizedString` in-place
+
+        This method allows to modify a :class:`~tokenizers.NormalizedString` to
+        keep track of the alignment information. If you just want to see the result
+        of the normalization on a raw string, you can use
+        :meth:`~tokenizers.normalizers.Normalizer.normalize_str`
+
+        Args:
+            normalized (:class:`~tokenizers.NormalizedString`):
+                The normalized string on which to apply this
+                :class:`~tokenizers.normalizers.Normalizer`
+        """
+        pass
+
+    def normalize_str(self, sequence):
+        """
+        Normalize the given string
+
+        This method provides a way to visualize the effect of a
+        :class:`~tokenizers.normalizers.Normalizer` but it does not keep track of the alignment
+        information. If you need to get/convert offsets, you can use
+        :meth:`~tokenizers.normalizers.Normalizer.normalize`
+
+        Args:
+            sequence (:obj:`str`):
+                A string to normalize
+
+        Returns:
+            :obj:`str`: A string after normalization
+        """
+        pass
+
 class Lowercase(Normalizer):
     """
     Lowercase Normalizer
diff --git a/bindings/python/src/normalizers.rs b/bindings/python/src/normalizers.rs
index 645852fa8..864947e39 100644
--- a/bindings/python/src/normalizers.rs
+++ b/bindings/python/src/normalizers.rs
@@ -9,8 +9,8 @@ use crate::utils::{PyNormalizedString, PyNormalizedStringRefMut, PyPattern};
 use serde::ser::SerializeStruct;
 use serde::{Deserialize, Deserializer, Serialize, Serializer};
 use tk::normalizers::{
-    BertNormalizer, Lowercase, Nmt, NormalizerWrapper, Precompiled, Prepend, Replace, Strip,
-    StripAccents, NFC, NFD, NFKC, NFKD,
+    BertNormalizer, ByteLevel, Lowercase, Nmt, NormalizerWrapper, Precompiled, Prepend, Replace,
+    Strip, StripAccents, NFC, NFD, NFKC, NFKD,
 };
 use tk::{NormalizedString, Normalizer};
 use tokenizers as tk;
@@ -70,6 +70,9 @@ impl PyNormalizer {
                         Py::new(py, (PyBertNormalizer {}, base))?.into_py(py)
                     }
                     NormalizerWrapper::Prepend(_) => Py::new(py, (PyPrepend {}, base))?.into_py(py),
+                    NormalizerWrapper::ByteLevel(_) => {
+                        Py::new(py, (PyByteLevel {}, base))?.into_py(py)
+                    }
                     NormalizerWrapper::StripAccents(_) => {
                         Py::new(py, (PyStripAccents {}, base))?.into_py(py)
                     }
@@ -435,6 +438,18 @@ impl PyPrepend {
     }
 }
 
+/// Bytelevel Normalizer
+#[pyclass(extends=PyNormalizer, module = "tokenizers.normalizers", name = "ByteLevel")]
+pub struct PyByteLevel {}
+#[pymethods]
+impl PyByteLevel {
+    #[new]
+    #[pyo3(text_signature = "(self)")]
+    fn new() -> (Self, PyNormalizer) {
+        (PyByteLevel {}, ByteLevel::new().into())
+    }
+}
+
 /// StripAccents normalizer
 #[pyclass(extends=PyNormalizer, module = "tokenizers.normalizers", name = "StripAccents")]
 pub struct PyStripAccents {}
@@ -647,6 +662,7 @@ pub fn normalizers(m: &Bound<'_, PyModule>) -> PyResult<()> {
     m.add_class::<PyStrip>()?;
     m.add_class::<PyStripAccents>()?;
     m.add_class::<PyPrepend>()?;
+    m.add_class::<PyByteLevel>()?;
     m.add_class::<PyNmt>()?;
     m.add_class::<PyPrecompiled>()?;
     m.add_class::<PyReplace>()?;
diff --git a/tokenizers/src/normalizers/byte_level.rs b/tokenizers/src/normalizers/byte_level.rs
new file mode 100644
index 000000000..42c7fa510
--- /dev/null
+++ b/tokenizers/src/normalizers/byte_level.rs
@@ -0,0 +1,180 @@
+use crate::processors::byte_level::bytes_char;
+use crate::tokenizer::{NormalizedString, Normalizer, Result};
+use serde::{Deserialize, Serialize};
+use std::collections::{HashMap, HashSet};
+
+#[derive(Clone, Debug, Deserialize, Serialize)]
+#[serde(tag = "type")]
+pub struct ByteLevel {}
+
+lazy_static! {
+    static ref BYTES_CHAR: HashMap<u8, char> = bytes_char();
+    static ref CHAR_BYTES: HashMap<char, u8> =
+        bytes_char().into_iter().map(|(c, b)| (b, c)).collect();
+}
+
+impl Default for ByteLevel {
+    fn default() -> Self {
+        Self::new()
+    }
+}
+
+impl ByteLevel {
+    pub fn new() -> Self {
+        Self {}
+    }
+
+    pub fn alphabet() -> HashSet<char> {
+        BYTES_CHAR.values().copied().collect()
+    }
+}
+
+impl Normalizer for ByteLevel {
+    /// Strip the normalized string inplace
+    fn normalize(&self, normalized: &mut NormalizedString) -> Result<()> {
+        if !normalized.is_empty() {
+            let s = normalized.get();
+            let mut transformations: Vec<(char, isize)> = Vec::with_capacity(s.len());
+            let mut i = 0;
+            for cur_char in s.chars() {
+                let size = cur_char.len_utf8();
+                let bytes = s[i..i + size].as_bytes();
+                i += size;
+                transformations.extend(
+                    bytes
+                        .iter()
+                        .enumerate()
+                        .map(|(i, b)| (BYTES_CHAR[b], isize::from(i > 0))),
+                );
+            }
+            normalized.transform(transformations, 0);
+        }
+        Ok(())
+    }
+}
+
+#[cfg(test)]
+mod tests {
+
+    use super::*;
+
+    #[test]
+    fn test_byte_level_normalize() {
+        let original = "Hello 我今天能为你做什么";
+        let normalized = "HelloĠæĪĳä»Ĭå¤©èĥ½ä¸ºä½łåģļä»Ģä¹Ī";
+        assert_ne!(original, normalized);
+        let mut n = NormalizedString::from(original);
+        let byte_level = ByteLevel::new();
+        byte_level.normalize(&mut n).unwrap();
+        assert_eq!(&n.get(), &normalized);
+        assert_eq!(
+            n,
+            NormalizedString::new(
+                original.to_string(),
+                normalized.to_string(),
+                vec![
+                    (0, 1),
+                    (1, 2),
+                    (2, 3),
+                    (3, 4),
+                    (4, 5),
+                    (5, 6),
+                    (5, 6),
+                    (6, 9),
+                    (6, 9),
+                    (6, 9),
+                    (6, 9),
+                    (6, 9),
+                    (6, 9),
+                    (9, 12),
+                    (9, 12),
+                    (9, 12),
+                    (9, 12),
+                    (9, 12),
+                    (9, 12),
+                    (12, 15),
+                    (12, 15),
+                    (12, 15),
+                    (12, 15),
+                    (12, 15),
+                    (12, 15),
+                    (15, 18),
+                    (15, 18),
+                    (15, 18),
+                    (15, 18),
+                    (15, 18),
+                    (15, 18),
+                    (18, 21),
+                    (18, 21),
+                    (18, 21),
+                    (18, 21),
+                    (18, 21),
+                    (18, 21),
+                    (21, 24),
+                    (21, 24),
+                    (21, 24),
+                    (21, 24),
+                    (21, 24),
+                    (21, 24),
+                    (24, 27),
+                    (24, 27),
+                    (24, 27),
+                    (24, 27),
+                    (24, 27),
+                    (24, 27),
+                    (27, 30),
+                    (27, 30),
+                    (27, 30),
+                    (27, 30),
+                    (27, 30),
+                    (27, 30),
+                    (30, 33),
+                    (30, 33),
+                    (30, 33),
+                    (30, 33),
+                    (30, 33),
+                    (30, 33)
+                ],
+                0
+            )
+        );
+        assert_eq!(
+            n.alignments_original(),
+            vec![
+                (0, 1),
+                (1, 2),
+                (2, 3),
+                (3, 4),
+                (4, 5),
+                (5, 7),
+                (7, 13),
+                (7, 13),
+                (7, 13),
+                (13, 19),
+                (13, 19),
+                (13, 19),
+                (19, 25),
+                (19, 25),
+                (19, 25),
+                (25, 31),
+                (25, 31),
+                (25, 31),
+                (31, 37),
+                (31, 37),
+                (31, 37),
+                (37, 43),
+                (37, 43),
+                (37, 43),
+                (43, 49),
+                (43, 49),
+                (43, 49),
+                (49, 55),
+                (49, 55),
+                (49, 55),
+                (55, 61),
+                (55, 61),
+                (55, 61)
+            ]
+        );
+    }
+}
diff --git a/tokenizers/src/normalizers/mod.rs b/tokenizers/src/normalizers/mod.rs
index 8ac4c58ec..c5144be14 100644
--- a/tokenizers/src/normalizers/mod.rs
+++ b/tokenizers/src/normalizers/mod.rs
@@ -1,19 +1,19 @@
 pub mod bert;
+pub mod byte_level;
 pub mod precompiled;
 pub mod prepend;
 pub mod replace;
 pub mod strip;
 pub mod unicode;
 pub mod utils;
-
 pub use crate::normalizers::bert::BertNormalizer;
+pub use crate::normalizers::byte_level::ByteLevel;
 pub use crate::normalizers::precompiled::Precompiled;
 pub use crate::normalizers::prepend::Prepend;
 pub use crate::normalizers::replace::Replace;
 pub use crate::normalizers::strip::{Strip, StripAccents};
 pub use crate::normalizers::unicode::{Nmt, NFC, NFD, NFKC, NFKD};
 pub use crate::normalizers::utils::{Lowercase, Sequence};
-
 use serde::{Deserialize, Serialize};
 
 use crate::{NormalizedString, Normalizer};
@@ -35,6 +35,7 @@ pub enum NormalizerWrapper {
     Precompiled(Precompiled),
     Replace(Replace),
     Prepend(Prepend),
+    ByteLevel(ByteLevel),
 }
 
 impl Normalizer for NormalizerWrapper {
@@ -53,6 +54,7 @@ impl Normalizer for NormalizerWrapper {
             Self::Precompiled(lc) => lc.normalize(normalized),
             Self::Replace(lc) => lc.normalize(normalized),
             Self::Prepend(lc) => lc.normalize(normalized),
+            Self::ByteLevel(lc) => lc.normalize(normalized),
         }
     }
 }
@@ -70,3 +72,4 @@ impl_enum_from!(Nmt, NormalizerWrapper, Nmt);
 impl_enum_from!(Precompiled, NormalizerWrapper, Precompiled);
 impl_enum_from!(Replace, NormalizerWrapper, Replace);
 impl_enum_from!(Prepend, NormalizerWrapper, Prepend);
+impl_enum_from!(ByteLevel, NormalizerWrapper, ByteLevel);
diff --git a/tokenizers/src/pre_tokenizers/byte_level.rs b/tokenizers/src/pre_tokenizers/byte_level.rs
index 6343bbd07..2d3845b55 100644
--- a/tokenizers/src/pre_tokenizers/byte_level.rs
+++ b/tokenizers/src/pre_tokenizers/byte_level.rs
@@ -11,7 +11,7 @@ use crate::utils::macro_rules_attribute;
 
 /// Converts bytes to unicode characters.
 /// See https://github.com/openai/gpt-2/blob/master/src/encoder.py#L9
-fn bytes_char() -> HashMap<u8, char> {
+pub(crate) fn bytes_char() -> HashMap<u8, char> {
     let mut bs: Vec<u8> = vec![];
     bs.extend(b'!'..=b'~');
     bs.extend(b'\xA1'..=b'\xAC');
diff --git a/tokenizers/src/tokenizer/added_vocabulary.rs b/tokenizers/src/tokenizer/added_vocabulary.rs
index 301d9bc81..a0c2f4542 100644
--- a/tokenizers/src/tokenizer/added_vocabulary.rs
+++ b/tokenizers/src/tokenizer/added_vocabulary.rs
@@ -543,6 +543,7 @@ impl Serialize for AddedVocabulary {
 #[cfg(test)]
 mod tests {
     use super::*;
+    use crate::normalizers::byte_level::ByteLevel as ByteLevelNormalizer;
     use crate::normalizers::utils::Lowercase;
     use crate::normalizers::NormalizerWrapper;
     use crate::{OffsetReferential, OffsetType, Result, Token, Trainer};
@@ -1000,4 +1001,32 @@ mod tests {
             ]
         );
     }
+    #[test]
+    fn byte_level_normalizer() {
+        // Is able to extract both normal and special tokens
+        let model = ModelMock::new(&[]);
+        let mut vocab = AddedVocabulary::new();
+        let from = NormalizerWrapper::from(ByteLevelNormalizer::new());
+        let normalizer: Option<&NormalizerWrapper> = Some(&from);
+
+        vocab.add_tokens(
+            &[AddedToken::from("my", false), AddedToken::from("今", false)],
+            &model,
+            normalizer,
+        );
+        let result = vocab.extract_and_normalize(normalizer, "my今");
+        assert_eq!(
+            result
+                .get_splits(OffsetReferential::Original, OffsetType::Byte)
+                .into_iter()
+                .map(|(s, _, tokens)| (
+                    s,
+                    tokens
+                        .as_ref()
+                        .map(|t| t.iter().map(|t| t.id).collect::<Vec<_>>())
+                ))
+                .collect::<Vec<_>>(),
+            vec![("my", Some(vec![0])), ("ä»Ĭ", Some(vec![1])),]
+        );
+    }
 }
diff --git a/tokenizers/src/tokenizer/mod.rs b/tokenizers/src/tokenizer/mod.rs
index b0836ca3c..99e2b7127 100644
--- a/tokenizers/src/tokenizer/mod.rs
+++ b/tokenizers/src/tokenizer/mod.rs
@@ -1294,3 +1294,61 @@ where
         Ok(())
     }
 }
+
+#[cfg(test)]
+mod test {
+
+    use crate::AddedToken;
+    use crate::Tokenizer;
+
+    #[cfg(feature = "http")]
+    #[test]
+    fn test_decoding_with_added_bpe() {
+        use crate::{
+            normalizers,
+            pre_tokenizers::split::{Split, SplitPattern},
+            NormalizerWrapper, PreTokenizerWrapper, SplitDelimiterBehavior,
+        };
+
+        let mut tokenizer = Tokenizer::from_pretrained("meta-llama/Meta-Llama-3-8B", None).unwrap();
+        tokenizer.normalizer = Some(NormalizerWrapper::from(normalizers::ByteLevel::new()));
+        tokenizer.pre_tokenizer = Some(PreTokenizerWrapper::Split(
+            Split::new(
+                SplitPattern::Regex(r"(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\\r\\n\\p{L}\\p{N}]?\\p{L}+|\\p{N}{1,3}| ?[^\\s\\p{L}\\p{N}]+[\\r\\n]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+".into()),
+                SplitDelimiterBehavior::Isolated,
+                false,
+            )
+            .unwrap(),
+        ));
+        tokenizer.add_tokens(&[AddedToken::from("嗎", false).normalized(false)]);
+        let encoded = tokenizer
+            .encode("Hey! how is this token: 嗎", false)
+            .unwrap();
+        assert_eq!(
+            encoded.get_ids(),
+            [19182, 0, 1268, 602, 82, 62428, 82, 4037, 25, 220, 128256]
+        );
+        assert_eq!(
+            encoded.get_tokens(),
+            ["Hey", "!", "Ġhow", "Ġi", "s", "Ġthi", "s", "Ġtoken", ":", "Ġ", "嗎"]
+        );
+
+        let decoded = tokenizer.decode(encoded.get_ids(), false);
+        assert_eq!(decoded.unwrap(), "Hey! how is this token: 嗎");
+
+        tokenizer.add_tokens(&[AddedToken::from("д", false).normalized(true)]);
+        let encoded = tokenizer
+            .encode("Hey! how is this token: д", false)
+            .unwrap();
+        assert_eq!(
+            encoded.get_ids(),
+            [19182, 0, 1268, 602, 82, 62428, 82, 4037, 25, 220, 128257]
+        );
+        assert_eq!(
+            encoded.get_tokens(),
+            ["Hey", "!", "Ġhow", "Ġi", "s", "Ġthi", "s", "Ġtoken", ":", "Ġ", "Ð´"]
+        );
+        let decoded = tokenizer.decode(encoded.get_ids(), false);
+        assert_eq!(decoded.unwrap(), "Hey! how is this token: д")
+    }
+}


Test patch:
diff --git a/bindings/python/tests/bindings/test_tokenizer.py b/bindings/python/tests/bindings/test_tokenizer.py
index 3ac50e00c..39f110d07 100644
--- a/bindings/python/tests/bindings/test_tokenizer.py
+++ b/bindings/python/tests/bindings/test_tokenizer.py
@@ -150,6 +150,8 @@ def test_encode(self):
         assert len(output) == 2
 
     def test_encode_formats(self, bert_files):
+        print("Broken by the change from std::usize::Max to usixeMax")
+        return 0
         with pytest.deprecated_call():
             tokenizer = BertWordPieceTokenizer(bert_files["vocab"])