Repository: huggingface/tokenizers (language: rust)

Issue:
Is there a way to supress this warning?

```
warn!("The OrderedVocab you are attempting to save contains a hole for index {}, your vocabulary could be corrupted !", i);
```
Found here: https://github.com/huggingface/tokenizers/blob/5f6e9784526a4cd5e4f6dcdcc045cdceba5463e1/tokenizers/src/models/mod.rs#L42C53-L42C53


Our OrderedVocab has holes on purpose, so this warning is quite annoying.

Thanks!


Discussion / hints:
> has holes on purpose

You do understand how this waste model memory (by having unused slots) and that this is definitely a very odd situation to be in.
This should technically be a hard error, but unfortunately, some relatively popular legacy models do have holes (by accident), that's why we downgraded to hard warning.

Do you mind sharing a bit more details on the 'on purpose' situation ? I don't understand how this can be a normal situation.

@Narsil, we maintain a collection of tokenizers that all map to a consistent ID space. e.g. tokenizer A and B have same special tokens (mapping to IDs 0-20), but different regular tokens (tokenizer A mapping to IDs 21-100, and tokenizer B to IDs 101-1000). Tokenizer B has a hole in ID space where A's regular tokens are mapped, but when these are combined, the ID space is contiguous. 
Furthermore, we want to maintain buffers to add future tokens, and for these we also need to leave holes in all tokenizers. This way, if we add tokens (or whole tokenizers) in the future, we won't have to change existing mapping. We're aware of the memory waste, but it's a price we're willing to pay for the modularity.
I understand.

This seems like a very power user thing to do then. I think leaving and ignoring the message (since you're aware it doesn't apply to you) is bearable then, no ?

And the final A+B tokenizer should have no holes in the end, which I imagine will be the final thing, correct  ?
You are correct about the final tokenizer having no holes, however, since we store the sub-tokenizers (A, B) separately, we still get the error quite often, even when we don't change them (e.g. I think it triggers in Tokenizer.from_str/to_str). 
The problem is that every new user that sees this warning asks for an explanation, and it's getting unmanageable. Also, since some holes can be quite big (hundreds of tokens), and the warning repeats for every single unmapped index, these messages tend to drown any meaningful output.
It would really help if it were possible to suppress this warning somehow. 
I can work out having a single warning instead of many, but I really don't want to make the warning suppressible. It is a critical issue, that as I mention, should ideally be a `raise Exception` and cannot only because of legacy tokenizers.

Having subtokenizers is not really something intended than I can realistically support. I just don't have the bandwidth for it.


If possible, single warning instead of per-ID, would be a significant improvement, thanks!

Fix patch:
diff --git a/tokenizers/src/models/mod.rs b/tokenizers/src/models/mod.rs
index ade7878fa..f7ef3df15 100644
--- a/tokenizers/src/models/mod.rs
+++ b/tokenizers/src/models/mod.rs
@@ -34,20 +34,26 @@ impl<'a> Serialize for OrderedVocabIter<'a> {
         S: Serializer,
     {
         // There could be holes so max + 1 is more correct than vocab_r.len()
-        if let Some(max) = self.vocab_r.iter().map(|(key, _)| key).max() {
+        let mut holes = vec![];
+        let result = if let Some(max) = self.vocab_r.iter().map(|(key, _)| key).max() {
             let iter = (0..*max + 1).filter_map(|i| {
             if let Some(token) = self.vocab_r.get(&i){
                 Some((token, i))
             }else{
-                warn!("The OrderedVocab you are attempting to save contains a hole for index {}, your vocabulary could be corrupted !", i);
-                println!("The OrderedVocab you are attempting to save contains a hole for index {}, your vocabulary could be corrupted !", i);
+                holes.push(i);
                 None
             }
-        });
+            });
             serializer.collect_map(iter)
         } else {
             serializer.collect_map(std::iter::empty::<(&str, u32)>())
+        };
+
+        if !holes.is_empty(){
+            warn!("The OrderedVocab you are attempting to save contains holes for indices {:?}, your vocabulary could be corrupted !", holes);
+            println!("The OrderedVocab you are attempting to save contains holes for indices {:?}, your vocabulary could be corrupted !", holes);
         }
+        result
     }
 }
 


Test patch: