Repository: huggingface/tokenizers (language: rust)

Issue:
Load pre-trained tokenizer from memory
Hello guys,

`tokenizers::Tokenizer` has two methods to create a pre-trained tokenizer: `Tokenizer::from_pretrained` and `Tokenizer::from_file`. It's quite common that we need to download `tokenizer.json` from remote, now we have to save the remote data to a local file and then load the tokenizer from local file. We also have to delete it after loading. 

If we have something like `Tokenizer::from_memory(data_bytes: &[u8])`, we can just download the remote tokenizer to memory and then load it directly. It's quick and safe, you'll never need to delete the local file after using.


Discussion / hints:
This is a totally reasonable thing to ask.

Actually if you are in pure Rust, it already supports serde so

`let tokenizer: Tokenizer = serde_json::from_slice(&data_bytes).unwrap();` should work out of the box.

That being said adding a new function is definitely something we could do. `from_bytes` or `from_slice` seem like better names considering `std` but overall would still be a nice addition IMHO.

Fix patch:
diff --git a/tokenizers/src/tokenizer/mod.rs b/tokenizers/src/tokenizer/mod.rs
index 2595bfd9c..bff0e2a2e 100644
--- a/tokenizers/src/tokenizer/mod.rs
+++ b/tokenizers/src/tokenizer/mod.rs
@@ -402,6 +402,10 @@ impl Tokenizer {
         let tokenizer = serde_json::from_str(&content)?;
         Ok(tokenizer)
     }
+    pub fn from_bytes<P: AsRef<[u8]>>(bytes: P) -> Result<Self> {
+        let tokenizer = serde_json::from_slice(bytes.as_ref())?;
+        Ok(tokenizer)
+    }
     #[cfg(feature = "http")]
     pub fn from_pretrained<S: AsRef<str>>(
         identifier: S,
@@ -1133,6 +1137,21 @@ where
     }
 }
 
+impl<M, N, PT, PP, D> TokenizerImpl<M, N, PT, PP, D>
+where
+    M: DeserializeOwned + Model,
+    N: DeserializeOwned + Normalizer,
+    PT: DeserializeOwned + PreTokenizer,
+    PP: DeserializeOwned + PostProcessor,
+    D: DeserializeOwned + Decoder,
+{
+    /// Instantiate a new Tokenizer from bytes
+    pub fn from_bytes<P: AsRef<[u8]>>(bytes: P) -> Result<Self> {
+        let tokenizer = serde_json::from_slice(bytes.as_ref())?;
+        Ok(tokenizer)
+    }
+}
+
 impl<M, N, PT, PP, D> TokenizerImpl<M, N, PT, PP, D>
 where
     M: DeserializeOwned + Model,


Test patch: