Repository: speechbrain/speechbrain

Issue:
minloglevel issue with SentencePiece
Hi @aheba,

It's not the first time that I face this problem, and it's a big one on Cedar (could be the case on other clusters). SentencePiece 0.1.92 cannot be installed (as it is yanked according to the official GitHub, the version has some big regression bugs). Problem: 0.1.91 doesn't accept minloglevel as an argument raising an error 

**OSError: Not found: unknown field name "minloglevel" in TrainerSpec.**
I suppose that we have to remove this option so it always work, but the logs are terrible ... any ideas ?


Discussion:
Adding to the problem, on Cedar (or some cluster), it seems the SentencePiece tokenizer cannot be pickle-able which cause the failure of the DDP mode. Does anyone has some ideas to get around this issue?
Hello @TParcollet ,
as discussed there is a problem with `0.1.92` and have been yanked see: https://github.com/google/sentencepiece/issues/505
They will jump directly to `0.1.93`, so let me push a PR for handling the `minloglevel` for versions <= `0.1.91`.

@JianyuanZhong , here is how we can have a pickle-able tokenizer:
https://github.com/google/sentencepiece/issues/387
Did you solve the issue ?, I can add the modification in the PR
@JianyuanZhong  I found this: https://forums.fast.ai/t/trouble-saving-with-sentencepiece-tokenizer/42143
It may help.
Do you solve the problem of pickle-able on our tokenizer ? I can try DDP on the mila cluster.. !
Could you tell us where do you see this error ? which recipe I can use to reproduce your error ?
I also notice that the sentencepiece remove the `0.1.92`here:
https://pypi.org/project/sentencepiece/
so if we install our requirements with `pip install -r requirements.txt`, the sentencepiece lib will not be installed because of `sentencepiece>=0.1.92`

Resolving patch:
diff --git a/speechbrain/tokenizers/SentencePiece.py b/speechbrain/tokenizers/SentencePiece.py
--- a/speechbrain/tokenizers/SentencePiece.py
+++ b/speechbrain/tokenizers/SentencePiece.py
@@ -57,9 +57,6 @@ class SentencePiece:
         Default: -1, if -1 the bos_id = unk_id = 0. otherwise, bos_id = int.
     eos_id: int
         Default: -1, if -1 the bos_id = unk_id = 0. otherwise, bos_id = int.
-    minloglevel: int
-        Default: 1, if 1 then the log(INFO) for BPE training is skipped
-        if 0, the log is printed.
 
     Example
     -------
@@ -91,7 +88,6 @@ def __init__(
         eos_id=-1,
         pad_id=-1,
         unk_id=0,
-        minloglevel=1,
     ):
         if model_type not in ["unigram", "bpe", "char"]:
             raise ValueError("model_type must be one of : [unigram, bpe, char]")
@@ -118,7 +114,6 @@ def __init__(
         self.eos_id = str(eos_id)
         self.pad_id = str(pad_id)
         self.unk_id = str(unk_id)
-        self.minloglevel = str(minloglevel)
 
         if not os.path.isfile(self.prefix_model_file + ".model"):
             logger.info("Train tokenizer with type:" + self.model_type)
@@ -188,8 +183,6 @@ def _train_BPE(self):
             + self.max_sentencepiece_length
             + " --character_coverage="
             + self.character_coverage
-            + " --minloglevel="
-            + self.minloglevel
         )
         if self.model_type not in ["char"]:
             # include vocab_size


Test patch: