| Executing train_tokenizer.py | |
| ------------------------------ | |
| training bbpe tokenizer | |
| Initialize an empty tokenizer | |
| training | |
| saving model tokenizer to /home/shared/dt01/temutauro/ccasimiro/corpus-utils-lm/output/model-ready_output/bio-clinical-vocab-50262-2021-12-07-1604-d1d3-849e/train_tokenizer_output/train-tokenizer-2021-12-07-1625-d1d3-74b8 | |
| saving pretrained to /home/shared/dt01/temutauro/ccasimiro/corpus-utils-lm/output/model-ready_output/bio-clinical-vocab-50262-2021-12-07-1604-d1d3-849e/train_tokenizer_output/train-tokenizer-2021-12-07-1625-d1d3-74b8 | |
| saving config to /home/shared/dt01/temutauro/ccasimiro/corpus-utils-lm/output/model-ready_output/bio-clinical-vocab-50262-2021-12-07-1604-d1d3-849e/train_tokenizer_output/train-tokenizer-2021-12-07-1625-d1d3-74b8 | |