File size: 769 Bytes
4886c47
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
Executing train_tokenizer.py
------------------------------
training bbpe tokenizer
Initialize an empty tokenizer
training
saving model tokenizer to /home/shared/dt01/temutauro/ccasimiro/corpus-utils-lm/output/model-ready_output/biomedical-vocab-50262-2021-12-09-1207-d1d3-e42b/train_tokenizer_output/train-tokenizer-2021-12-09-1223-d1d3-4dfc
saving pretrained to /home/shared/dt01/temutauro/ccasimiro/corpus-utils-lm/output/model-ready_output/biomedical-vocab-50262-2021-12-09-1207-d1d3-e42b/train_tokenizer_output/train-tokenizer-2021-12-09-1223-d1d3-4dfc
saving config to /home/shared/dt01/temutauro/ccasimiro/corpus-utils-lm/output/model-ready_output/biomedical-vocab-50262-2021-12-09-1207-d1d3-e42b/train_tokenizer_output/train-tokenizer-2021-12-09-1223-d1d3-4dfc