Finnish-NLP
/

roberta-large-finnish-v2

Fill-Mask Transformers PyTorch JAX TensorBoard

Finnish roberta finnish Inference Endpoints

Model card Files Files and versions Metrics Training metrics Community

roberta-large-finnish-v2 / train_tokenizer.py

aapot

Add tokenizer

0b21f29 over 2 years ago

raw history blame contribute delete

No virus

962 Bytes

	from datasets import load_from_disk
	from tokenizers import trainers, Tokenizer, normalizers, ByteLevelBPETokenizer
	from transformers import AutoConfig, AutoTokenizer


	model_dir = "./"

	# load roberta-large config
	config = AutoConfig.from_pretrained("roberta-large")
	config.save_pretrained(model_dir)

	# load dataset
	dataset = load_from_disk("/researchdisk/training_dataset_full")
	dataset = dataset["train"]

	# Instantiate tokenizer
	tokenizer = ByteLevelBPETokenizer()
	def batch_iterator(batch_size=1000):
	for i in range(0, len(dataset), batch_size):
	yield dataset[i: i + batch_size]["text"]

	# Customized training
	tokenizer.train_from_iterator(batch_iterator(), vocab_size=config.vocab_size, min_frequency=2, special_tokens=[
	"<s>",
	"<pad>",
	"</s>",
	"<unk>",
	"<mask>",
	])

	# Save files to disk
	tokenizer.save(f"{model_dir}/tokenizer.json")
	tokenizer = AutoTokenizer.from_pretrained(model_dir)
	tokenizer.save_pretrained(model_dir)