Skier8402
/

code-search-net-tokenizer

Model card Files Files and versions Community

This is an adapted tokenizer from GPT2 that can recognize tokens to do with Python coding. It is part of the huggingfaceNLP course exercise. It uses the method train_new_from_iterator()

Downloads last month: 0

Dataset used to train Skier8402/code-search-net-tokenizer