metadata
license: cc0-1.0
datasets:
- code_search_net
library_name: transformers
tags:
- text-generation
- code
- python
This is an adapted tokenizer from GPT2 that can recognize tokens to do with Python coding. It is part of the huggingfaceNLP course exercise. It uses the method train_new_from_iterator()