How to tokenize within getitem() of Dataset with HuggingFace tokenizers
Hi All,
I see in the docs that we should tokenize everything in prepare_data(), but my dataset doesn't fit into memory, so I tokenize in the dataset.getitem() call, such that this happens upon each batch call.
I tried this out but the tokenizer instance (and location) changes between the lightningmodule.prepare_data() and the dataset.getitem() calls. The tokenizer instance in lightningmodule.prepare_data() is perfect, but the latter instance is just an empty copy with empty attributes - which leads to the error "padding was requested but your tokenizer has no padding token" - despite the tokenizer being initialized with a padding token in lightningmodule.prepare_data().
Any clues?