should model(tokenizer(text)) work for bigcode/santacoder?
#13
by
Dzmitry
- opened
The bigcode/santacoder tokenizer produces token_type_ids tensor. AFAIK the model was not trained to receive it as input. So model(tokenizer(text)["input_ids"])works differently from model(tokenizer(text)) (the former seems correct whereas the latter seems at least risky).
christopher
changed discussion status to
closed