Fix AutoTokenizerFromPretrained crash with transformer 4.39.3
#48
by
Qubitium
- opened
There is no need to add auto_map to config. This is used for remote_code tokenizer and cohere's tokenizer is already in hf repo and there is no remote code tokenizer available. Having this will actually crash transformer AutoTokenzierFromPretrained the second you use "use_fast=True" or even "use_fast=False". Without this PR, you need to not pass "use_fast" to not crash .
Qubitium
changed pull request title from
Fix compat with transformer 4.39.3
to Fix AutoTokenizerFromPretrained crash with transformer 4.39.3
Thanks!
ahmetustun
changed pull request status to
closed
@ahmetustun Why was this closed and not merged?
@Qubitium there was a merge conflict due to another merge however I incorporated the suggested change.