Update tokenization_chatglm.py

#7

fix "piece id is out of range"

I also encountered this problem, but it had no effect after modifying it like this

Knowledge Engineering Group (KEG) & Data Mining at Tsinghua University org
edited Dec 7, 2023

We have been paying attention to this problem for a long time. Although we suspect that it may be caused by the user did not input in the correct format, we still added this line of code . Minimize the occurrence of accidents.
thank you for your suggestion

zRzRzRzRzRzRzR changed pull request status to closed

按照最新的配置文件,仍然出现这个错误,请问一下是什么原因呢?

Sign up or log in to comment