Salesforce

/

codegen25-7b-multi_P

like 131

CodeGen25Tokenizer correctly implements the _decode interface from PreTrainedTokenizer with the following signature

def _decode(self, token_ids: List[int], ...,) -> str:
   ...

PreTrainedTokenizer, however, incorrectly shadows the _decode function of its base class PreTrainedTokenizerBase, which is defined like this:

def _decode(self, token_ids: Union[int, List[int]], ...,) -> str:
   ...

As a result, CodeGen25Tokenizer cannot be used as a drop-in tokenizer in some codebases (like TGI). This fix doesn't break any previous behaviour, but simply allows decode to also accept plain int values instead of only list[int].