PAD tokens is present in `tokenizer.json` but not in `special_tokens_map.json`.

by sadra-barikbin - opened Jun 6

Jun 6

Hi there!
PAD token is present in tokenizer's added tokens in tokenizer.json as a special token with content "<|padding|>", while it's not in special_tokens_map.json and also in tokenizer_config.json which results in tokenizer.pad_token being None.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment