Delete tokenizer.json

by rabeya100X - opened 2 days ago

base: refs/heads/main

←

from: refs/pr/4

Discussion Files changed

-32150

rabeya100X

2 days ago

Removing stale tokenizer.json (32K vocab) as discussed in transformers PR #48973 so AutoTokenizer relies on vocab.txt (100K vocab) via tokenizer_config.json.

Delete tokenizer.json2e151833

AMR-KELEG

Modeling the Variation in Arabic org 1 day ago

Thanks @rabeya100X !
I believe this indeed is enough to fix the bug.

AMR-KELEG changed pull request status to merged 1 day ago

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment