GitHub repository: malaysia-ai/prepare-tokenizer
Primary language: Python
Created: 2023-10-17T15:20:27Z
Description: Prepare SentencePiece and BPE on Malaysian texts (Jawi, Melayu, Manglish, Mandarin, Tamil).