Package: tokengeex
TokenGeeX - Efficient Tokenizer for CodeGeeXThis repository holds the code for the TokenGeeX Rust crate and Python package. TokenGeeX is a tokenizer forCodeGeeXaimed at code and Chinese. It is based onUnigramLM (Taku Kudo 2018)andTokenMonster.PythonYou can install thePyPI TokenGeeX packagethroughpip.pipinstalltokengeexExample usage:importtokengeextokenizer=tokengeex.load("code-32k-strict.json")# Vocabprint(tokenizer.vocab_size())# 32768print(tokenizer.token_to_id("token"))# 13513print(tokenizer.id_to_token(13513))# "token"# Encodeids=tokenizer.encode("def main(): print(\"Hello world!\")")print(ids)# [68, 437, 12747, 58, 14653, 2807, 1735, 10120]# Decodeprint(tokenizer.decode(ids))# "def main(): print(\"Hello world!\")"# Byte fallbacksprint([tokenizer.id_to_token(id)foridintokenizer.encode("电脑")])# ["电", "<0xe8>", "<0x84>", "<0x91>"]RustYou can install theRust library cratethroughcargo.cargoaddtokengeexExample usage:fnmain(){lettokenizer=tokengeex::load("code-32k-strict.json").unwrap();// Vocabprintln!("{}",tokenizer.vocab_size());// 32768println!("{}",.token_to_id("token").unwrap())// 13513println!("{:?}",.id_to_token(13513).unwrap())// "token"// Encodeletids=tokenizer.encode("def main(): print(\"Hello world!\")");println!("{:?}",ids);// [68, 437, 12747, 58, 14653, 2807, 1735, 10120]// Decodeprintln!("{:?}",tokenizer.decode(ids));// "def main(): print(\"Hello world!\")"// Byte fallbacksprintln!("{:?}",tokenizer.encode("电脑").map(|id|tokenizer.id_to_token(id)));// ["电", "<0xe8>", "<0x84>", "<0x91>"]}CLIYou can install theRust binary cratethroughcargo.cargo install tokengeex --features cliHere's a sample command to train a strict 1k vocabulary on 250MB of data.RUST_LOG=infoTOKENGEEX_PARALLELISM=truetokengeextrain--model'unigram'\--input'data/train/code-250MB.bin'\--output'data/vocab/code-1k-strict.json'\--special-token'
'\--special-token'
'\--special-token'
'\--special-token'
'\--vocab-size1024\--shrinking-factor'0.7'\--num-sub-iterations'2'\--vg-max-token-length'8'\--vg-max-words-per-token'2'\--vg-initial-vocab-size'4096'\--vg-insert-probability'0.01'\--vg-cache'data/cache/code-4k-250MB-strict.json'\--vg-stricttrueHere's a sample command to train a strict 4k vocabulary on 250MB of data.RUST_LOG=infoTOKENGEEX_PARALLELISM=truetokengeextrain--model'unigram'\--input'data/train/code-250MB.bin'\--output'data/vocab/code-4k-strict.json'\--special-token'
'\--special-token'
'\--special-token'
'\--special-token'
'\--vocab-size4096\--shrinking-factor'0.7'\--num-sub-iterations'2'\--suggested-tokens-file'data/tokens/suggested.json'\--added-tokens-file'data/tokens/added.json'\--vg-max-token-length'16'\--vg-max-words-per-token'3'\--vg-initial-vocab-size'100000'\--vg-insert-probability'0.01'\--vg-cache'data/cache/code-100k-250MB-strict.json'\--vg-stricttrueHere's a sample command to train a strict 16k vocabulary on 250MB of data.RUST_LOG=infoTOKENGEEX_PARALLELISM=truetokengeextrain--model'unigram'\--input'data/train/code-250MB.bin'\--output'data/vocab/code-16k-strict.json'\--special-token'
'\--special-token'
'\--special-token'
'\--special-token'
'\--vocab-size16384\--shrinking-factor'0.7'\--num-sub-iterations'2'\--suggested-tokens-file'data/tokens/suggested.json'\--added-tokens-file'data/tokens/added.json'\--vg-max-token-length'24'\--vg-max-words-per-token'3'\--vg-initial-vocab-size'1000000'\--vg-insert-probability'0.01'\--vg-cache'data/cache/code-1000k-250MB-strict.json'\--vg-stricttrueHere's a sample command to train a strict 32k vocabulary on 250MB of data.RUST_LOG=infoTOKENGEEX_PARALLELISM=truetokengeextrain--model'unigram'\--input'data/train/code-100MB.bin'\--output'data/vocab/code-32k-strict.json'\--special-token'
'\--special-token'
'\--special-token'
'\--special-token'
'\--vocab-size32768\--shrinking-factor'0.7'\--num-sub-iterations'2'\--suggested-tokens-file'data/tokens/suggested.json'\--added-tokens-file'data/tokens/added.json'\--vg-max-token-length'24'\--vg-max-words-per-token'3'\--vg-initial-vocab-size'1000000'\--vg-insert-probability'0.01'\--vg-cache'data/cache/code-1000k-100MB-strict.json'\--vg-stricttrue