GitHub repository: Dmitri9149/Rust_Tokenizers
Primary language: Rust
Created: 2021-02-02T12:53:16Z
Description: Tokenizers in Rust : from BPE to Sentence Piece and some modifications