GitHub repository: EleutherAI/tokengrams
Primary language: Rust
Created: 2023-12-21T15:22:20Z
Description: Efficiently computing & storing token n-grams from large corpora