GitHub repository: sayakpaul/count-tokens-hf-datasets
Created: 2022-06-10T03:25:56Z
Description: This project shows how to derive the total number of training tokens from a large text dataset from 🤗 datasets with Apache Beam and Dataflow.