Hugging Face
Models
Datasets
Spaces
Posts
Docs
Solutions
Pricing
Log In
Sign Up
balochiml
/
balochi-tokenizer
like
1
Baluchi
balochi
tokenizer
License:
apache-2.0
Model card
Files
Files and versions
Community
main
balochi-tokenizer
/
src
/
train_tokenizer.ipynb
Commit History
more spacy
e938b17
strickvl
commited on
Jun 3, 2023
add spacy trainer
b1d540a
strickvl
commited on
Jun 3, 2023
update trainer script
e8868b3
strickvl
commited on
Jun 3, 2023
update tokenizer script
57c829b
strickvl
commited on
Jun 3, 2023
add tokenization process
24469e0
strickvl
commited on
Jun 3, 2023
process and clean files
b3db2b8
strickvl
commited on
Jun 3, 2023
Get all text paths
470e696
strickvl
commited on
Jun 3, 2023
add data folder to gitignore
c9036ae
strickvl
commited on
Jun 3, 2023