fa-floret-wiki-vectors

Floret word vectors for Persian, trained on Persian Wikipedia with floret-torch, a GPU (PyTorch) port of explosion/floret.

Trained on the full fawiki dump (WikiExtractor, spaCy blank("fa") tokenization): 8,428,449 sentences, 190,781,621 tokens.

--model cbow --mode floret --dim 300 --minn 5 --maxn 5 \
--hashCount 2 --bucket 200000 --neg 10 --epoch 5 --lr 0.05 \
--minCount 20 --batch 8192

This is the table the fa_dep_news_lg / fa_core_news_lg / fa_ent_news_lg pipelines are trained against.

Files

• fa.floret — hashed bucket table. Reconstructs a vector for any word, including ones never seen in training, via subword n-gram hashing.

  python -m spacy init vectors fa fa.floret ./fa_pipeline --mode floret
  

• fa.vec — plain word2vec text format, fixed vocabulary above minCount 20. No OOV lookup.

  from gensim.models import KeyedVectors
  kv = KeyedVectors.load_word2vec_format("fa.vec")
  

• fa_floret_wiki_200k-0.1.0-py3-none-any.whl — installable vectors-only spaCy pipeline (pipeline: []) with fa.floret embedded as a 200,000-row x 300d table.

  pip install https://huggingface.co/Phazel/fa-floret-wiki-vectors/resolve/main/fa_floret_wiki_200k-0.1.0-py3-none-any.whl
  

  import spacy
  nlp = spacy.load("fa_floret_wiki_200k")
  print(nlp.vocab.vectors.shape)   # (200000, 300)
  

  To train against it without installing anything, point spaCy's --paths.vectors at an unpacked copy (scripts/unpack_vectors.py in spacy-persian).

Related tables

Package	Rows	Corpus	Used by
fa_floret_400k	50k	400k Persian documents	md tier
fa_floret_full_wiki	50k	full Persian Wikipedia dump	—
fa_floret_wiki_200k (this repo)	200k	full Persian Wikipedia dump, 5 epochs	lg tier

Licence

CC BY-SA 4.0. The vectors are a derivative of Persian Wikipedia text (CC BY-SA 4.0), so the share-alike condition carries over; this is also the licence recorded for this table in the lg pipelines' meta.json. The floret-torch training tooling is MIT and imposes no condition on the output.