metadata
datasets:
- armvectores/hy_wikipedia_2023
pipeline_tag: feature-extraction
language:
- hy
library_name: fasttext
414M tokens
- 73M hy wikipedia
- 341M arlis database
74951 unique words
3-5 ngrams
5 window length
300 embedding dim
skipgram
minimum number of words 150
100 epochs, 0.05 start lr
26 hours on 20 xeon gold cores
How to use
- Install fastText
pip install fasttext-wheel
- Import fastText in python
import fasttext
from huggingface_hub import hf_hub_download
model_path = hf_hub_download(local_dir=".",
repo_id="armvectores/wikipedia_arlis_tokens_fasttextskipgram_300_5",
filename="model.bin")
model = fasttext.load_model(model_path)
- Examples of usage
word = 'զենքեր'
print(model.get_nearest_neighbors(word))
print(model.get_sentence_vector(word))