---
datasets:
- armvectores/hy_wikipedia_2023
pipeline_tag: feature-extraction
language:
- hy
library_name: fasttext
---

414M tokens
1) 73M hy wikipedia
2) 341M arlis database 

74951 unique words

3-5 ngrams

5 window length

300 embedding dim

skipgram

minimum number of words 150

100 epochs, 0.05 start lr

26 hours on 20 xeon gold cores

How to use

1) Install fastText

```
pip install fasttext-wheel
```

2) Import fastText in python
   
```
import fasttext
from huggingface_hub import hf_hub_download

model_path = hf_hub_download(local_dir=".",
                             repo_id="armvectores/wikipedia_arlis_tokens_fasttextskipgram_300_5",
                             filename="model.bin")
model = fasttext.load_model(model_path)

```

3) Examples of usage

```
word = 'զենքեր'
print(model.get_nearest_neighbors(word))
print(model.get_sentence_vector(word))

```