|
--- |
|
language: "ru" |
|
tags: |
|
- paraphrasing |
|
- russian |
|
license: mit |
|
--- |
|
|
|
This is a small Russian paraphraser based on the [google/mt5-small](https://huggingface.co/google/mt5-small) model. |
|
|
|
It was obtained by taking the [alenusch/mt5small-ruparaphraser](https://huggingface.co/alenusch/mt5small-ruparaphraser) model and stripping 96% of its vocabulary which is unrelated to the Russian language or infrequent. |
|
|
|
* The original model has 300M parameters, with 256M of them being input and output embeddings. |
|
* After shrinking the `sentencepiece` vocabulary from 250K to 20K the number of model parameters reduced from 1.1GB to 246MB. |
|
* The first 5K tokens in the new vocabulary are taken from the original `mt5-small`. |
|
* The next 15K tokens are the most frequent tokens obtained by tokenizing a Russian web corpus from [the Leipzig corpora collection](https://wortschatz.uni-leipzig.de/en/download/Russian). |
|
|
