Commit
•
d8e1cb8
1
Parent(s):
81b7b8a
Create README.md
Browse files
README.md
ADDED
|
@@ -0,0 +1,16 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
language: "ru"
|
| 3 |
+
tags:
|
| 4 |
+
- paraphrasing
|
| 5 |
+
- russian
|
| 6 |
+
license: mit
|
| 7 |
+
---
|
| 8 |
+
|
| 9 |
+
This is a small Russian paraphraser based on the [google/mt5-small](https://huggingface.co/google/mt5-small) model.
|
| 10 |
+
|
| 11 |
+
It was obtained by taking the [alenusch/mt5small-ruparaphraser](https://huggingface.co/alenusch/mt5small-ruparaphraser) model and stripping 96% of its vocabulary which is unrelated to the Russian language or infrequent.
|
| 12 |
+
|
| 13 |
+
* The original model has 300M parameters, with 256M of them being input and output embeddings.
|
| 14 |
+
* After shrinking the `sentencepiece` vocabulary from 250K to 20K the number of model parameters reduced from 1.1GB to 246MB.
|
| 15 |
+
* The first 5K tokens in the new vocabulary are taken from the original `mt5-small`.
|
| 16 |
+
* The next 15K tokens are the most frequent tokens obtained by tokenizing a Russian web corpus from [the Leipzig corpora collection](https://wortschatz.uni-leipzig.de/en/download/Russian).
|