Update README.md

README.md

@@ -1,3 +1,206 @@
 ---
+language:
+- ru
+tags:
+- spellchecking
+- M2M100
+- pytorch
+- natural language generation
 license: mit
+datasets:
+- ai-forever/spellcheck_benchmark
+metrics:
+- precision
+- recall
+- f1
+library_name: transformers
+model-index:
+- name: sage-mt5-large
+  results:
+  - task:
+      type: text-generation
+    dataset:
+      type: spellcheck_benchmark
+      name: RUSpellRU 
+    metrics:
+    - name: Precision
+      type: precision
+      value: 88.8
+      verified: false
+    - name: Recall
+      type: recall
+      value: 71.5
+      verified: false
+    - name: F1 
+      type: f1
+      value: 79.2
+      verified: false
+  - task:
+      type: text-generation
+    dataset:
+      type: spellcheck_benchmark
+      name: MultidomainGold 
+    metrics:
+    - name: Precision
+      type: precision
+      value: 63.8
+      verified: false
+    - name: Recall
+      type: recall
+      value: 61.1
+      verified: false
+    - name: F1 
+      type: f1
+      value: 62.4
+      verified: false
+  - task:
+      type: text-generation
+    dataset:
+      type: spellcheck_benchmark
+      name: MedSpellchecker
+    metrics:
+    - name: Precision
+      type: precision
+      value: 78.8
+      verified: false
+    - name: Recall
+      type: recall
+      value: 71.4
+      verified: false
+    - name: F1 
+      type: f1
+      value: 74.9
+      verified: false
+  - task:
+      type: text-generation
+    dataset:
+      type: spellcheck_benchmark
+      name: GitHubTypoCorpusRu
+    metrics:
+    - name: Precision
+      type: precision
+      value: 47.1
+      verified: false
+    - name: Recall
+      type: recall
+      value: 42.9
+      verified: false
+    - name: F1 
+      type: f1
+      value: 44.9
+      verified: false
 ---
+# sage-m2m100-1.2B model
+
+## Summary
+
+The model corrects spelling errors and typos by bringing all the words in the text to the norm of the Russian language.
+Corrector was trained based on the model [M2M100-1.2B](https://huggingface.co/facebook/m2m100_1.2B). 
+An extensive dataset with “artificial” errors was taken as a training corpus: the corpus was assembled on the basis of the Russian-language Wikipedia and transcripts of Russian-language videos, then typos and spelling errors were automatically introduced into it using the library [SAGE](https://github.com/ai-forever/sage).
+The model is the fine-tuned version of the [pre-train](https://huggingface.co/ai-forever/RuM2M100-1.2B). 
+
+## Public references
+- [SAGE library announcement](https://youtu.be/yFfkV0Qjuu0), DataFest 2023
+- [Paper about synthetic error generation methods](https://www.dialog-21.ru/media/5914/martynovnplusetal056.pdf), Dialogue 2023
+- [SAGE EACL 2024 paper](https://aclanthology.org/2024.findings-eacl.10/)
+
+
+## Examples
+| Input | Output |
+| --- | --- |
+| Думю ешцъа лет череа 10 ретроспективно просматривотьэ то будкетцц мне невероя тна ин те р но | Думаю что лет через 10 ретроспективно просматривать это будет мне невероятно интересно |
+| Основая цель мероприятия - практическая отработка навыков по оказанию помощи гражданам, попавшим в ДТП, а также повышение и совершенствование уровня профессиональной подготовки сотрудников МЧС при проведении аварийно-спасательных работ по ликвидации последствий дорожно-транспортных проишествий, сокращение временных показателей реагирования. | Основная цель мероприятия - практическая отработка навыков по оказанию помощи гражданам, попавшим в ДТП, а также повышение и совершенствование уровня профессиональной подготовки сотрудников МЧС при проведении аварийно-спасательных работ по ликвидации последствий дорожно-транспортных происшествий, сокращение временных показателей реагирования. |
+| прийдя в МГТУ я был удивлен никого необноружив там… | придя в МГТУ я был удивлен никого не обнаружив там |
+|  |  |
+
+## Metrics
+### Quality
+Below are automatic metrics for determining the correctness of the spell checkers. 
+We compare our solution with both open automatic spell checkers and the ChatGPT family of models on all four available datasets:
+- **RUSpellRU**: texts collected from ([LiveJournal](https://www.livejournal.com/media)), with manually corrected typos and errors;
+- **MultidomainGold**: examples from 7 text sources, including the open web, news, social media, reviews, subtitles, policy documents and literary works;
+- **MedSpellChecker**: texts with errors from medical anamnesis;
+- **GitHubTypoCorpusRu**: spelling errors and typos in commits from [GitHub](https://github.com);
+
+**RUSpellRU**
+| Model | Precision | Recall | F1 |
+| --- | --- | --- | --- |
+| sage-m2m100-1.2B | 88.8 | 71.5 | 79.2 |
+| sage-ai-service | 93.5 | 82.4 | 87.6 | 
+| gpt-3.5-turbo | 39.6 | 62.3 | 48.5 | 
+| gpt-4 | 69.5 | 81.0 | 74.8 | 
+| Yandex.Speller | 83.0 | 59.8 | 69.5 |
+| JamSpell | 42.1 | 32.8 | 36.9 |
+| HunSpell | 31.3 | 34.9 | 33.0 |
+
+**MultidomainGold**
+| Model | Precision | Recall | F1 |
+| --- | --- | --- | --- |
+| sage-m2m100-1.2B | 63.8 | 61.1 | 62.4 |
+| sage-ai-service | 70.9 | 68.8 | 69.9 | 
+| gpt-3.5-turbo | 17.8 | 56.1 | 27.0 | 
+| gpt-4 | 31.1 | 78.1 | 44.5 |
+| Yandex.Speller | 52.9 | 51.4 | 52.2 |
+| JamSpell | 25.7 | 30.6 | 28.0 |
+| HunSpell | 16.2 | 40.1 | 23.0 |
+
+**MedSpellChecker**
+| Model | Precision | Recall | F1 |
+| --- | --- | --- | --- |
+| sage-m2m100-1.2B | 78.8 | 71.4 | 74.9 |
+| sage-ai-service | 73.4 | 76.2 | 74.9 | 
+| gpt-3.5-turbo | 15.1 | 53.6 | 23.5 | 
+| gpt-4 | 48.9 | 88.7 | 63.1 | 
+| Yandex.Speller | 80.6 | 47.8 | 60.0 |
+| JamSpell | 24.6 | 29.7 | 26.9 |
+| HunSpell | 10.3 | 40.2 | 16.4 |
+
+**GitHubTypoCorpusRu**
+| Model | Precision | Recall | F1 |
+| --- | --- | --- | --- |
+| sage-m2m100-1.2B | 47.1 | 42.9 | 44.9 |
+| sage-ai-service | 76.1 | 51.2 | 61.2 | 
+| gpt-3.5-turbo | 23.7 | 43.9 | 30.8 | 
+| gpt-4 | 34.7  | 60.5 | 44.1| 
+| Yandex.Speller | 67.7 | 37.5 | 48.3 |
+| JamSpell | 49.5 | 29.9 | 37.3 |
+| HunSpell | 28.5 | 30.7 | 29.6 |
+
+## How to use
+```python
+from transformers import M2M100ForConditionalGeneration, M2M100Tokenizer
+
+path_to_model = "ai-forever/sage-m2m100-1.2B"
+model = M2M100ForConditionalGeneration.from_pretrained(path_to_model)
+tokenizer = M2M100Tokenizer.from_pretrained(path_to_model, src_lang="ru", tgt_lang="ru")
+
+sentence = "прийдя в МГТУ я был удивлен никого необноружив там…"
+encodings = tokenizer(sentence, return_tensors="pt")
+generated_tokens = model.generate(
+        **encodings, forced_bos_token_id=tokenizer.get_lang_id("ru"))
+answer = tokenizer.batch_decode(generated_tokens, skip_special_tokens=True)
+
+print(answer)
+#["прийдя в МГТУ я был удивлен никого не обнаружив там..."]
+```
+
+## Resources
+- [SAGE library](https://github.com/ai-forever/sage), GitHub
+- [sage-fredt5-large](https://huggingface.co/ai-forever/sage-fredt5-large), HuggingFace
+- [sage-fredt5-distilled-95m](https://huggingface.co/ai-forever/sage-fredt5-distilled-95m), HuggingFace
+- [sage-m2m100-1.2B](https://huggingface.co/ai-forever/sage-m2m100-1.2B), HuggingFace
+- [sage-mt5-large](https://huggingface.co/ai-forever/sage-mt5-large), HuggingFace
+
+## License
+Model [M2M100-1.2B](https://huggingface.co/facebook/m2m100_1.2B), on the basis of which our solution is made, and its source code are supplied under the MIT open license. 
+Our solution also comes with MIT license.
+
+## Specifications
+- File size: 5 Gb;
+- Framework: pytorch
+- Format: AI Service
+- Version: v2.0
+- Developer: SberDevices, AGI NLP
+
+## Contacts
+nikita.martynov.98@list.ru