Commit
•
3000577
1
Parent(s):
f3b1f58
Update README.md
Browse files
README.md
CHANGED
|
@@ -12,12 +12,11 @@ widget:
|
|
| 12 |
candidate_labels: "спорт,путешествия,музыка,кино,книги,наука,политика"
|
| 13 |
hypothesis_template: "Тема текста - {}."
|
| 14 |
---
|
| 15 |
-
# RuBERT
|
| 16 |
-
The model has been trained on a series of NLI datasets automatically translated to Russian from English [from this repo](https://github.com/felipessalvatore/NLI_datasets).
|
| 17 |
-
|
| 18 |
-
It predicts the logical relationship between two short texts: entailment, contradiction, or neutral.
|
| 19 |
|
|
|
|
| 20 |
|
|
|
|
| 21 |
How to run the model for NLI:
|
| 22 |
```python
|
| 23 |
# !pip install transformers sentencepiece --quiet
|
|
@@ -59,4 +58,46 @@ predict_zero_shot('Какая вкусная эта ваша заливная р
|
|
| 59 |
# array([0.9059292 , 0.09407079], dtype=float32)
|
| 60 |
```
|
| 61 |
|
| 62 |
-
Alternatively, you can use [Huggingface pipelines](https://huggingface.co/transformers/main_classes/pipelines.html) for inference.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 12 |
candidate_labels: "спорт,путешествия,музыка,кино,книги,наука,политика"
|
| 13 |
hypothesis_template: "Тема текста - {}."
|
| 14 |
---
|
| 15 |
+
# RuBERT for NLI (natural language inference)
|
|
|
|
|
|
|
|
|
|
| 16 |
|
| 17 |
+
This is the [DeepPavlov/rubert-base-cased](https://huggingface.co/DeepPavlov/rubert-base-cased) fine-tuned to predict the logical relationship between two short texts: entailment, contradiction, or neutral.
|
| 18 |
|
| 19 |
+
## Usage
|
| 20 |
How to run the model for NLI:
|
| 21 |
```python
|
| 22 |
# !pip install transformers sentencepiece --quiet
|
| 58 |
# array([0.9059292 , 0.09407079], dtype=float32)
|
| 59 |
```
|
| 60 |
|
| 61 |
+
Alternatively, you can use [Huggingface pipelines](https://huggingface.co/transformers/main_classes/pipelines.html) for inference.
|
| 62 |
+
|
| 63 |
+
## Sources
|
| 64 |
+
The model has been trained on a series of NLI datasets automatically translated to Russian from English.
|
| 65 |
+
|
| 66 |
+
Most datasets were taken [from the repo of Felipe Salvatore](https://github.com/felipessalvatore/NLI_datasets):
|
| 67 |
+
[JOCI](https://github.com/sheng-z/JOCI),
|
| 68 |
+
[MNLI](https://cims.nyu.edu/~sbowman/multinli/),
|
| 69 |
+
[MPE](https://aclanthology.org/I17-1011/),
|
| 70 |
+
[SICK](http://www.lrec-conf.org/proceedings/lrec2014/pdf/363_Paper.pdf),
|
| 71 |
+
[SNLI](https://nlp.stanford.edu/projects/snli/).
|
| 72 |
+
|
| 73 |
+
Some datasets obtained from the original sources:
|
| 74 |
+
[ANLI](https://github.com/facebookresearch/anli),
|
| 75 |
+
[NLI-style FEVER](https://github.com/easonnie/combine-FEVER-NSMN/blob/master/other_resources/nli_fever.md),
|
| 76 |
+
[IMPPRES](https://github.com/facebookresearch/Imppres).
|
| 77 |
+
|
| 78 |
+
## Performance
|
| 79 |
+
|
| 80 |
+
The table below shows ROC AUC for three models on small samples of the DEV sets:
|
| 81 |
+
- [tiny](https://huggingface.co/cointegrated/rubert-tiny-bilingual-nli): a small BERT predicting entailment vs not_entailment
|
| 82 |
+
- [twoway](https://huggingface.co/cointegrated/rubert-base-cased-nli-twoway): a base-sized BERT predicting entailment vs not_entailment
|
| 83 |
+
- [threeway](https://huggingface.co/cointegrated/rubert-base-cased-nli-threeway) (**this model**): a base-sized BERT predicting entailment vs contradiction vs neutral
|
| 84 |
+
|
| 85 |
+
|model |tiny/entailment|twoway/entailment|threeway/entailment|threeway[3]/contradiction|threeway[3]/neutral|
|
| 86 |
+
|-----------|---------------|-----------------|-------------------|-------------------------|-------------------|
|
| 87 |
+
|add_one_rte|0.82 |0.90 |0.92 | | |
|
| 88 |
+
|anli_r1 |0.50 |0.68 |0.66 |0.70 |0.75 |
|
| 89 |
+
|anli_r2 |0.55 |0.62 |0.62 |0.62 |0.69 |
|
| 90 |
+
|anli_r3 |0.50 |0.63 |0.59 |0.62 |0.64 |
|
| 91 |
+
|copa |0.55 |0.60 |0.62 | | |
|
| 92 |
+
|fever |0.88 |0.94 |0.94 |0.91 |0.92 |
|
| 93 |
+
|help |0.74 |0.87 |0.46 | | |
|
| 94 |
+
|iie |0.79 |0.85 |0.54 | | |
|
| 95 |
+
|imppres |0.94 |0.99 |0.99 |0.99 |0.99 |
|
| 96 |
+
|joci |0.87 |0.93 |0.93 |0.85 |0.80 |
|
| 97 |
+
|mnli |0.87 |0.92 |0.93 |0.89 |0.86 |
|
| 98 |
+
|monli |0.94 |1.00 |0.67 | | |
|
| 99 |
+
|mpe |0.82 |0.90 |0.90 |0.91 |0.80 |
|
| 100 |
+
|scitail |0.80 |0.96 |0.85 | | |
|
| 101 |
+
|sick |0.97 |0.99 |0.99 |0.98 |0.96 |
|
| 102 |
+
|snli |0.95 |0.98 |0.98 |0.99 |0.97 |
|
| 103 |
+
|terra |0.73 |0.93 |0.93 | | |
|