File size: 9,288 Bytes
a708ccc
34bc863
a708ccc
35cdaef
a708ccc
 
 
 
 
4954f85
 
 
 
b717ad8
34bc863
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
3a9a7f3
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
c48faaa
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
6393711
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
a708ccc
 
 
 
f2c795e
a708ccc
35cdaef
 
3a12891
 
 
 
 
e5350ef
 
3a12891
 
 
 
 
a708ccc
 
 
35cdaef
f2c795e
a708ccc
 
 
f2c795e
 
 
 
 
 
a708ccc
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
267172a
4b15c4c
267172a
 
 
c72e58a
a708ccc
 
 
35cdaef
 
3f00ab8
 
 
 
 
35cdaef
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
---
language:
- en
license: mit
tags:
- text-classification
- zero-shot-classification
metrics:
- accuracy
datasets:
- multi_nli
- anli
- fever
pipeline_tag: zero-shot-classification
model-index:
- name: MoritzLaurer/DeBERTa-v3-base-mnli-fever-anli
  results:
  - task:
      type: natural-language-inference
      name: Natural Language Inference
    dataset:
      name: anli
      type: anli
      config: plain_text
      split: test_r3
    metrics:
    - name: Accuracy
      type: accuracy
      value: 0.495
      verified: true
    - name: Precision Macro
      type: precision
      value: 0.4984740618243923
      verified: true
    - name: Precision Micro
      type: precision
      value: 0.495
      verified: true
    - name: Precision Weighted
      type: precision
      value: 0.4984357572868885
      verified: true
    - name: Recall Macro
      type: recall
      value: 0.49461028192371476
      verified: true
    - name: Recall Micro
      type: recall
      value: 0.495
      verified: true
    - name: Recall Weighted
      type: recall
      value: 0.495
      verified: true
    - name: F1 Macro
      type: f1
      value: 0.4942810999491704
      verified: true
    - name: F1 Micro
      type: f1
      value: 0.495
      verified: true
    - name: F1 Weighted
      type: f1
      value: 0.4944671868893595
      verified: true
    - name: loss
      type: loss
      value: 1.8788293600082397
      verified: true
  - task:
      type: natural-language-inference
      name: Natural Language Inference
    dataset:
      name: anli
      type: anli
      config: plain_text
      split: test_r1
    metrics:
    - name: Accuracy
      type: accuracy
      value: 0.712
      verified: true
    - name: Precision Macro
      type: precision
      value: 0.7134839439315348
      verified: true
    - name: Precision Micro
      type: precision
      value: 0.712
      verified: true
    - name: Precision Weighted
      type: precision
      value: 0.7134676028447461
      verified: true
    - name: Recall Macro
      type: recall
      value: 0.7119814425203647
      verified: true
    - name: Recall Micro
      type: recall
      value: 0.712
      verified: true
    - name: Recall Weighted
      type: recall
      value: 0.712
      verified: true
    - name: F1 Macro
      type: f1
      value: 0.7119226991285647
      verified: true
    - name: F1 Micro
      type: f1
      value: 0.712
      verified: true
    - name: F1 Weighted
      type: f1
      value: 0.7119242267218338
      verified: true
    - name: loss
      type: loss
      value: 1.0105403661727905
      verified: true
  - task:
      type: natural-language-inference
      name: Natural Language Inference
    dataset:
      name: multi_nli
      type: multi_nli
      config: default
      split: validation_mismatched
    metrics:
    - name: Accuracy
      type: accuracy
      value: 0.902766476810415
      verified: true
    - name: Precision Macro
      type: precision
      value: 0.9023816542652491
      verified: true
    - name: Precision Micro
      type: precision
      value: 0.902766476810415
      verified: true
    - name: Precision Weighted
      type: precision
      value: 0.9034597464719761
      verified: true
    - name: Recall Macro
      type: recall
      value: 0.9024304801555488
      verified: true
    - name: Recall Micro
      type: recall
      value: 0.902766476810415
      verified: true
    - name: Recall Weighted
      type: recall
      value: 0.902766476810415
      verified: true
    - name: F1 Macro
      type: f1
      value: 0.9023086094638595
      verified: true
    - name: F1 Micro
      type: f1
      value: 0.902766476810415
      verified: true
    - name: F1 Weighted
      type: f1
      value: 0.9030161011457231
      verified: true
    - name: loss
      type: loss
      value: 0.3283354640007019
      verified: true
  - task:
      type: natural-language-inference
      name: Natural Language Inference
    dataset:
      name: anli
      type: anli
      config: plain_text
      split: dev_r1
    metrics:
    - name: Accuracy
      type: accuracy
      value: 0.737
      verified: true
    - name: Precision Macro
      type: precision
      value: 0.737681071614645
      verified: true
    - name: Precision Micro
      type: precision
      value: 0.737
      verified: true
    - name: Precision Weighted
      type: precision
      value: 0.7376755842752241
      verified: true
    - name: Recall Macro
      type: recall
      value: 0.7369675064285843
      verified: true
    - name: Recall Micro
      type: recall
      value: 0.737
      verified: true
    - name: Recall Weighted
      type: recall
      value: 0.737
      verified: true
    - name: F1 Macro
      type: f1
      value: 0.7366853496239583
      verified: true
    - name: F1 Micro
      type: f1
      value: 0.737
      verified: true
    - name: F1 Weighted
      type: f1
      value: 0.7366990292378379
      verified: true
    - name: loss
      type: loss
      value: 0.9349392056465149
      verified: true
---
# DeBERTa-v3-base-mnli-fever-anli
## Model description
This model was trained on the MultiNLI, Fever-NLI and Adversarial-NLI (ANLI) datasets, which comprise 763 913 NLI hypothesis-premise pairs. This base model outperforms almost all large models on the [ANLI benchmark](https://github.com/facebookresearch/anli). 
The base model is [DeBERTa-v3-base from Microsoft](https://huggingface.co/microsoft/deberta-v3-base). The v3 variant of DeBERTa substantially outperforms previous versions of the model by including a different pre-training objective, see annex 11 of the original [DeBERTa paper](https://arxiv.org/pdf/2006.03654.pdf). 

For highest performance (but less speed), I recommend using https://huggingface.co/MoritzLaurer/DeBERTa-v3-large-mnli-fever-anli-ling-wanli.


### How to use the model
#### Simple zero-shot classification pipeline
```python
from transformers import pipeline
classifier = pipeline("zero-shot-classification", model="MoritzLaurer/DeBERTa-v3-base-mnli-fever-anli")
sequence_to_classify = "Angela Merkel is a politician in Germany and leader of the CDU"
candidate_labels = ["politics", "economy", "entertainment", "environment"]
output = classifier(sequence_to_classify, candidate_labels, multi_label=False)
print(output)
```
#### NLI use-case
```python
from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch
device = torch.device("cuda") if torch.cuda.is_available() else torch.device("cpu")

model_name = "MoritzLaurer/DeBERTa-v3-base-mnli-fever-anli"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSequenceClassification.from_pretrained(model_name)

premise = "I first thought that I liked the movie, but upon second thought it was actually disappointing."
hypothesis = "The movie was good."

input = tokenizer(premise, hypothesis, truncation=True, return_tensors="pt")
output = model(input["input_ids"].to(device))  # device = "cuda:0" or "cpu"
prediction = torch.softmax(output["logits"][0], -1).tolist()
label_names = ["entailment", "neutral", "contradiction"]
prediction = {name: round(float(pred) * 100, 1) for pred, name in zip(prediction, label_names)}
print(prediction)
```
### Training data
DeBERTa-v3-base-mnli-fever-anli was trained on the MultiNLI, Fever-NLI and Adversarial-NLI (ANLI) datasets, which comprise 763 913 NLI hypothesis-premise pairs.

### Training procedure
DeBERTa-v3-base-mnli-fever-anli was trained using the Hugging Face trainer with the following hyperparameters.
```
training_args = TrainingArguments(
    num_train_epochs=3,              # total number of training epochs
    learning_rate=2e-05,
    per_device_train_batch_size=32,   # batch size per device during training
    per_device_eval_batch_size=32,    # batch size for evaluation
    warmup_ratio=0.1,                # number of warmup steps for learning rate scheduler
    weight_decay=0.06,               # strength of weight decay
    fp16=True                        # mixed precision training
)
```
### Eval results
The model was evaluated using the test sets for MultiNLI and ANLI and the dev set for Fever-NLI. The metric used is accuracy.

mnli-m | mnli-mm | fever-nli | anli-all | anli-r3
---------|----------|---------|----------|----------
0.903 | 0.903 | 0.777 | 0.579 | 0.495

## Limitations and bias
Please consult the original DeBERTa paper and literature on different NLI datasets for potential biases. 

## Citation
If you use this model, please cite: Laurer, Moritz, Wouter van Atteveldt, Andreu Salleras Casas, and Kasper Welbers. 2022. ‘Less Annotating, More Classifying – Addressing the Data Scarcity Issue of Supervised Machine Learning with Deep Transfer Learning and BERT - NLI’. Preprint, June. Open Science Framework. https://osf.io/74b8k.

### Ideas for cooperation or questions?
If you have questions or ideas for cooperation, contact me at m{dot}laurer{at}vu{dot}nl or [LinkedIn](https://www.linkedin.com/in/moritz-laurer/)

### Debugging and issues
Note that DeBERTa-v3 was released on 06.12.21 and older versions of HF Transformers seem to have issues running the model (e.g. resulting in an issue with the tokenizer). Using Transformers>=4.13 might solve some issues.