Update README.md
Browse files
README.md
CHANGED
@@ -1 +1,97 @@
|
|
1 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
---
|
2 |
+
language: "english"
|
3 |
+
tags:
|
4 |
+
- race
|
5 |
+
- multiple choice
|
6 |
+
license: "any valid license identifier"
|
7 |
+
datasets:
|
8 |
+
- RACE
|
9 |
+
metrics:
|
10 |
+
- accuracy
|
11 |
+
---
|
12 |
+
|
13 |
+
# Roberta Large Fine Tuned on RACE
|
14 |
+
|
15 |
+
## Model description
|
16 |
+
|
17 |
+
This model is a fine-tuned model of Roberta-large applied on RACE
|
18 |
+
|
19 |
+
#### How to use
|
20 |
+
|
21 |
+
```pythonimport datasets
|
22 |
+
from transformers import RobertaTokenizer, RobertaForMultipleChoice
|
23 |
+
tokenizer = RobertaTokenizer.from_pretrained('LIAMF-USP/roberta-large-finetuned-race')
|
24 |
+
model = RobertaForMultipleChoice.from_pretrained("LIAMF-USP/roberta-large-finetuned-race")
|
25 |
+
dataset = datasets.load_dataset(
|
26 |
+
"race",
|
27 |
+
"all",
|
28 |
+
split=["train", "validation", "test"],
|
29 |
+
)training_examples = dataset[0]
|
30 |
+
evaluation_examples = dataset[1]
|
31 |
+
test_examples = dataset[2]example=training_examples[0] example_id = example["example_id"]question = example["question"]
|
32 |
+
context = example["article"]
|
33 |
+
options = example["options"]
|
34 |
+
label_example = example["answer"]
|
35 |
+
label_map = {label: i for i, label in enumerate(["A", "B", "C", "D"])}
|
36 |
+
choices_inputs = []
|
37 |
+
for ending_idx, (_, ending) in enumerate(zip(context, options)):
|
38 |
+
if question.find("_") != -1:
|
39 |
+
# fill in the banks questions
|
40 |
+
question_option = question.replace("_", ending)
|
41 |
+
else:
|
42 |
+
question_option = question + " " + ending
|
43 |
+
inputs = tokenizer(
|
44 |
+
context,
|
45 |
+
question_option,
|
46 |
+
add_special_tokens=True,
|
47 |
+
max_length=MAX_SEQ_LENGTH,
|
48 |
+
padding="max_length",
|
49 |
+
truncation=True,
|
50 |
+
return_overflowing_tokens=False,
|
51 |
+
)
|
52 |
+
label = label_map[label_example]
|
53 |
+
input_ids = [x["input_ids"] for x in choices_inputs]
|
54 |
+
attention_mask = (
|
55 |
+
[x["attention_mask"] for x in choices_inputs]
|
56 |
+
# as the senteces follow the same structure, just one of them is
|
57 |
+
# necessary to check
|
58 |
+
if "attention_mask" in choices_inputs[0]
|
59 |
+
else None
|
60 |
+
)
|
61 |
+
example_encoded = {
|
62 |
+
"example_id": example_id,
|
63 |
+
"input_ids": input_ids,
|
64 |
+
"attention_mask": attention_mask,
|
65 |
+
"label": label,
|
66 |
+
}
|
67 |
+
output = model(**example_encoded)
|
68 |
+
```
|
69 |
+
|
70 |
+
|
71 |
+
## Training data
|
72 |
+
|
73 |
+
The initial model was [roberta large model](https://huggingface.co/roberta-large) which was then fine-tuned on [RACE dataset](https://www.cs.cmu.edu/~glai1/data/race/)
|
74 |
+
|
75 |
+
## Training procedure
|
76 |
+
|
77 |
+
It was necessary to preprocess the data with a method that is exemplified for a single instance in the _How to use_ section. The used hyperparameters were the following:
|
78 |
+
|
79 |
+
| Hyperparameter | Value |
|
80 |
+
|:----:|:----:|
|
81 |
+
| adam_beta1 | 0.9 |
|
82 |
+
| adam_beta2 | 0.98 |
|
83 |
+
| adam_epsilon | 1.000e-8 |
|
84 |
+
| eval_batch_size | 32 |
|
85 |
+
| train_batch_size | 1 |
|
86 |
+
| fp16 | True |
|
87 |
+
| gradient_accumulation_steps | 16 |
|
88 |
+
| learning_rate | 0.00001 |
|
89 |
+
| warmup_steps | 1000 |
|
90 |
+
| max_length | 512 |
|
91 |
+
|
92 |
+
## Eval results:
|
93 |
+
| Dataset Acc | Eval | All Test |High School Test |Middle School Test |
|
94 |
+
|:----:|:----:|:----:|:----:|:----:|
|
95 |
+
| | 95.2 | 84.9|83.5|88.0|
|
96 |
+
|
97 |
+
**The model was trained with a Tesla V100-PCIE-16GB**
|