first model aristo-roberta done

README.md

@@ -0,0 +1,131 @@
+---
+language: "english"
+tags:
+license: "mit"
+datasets:
+- race
+- arc
+metrics:
+- accuracy
+---
+
+# Roberta Large Fine Tuned on RACE
+
+## Model description
+
+This model follows the implementation by Allen AI team about [Aristo Roberta V7 Model](https://leaderboard.allenai.org/arc/submission/blcotvl7rrltlue6bsv0) given in [ARC Challenge](https://leaderboard.allenai.org/arc/submissions/public)
+
+#### How to use
+
+```python
+
+import datasets
+from transformers import RobertaTokenizer
+from transformers import  RobertaForMultipleChoice
+
+tokenizer = RobertaTokenizer.from_pretrained(
+"LIAMF-USP/aristo-roberta")
+model = RobertaForMultipleChoice.from_pretrained(
+"LIAMF-USP/aristo-roberta")
+dataset = datasets.load_dataset(
+    "arc",,
+    split=["train", "validation", "test"],
+)
+training_examples = dataset[0]
+evaluation_examples = dataset[1]
+test_examples = dataset[2]
+
+example=training_examples[0] 
+example_id = example["example_id"]
+question = example["question"]
+label_example = example["answer"]
+options = example["options"]
+if label_example in ["A", "B", "C", "D", "E"]:
+    label_map = {label: i for i, label in enumerate(
+					["A", "B", "C", "D", "E"])}
+elif label_example in ["1", "2", "3", "4", "5"]:
+    label_map = {label: i for i, label in enumerate(
+					["1", "2", "3", "4", "5"])}
+else:
+    print(f"{label_example} not found")
+while len(options) < 5:
+    empty_option = {}
+    empty_option['option_context'] = ''
+    empty_option['option_text'] = ''
+    options.append(empty_option)
+choices_inputs = []
+for ending_idx, option in enumerate(options):
+    ending = option["option_text"]
+    context = option["option_context"]
+    if question.find("_") != -1:
+        # fill in the banks questions
+        question_option = question.replace("_", ending)
+    else:
+        question_option = question + " " + ending
+    
+    inputs = tokenizer(
+        context,
+        question_option,
+        add_special_tokens=True,
+        max_length=MAX_SEQ_LENGTH,
+        padding="max_length",
+        truncation=True,
+        return_overflowing_tokens=False,
+    )
+    
+    if "num_truncated_tokens" in inputs and inputs["num_truncated_tokens"] > 0:
+        logging.warning(f"Question: {example_id} with option {ending_idx} was truncated")
+    choices_inputs.append(inputs)
+label = label_map[label_example]
+input_ids = [x["input_ids"] for x in choices_inputs]
+attention_mask = (
+    [x["attention_mask"] for x in choices_inputs]
+     # as the senteces follow the same structure, just one of them is
+     # necessary to check
+    if "attention_mask" in choices_inputs[0]
+    else None
+)
+example_encoded = {
+    "example_id": example_id,
+    "input_ids": input_ids,
+    "attention_mask": attention_mask,
+    "token_type_ids": token_type_ids,
+    "label": label
+
+}
+output = model(**example_encoded)
+```
+
+
+## Training data
+
+the Training data was the same as proposed [here](https://leaderboard.allenai.org/arc/submission/blcotvl7rrltlue6bsv0) 
+
+The only diferrence was the hypeparameters of RACE fine tuned model, which were reported [here](https://huggingface.co/LIAMF-USP/roberta-large-finetuned-race#eval-results)
+
+## Training procedure
+
+It was necessary to preprocess the data with a method that is exemplified for a single instance in the _How to use_ section. The used hyperparameters were the following:
+
+| Hyperparameter | Value |
+|:----:|:----:|
+| adam_beta1                  | 0.9      |
+| adam_beta2                  | 0.98     |
+| adam_epsilon                | 1.000e-8 |
+| eval_batch_size             | 16       |
+| train_batch_size            | 4        |
+| fp16                        | True     |
+| gradient_accumulation_steps | 4       |
+| learning_rate               | 0.00001  |
+| warmup_steps                | 67     |
+| max_length                  | 256      |
+| epochs                  |   2      |
+
+The other parameters were the default ones from [Trainer](https://huggingface.co/transformers/main_classes/trainer.html) and [Trainer Arguments](https://huggingface.co/transformers/main_classes/trainer.html#trainingarguments)
+
+## Eval results:
+| Dataset Acc | Challenge Test |
+|:----:|:----:|
+|      | 64.249 |
+
+**The model was trained with a TITAN RTX** 

config.json

@@ -0,0 +1,25 @@
+{
+  "_name_or_path": "/root/masters-project/aristo-roberta",
+  "architectures": [
+    "RobertaForMultipleChoice"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "bos_token_id": 0,
+  "eos_token_id": 2,
+  "gradient_checkpointing": false,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 1024,
+  "initializer_range": 0.02,
+  "intermediate_size": 4096,
+  "layer_norm_eps": 1e-05,
+  "max_position_embeddings": 514,
+  "model_type": "roberta",
+  "num_attention_heads": 16,
+  "num_hidden_layers": 24,
+  "pad_token_id": 1,
+  "position_embedding_type": "absolute",
+  "total_flos": 1502266556732252160,
+  "type_vocab_size": 1,
+  "vocab_size": 50265
+}

pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:84b4be015199610ee05fe79c12509eeda5b999ea150a398ab3958967bdf73416
+size 1421616585

special_tokens_map.json

@@ -0,0 +1 @@
+{"bos_token": {"content": "<s>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true}, "eos_token": {"content": "</s>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true}, "unk_token": {"content": "<unk>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true}, "sep_token": {"content": "</s>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true}, "pad_token": {"content": "<pad>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true}, "cls_token": {"content": "<s>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true}, "mask_token": {"content": "<mask>", "single_word": false, "lstrip": true, "rstrip": false, "normalized": true}}

tf_model.h5

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8a6b5172ebeb1677bd8be2a747a81601f497856f21e670b10e45d12d2c1a1f74
+size 1421961240

tokenizer_config.json

@@ -0,0 +1 @@
+{"unk_token": {"content": "<unk>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true, "__type": "AddedToken"}, "bos_token": {"content": "<s>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true, "__type": "AddedToken"}, "eos_token": {"content": "</s>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true, "__type": "AddedToken"}, "add_prefix_space": false, "errors": "replace", "sep_token": {"content": "</s>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true, "__type": "AddedToken"}, "cls_token": {"content": "<s>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true, "__type": "AddedToken"}, "pad_token": {"content": "<pad>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true, "__type": "AddedToken"}, "mask_token": {"content": "<mask>", "single_word": false, "lstrip": true, "rstrip": false, "normalized": true, "__type": "AddedToken"}, "model_max_length": 512, "special_tokens_map_file": "./results_model/bert-base-uncased/special_tokens_map.json", "name_or_path": "LIAMF-USP/roberta-large-finetuned-race"}

training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f16888460a9cb0ba9ef68f5eade7a345ea94798b2d91aeba8c7ca872ba77c421
+size 1839