SaveBertAndGpt commited on
Commit
356b8f5
1 Parent(s): 72ecaf9

Create README.md

Browse files
Files changed (1) hide show
  1. README.md +92 -0
README.md ADDED
@@ -0,0 +1,92 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language: "en" # Example: en
3
+ license: "cc-by-4.0" # Example: apache-2.0 or any license from https://hf.co/docs/hub/repositories-licenses
4
+ library_name: "transformers" # Optional. Example: keras or any library from https://github.com/huggingface/hub-docs/blob/main/js/src/lib/interfaces/Libraries.ts
5
+ ---
6
+ # Model description
7
+ This is the T5-3B model for System 2 as described in our paper Just-DREAM-about-it: Figurative Language Understanding with DREAM-FLUTE, FigLang workshop @ EMNLP 2022 (Arxiv link: TBD)
8
+
9
+ System 2: Jointly predicting the type of figurative language
10
+
11
+ Using type of figurative language provided as part of the training set (Chakrabarty et al., 2022), one of our models jointly predicts the type of figurative language, together with the target label and explanation:
12
+ ```
13
+ Input <Premise> <Hypothesis>
14
+ Output <Figurative-Language-Type> <Label> <Explanation>
15
+ ```
16
+
17
+ # How to use this model?
18
+ We provide a quick example of how you can try out System 2 in our paper with just a few lines of code:
19
+ ```
20
+ >>> from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
21
+ >>> model = AutoModelForSeq2SeqLM.from_pretrained("allenai/System2_FigLang2022")
22
+
23
+ >>> tokenizer = AutoTokenizer.from_pretrained("t5-3b")
24
+ >>> input_string = "Premise: Yesterday two gangs were fighting just in front of my home. Hypothesis: Yesterday I saw two gangs fighting right in front of my house and it totally didn't make me scared at all. What is the type of figurative language involved? Is there a contradiction or entailment between the premise and hypothesis?"
25
+ >>> input_ids = tokenizer.encode(input_string, return_tensors="pt")
26
+ >>> output = model.generate(input_ids, max_length=200)
27
+ >>> tokenizer.batch_decode(output, skip_special_tokens=True)
28
+ ['Answer : [Type] Sarcasm [Label] Contradiction. Explanation : Seeing two gangs of people fighting in public can be really dangerous and scary, so someone who claims that they were not scared at all is being sarcastic.']
29
+ ```
30
+
31
+ # More details about DREAM-FLUTE ...
32
+ For more details about DREAM-FLUTE, please refer to our:
33
+ * 📄Paper: (Arxiv link: TBD)
34
+ * 💻GitHub Repo: https://github.com/allenai/dream/
35
+
36
+ This model is part of our DREAM-series of works. This is a line of research where we make use of scene elaboration for building a "mental model" of situation given in text. Check out our GitHub Repo for more!
37
+
38
+ # More details about this model ...
39
+ ## Training and evaluation data
40
+
41
+ We use the FLUTE dataset for the FigLang2022SharedTask (https://huggingface.co/datasets/ColumbiaNLP/FLUTE) for training this model. ∼7500 samples are provided as the training set. We used a 80-20 split to create our own training (6027 samples) and validation (1507 samples) partitions on which we build our models. For details on how we make use of the training data provided in the FigLang2022 shared task, please refer to https://github.com/allenai/dream/blob/main/FigLang2022SharedTask/Process_Data_Train_Dev_split.ipynb.
42
+
43
+ ## Model details
44
+
45
+ This model is a fine-tuned version of [t5-3b](https://huggingface.co/t5-3b).
46
+
47
+ It achieves the following results on the evaluation set:
48
+ - Loss: 0.6078
49
+ - Rouge1: 62.8674
50
+ - Rouge2: 45.0585
51
+ - Rougel: 57.5618
52
+ - Rougelsum: 57.5172
53
+ - Gen Len: 50.7558
54
+
55
+ ## Training procedure
56
+
57
+ ### Training hyperparameters
58
+
59
+ The following hyperparameters were used during training:
60
+ - learning_rate: 5e-05
61
+ - train_batch_size: 1
62
+ - eval_batch_size: 1
63
+ - seed: 42
64
+ - distributed_type: multi-GPU
65
+ - num_devices: 2
66
+ - total_train_batch_size: 2
67
+ - total_eval_batch_size: 2
68
+ - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
69
+ - lr_scheduler_type: linear
70
+ - num_epochs: 3.0
71
+
72
+ ### Training results
73
+
74
+ | Training Loss | Epoch | Step | Validation Loss | Rouge1 | Rouge2 | Rougel | Rougelsum | Gen Len |
75
+ |:-------------:|:-----:|:----:|:---------------:|:-------:|:-------:|:-------:|:---------:|:-------:|
76
+ | 0.8068 | 0.33 | 1000 | 0.7251 | 30.6353 | 25.0792 | 30.619 | 30.6274 | 19.0 |
77
+ | 0.7276 | 0.66 | 2000 | 0.6715 | 30.8651 | 26.1492 | 30.8543 | 30.8519 | 19.0 |
78
+ | 0.7063 | 1.0 | 3000 | 0.6338 | 31.0263 | 26.6749 | 31.0094 | 31.0098 | 19.0 |
79
+ | 0.4516 | 1.33 | 4000 | 0.6447 | 30.9942 | 26.5984 | 30.9834 | 30.9778 | 19.0 |
80
+ | 0.4538 | 1.66 | 5000 | 0.6183 | 31.0179 | 26.7012 | 31.005 | 31.0018 | 19.0 |
81
+ | 0.4373 | 1.99 | 6000 | 0.6078 | 31.0085 | 26.7116 | 30.9952 | 30.9894 | 19.0 |
82
+ | 0.2743 | 2.32 | 7000 | 0.6910 | 31.0051 | 26.7349 | 30.9975 | 30.9851 | 19.0 |
83
+ | 0.2819 | 2.65 | 8000 | 0.6831 | 31.0876 | 26.848 | 31.0766 | 31.0753 | 19.0 |
84
+ | 0.2849 | 2.99 | 9000 | 0.6673 | 30.9223 | 26.5899 | 30.9165 | 30.9073 | 19.0 |
85
+
86
+
87
+ ### Framework versions
88
+
89
+ - Transformers 4.22.0.dev0
90
+ - Pytorch 1.12.1+cu113
91
+ - Datasets 2.4.0
92
+ - Tokenizers 0.12.1