rabindralamsal commited on
Commit
96e9002
1 Parent(s): 126bf22

Create README.md

Browse files
Files changed (1) hide show
  1. README.md +50 -0
README.md ADDED
@@ -0,0 +1,50 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # CrisisTransformers
2
+ CrisisTransformers is a family of pre-trained language models and sentence encoders introduced in the paper "[CrisisTransformers: Pre-trained language models and sentence encoders for crisis-related social media texts](https://arxiv.org/abs/2309.05494)". The models were trained based on the RoBERTa pre-training procedure on a massive corpus of over 15 billion word tokens sourced from tweets associated with 30+ crisis events such as disease outbreaks, natural disasters, conflicts, etc. Please refer to the associated paper for more details.
3
+
4
+ CrisisTransformers were evaluated on 18 public crisis-specific datasets against strong baselines such as BERT, RoBERTa, BERTweet, etc. Our pre-trained models outperform the baselines across all 18 datasets in classification tasks, and our best-performing sentence-encoder outperforms the state-of-the-art by more than 17\% in sentence encoding tasks.
5
+
6
+ ## Uses
7
+ CrisisTransformers has 8 pre-trained models and a sentence encoder. The pre-trained models should be finetuned for downstream tasks just like [BERT](https://huggingface.co/bert-base-cased) and [RoBERTa](https://huggingface.co/roberta-base). The sentence encoder can be used out-of-the-box just like [Sentence-Transformers](https://huggingface.co/sentence-transformers/all-mpnet-base-v2) for sentence encoding to facilitate tasks such as semantic search, clustering, topic modelling.
8
+
9
+ ## Models and naming conventions
10
+ *CT-M1* models were trained from scratch up to 40 epochs, while *CT-M2* models were initialized with pre-trained RoBERTa's weights and *CT-M3* models were initialized with pre-trained BERTweet's weights and both trained for up to 20 epochs. *OneLook* represents the checkpoint after 1 epoch, *BestLoss* represents the checkpoint with the lowest loss during training, and *Complete* represents the checkpoint after completing all epochs. SE represents sentence encoder.
11
+
12
+ | pre-trained model | source |
13
+ |--|--|
14
+ |CT-M1-BestLoss|[crisistransformers/CT-M1-BestLoss](https://huggingface.co/crisistransformers/CT-M1-BestLoss)|
15
+ |CT-M1-Complete|[crisistransformers/CT-M1-Complete](https://huggingface.co/crisistransformers/CT-M1-Complete)|
16
+ |CT-M2-OneLook|[crisistransformers/CT-M2-OneLook](https://huggingface.co/crisistransformers/CT-M2-OneLook)|
17
+ |CT-M2-BestLoss|[crisistransformers/CT-M2-BestLoss](https://huggingface.co/crisistransformers/CT-M2-BestLoss)|
18
+ |CT-M2-Complete|[crisistransformers/CT-M2-Complete](https://huggingface.co/crisistransformers/CT-M2-Complete)|
19
+ |CT-M3-OneLook|[crisistransformers/CT-M3-OneLook](https://huggingface.co/crisistransformers/CT-M3-OneLook)|
20
+ |CT-M3-BestLoss|[crisistransformers/CT-M3-BestLoss](https://huggingface.co/crisistransformers/CT-M3-BestLoss)|
21
+ |CT-M3-Complete|[crisistransformers/CT-M3-Complete](https://huggingface.co/crisistransformers/CT-M3-Complete)|
22
+
23
+
24
+ | sentence encoder | source |
25
+ |--|--|
26
+ |CT-M1-Complete-SE|[crisistransformers/CT-M1-Complete-SE](https://huggingface.co/crisistransformers/CT-M1-Complete-SE)|
27
+
28
+
29
+ ## Results
30
+ Here are the main results from the associated paper.
31
+
32
+ <p float="left">
33
+ <img width="80%" alt="classification" src="https://raw.githubusercontent.com/rabindralamsal/images/main/cls.png" />
34
+ <img width="55%" alt="sentence encoding" src="https://raw.githubusercontent.com/rabindralamsal/images/main/se.png" />
35
+ </p>
36
+
37
+ ## Citation
38
+ If you use CrisisTransformers, please cite the following paper:
39
+ ```
40
+ @misc{lamsal2023crisistransformers,
41
+ title={CrisisTransformers: Pre-trained language models and sentence encoders for crisis-related social media texts},
42
+ author={Rabindra Lamsal and
43
+ Maria Rodriguez Read and
44
+ Shanika Karunasekera},
45
+ year={2023},
46
+ eprint={2309.05494},
47
+ archivePrefix={arXiv},
48
+ primaryClass={cs.CL}
49
+ }
50
+ ```