g8a9 commited on
Commit
d4e693c
1 Parent(s): ab9c651

Create README.md

Browse files
Files changed (1) hide show
  1. README.md +71 -0
README.md ADDED
@@ -0,0 +1,71 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language:
3
+ - en
4
+ license: gpl-3.0
5
+ tags:
6
+ - misogyny detection
7
+ - abusive language
8
+ - hate speech
9
+ - offensive language
10
+ widget:
11
+ - text: I believe religious minorities need to be protected more.
12
+ example_title: Hate Speech Detection Example 1
13
+ pipeline_tag: text-classification
14
+ datasets:
15
+ - nedjmaou/MLMA_hate_speech
16
+ ---
17
+
18
+ # Entropy-based Attention Regularization 👂
19
+
20
+ This is an English BERT fine-tuned with [Entropy-based Attention Regularization](https://aclanthology.org/2022.findings-acl.88/) to reduce lexical overfitting to specific words on the task of Misogyny Identification.
21
+ Use this model if you want a debiased alternative to a BERT classifier.
22
+
23
+ Please refer to the paper to know all the training details.
24
+
25
+ ## Model
26
+
27
+ This model is the fine-tuned version of the [bert-base-uncased](https://huggingface.co/bert-base-uncased) model.
28
+ We trained a total of three versions for Italian and English.
29
+
30
+ | Model | Download |
31
+ | ------ | -------------------------|
32
+ | `bert-base-uncased-ear-misogyny` | [Link](https://huggingface.co/MilaNLProc/bert-base-uncased-ear-misogyny) |
33
+ | `bert-base-uncased-ear-mlma` | [Link]() |
34
+ | `bert-base-uncased-ear-misogyny-italian` | [Link](https://huggingface.co/MilaNLProc/bert-base-uncased-ear-misogyny-italian) |
35
+
36
+ # Authors
37
+ - [Giuseppe Attanasio](https://gattanasio.cc/)
38
+ - [Debora Nozza](http://dnozza.github.io/)
39
+ - [Dirk Hovy](https://federicobianchi.io/)
40
+ - [Elena Baralis](https://dbdmg.polito.it/wordpress/people/elena-baralis/)
41
+
42
+ # Citation
43
+
44
+ Please use the following BibTeX entry if you use this model in your project:
45
+
46
+ ```
47
+ @inproceedings{attanasio-etal-2022-entropy,
48
+ title = "Entropy-based Attention Regularization Frees Unintended Bias Mitigation from Lists",
49
+ author = "Attanasio, Giuseppe and
50
+ Nozza, Debora and
51
+ Hovy, Dirk and
52
+ Baralis, Elena",
53
+ booktitle = "Findings of the Association for Computational Linguistics: ACL 2022",
54
+ month = may,
55
+ year = "2022",
56
+ address = "Dublin, Ireland",
57
+ publisher = "Association for Computational Linguistics",
58
+ url = "https://aclanthology.org/2022.findings-acl.88",
59
+ doi = "10.18653/v1/2022.findings-acl.88",
60
+ pages = "1105--1119",
61
+ abstract = "Natural Language Processing (NLP) models risk overfitting to specific terms in the training data, thereby reducing their performance, fairness, and generalizability. E.g., neural hate speech detection models are strongly influenced by identity terms like gay, or women, resulting in false positives, severe unintended bias, and lower performance.Most mitigation techniques use lists of identity terms or samples from the target domain during training. However, this approach requires a-priori knowledge and introduces further bias if important terms are neglected.Instead, we propose a knowledge-free Entropy-based Attention Regularization (EAR) to discourage overfitting to training-specific terms. An additional objective function penalizes tokens with low self-attention entropy.We fine-tune BERT via EAR: the resulting model matches or exceeds state-of-the-art performance for hate speech classification and bias metrics on three benchmark corpora in English and Italian.EAR also reveals overfitting terms, i.e., terms most likely to induce bias, to help identify their effect on the model, task, and predictions.",
62
+ }
63
+ ```
64
+
65
+ # Limitations
66
+
67
+ Entropy-Attention Regularization mitigates lexical overfitting but does not completely remove it. We expect the model still to show biases, e.g., peculiar keywords that induce a specific prediction regardless of the context. \
68
+ Please refer to our paper for a quantitative evaluation of this mitigation.
69
+
70
+ ## License
71
+ [GNU GPLv3](https://choosealicense.com/licenses/gpl-3.0/)