SlowGuess/ACL_2020_OCR huggingface dataset:
BiLSTM with attention. This model, also used by (Parikh et al., 2019), relies on a Bidirectional LSTM with an attention mechanism that attends over all hidden states and generates attention coefficients. The hidden states were then averaged using the attention coefficients in order to generate the final state, which was then fed to a one-layer feed-forward network in order to obtain the final label prediction. We experimented with different hidden state vector sizes, dropout values and attention vector sizes. The results reported in this paper
were obtained by using 300 hidden units, an 150 attention vector, a dropout of  $50\%$  and the Adam optimizer with a learning rate of  $10^{-3}$ .
$\mathbf{BERT}_{\mathrm{base}}$ . It uses the pre-trained BERT model (BERT-Base, Multilingual Cased) (Devlin et al., 2019) on top of which we added an untrained layer of neurons. We then used the HuggingFace's PyTorch implementation of BERT (Wolf et al., 2019) that we trained for 3 epochs.
$\mathbf{BERT}^{R}$ . We observed that about  $47\%$  of the tweets embed at least one URL. Due to the short length of a tweet, this is useful for amplifying the message, while also minimizing the time it takes to compose it. In order to feed more information to the classifier, instead of removing or replacing the URLs with replacement tokens as usually done in hate speech detection, we propose to substitute them with the title found at the given  $\mathsf{URL}^{10}$ . In addition, and based on the assumption that word embeddings capture the meaning of words better than emoji embeddings capture the meaning of emojis, we followed the strategy proposed by (Singh et al., 2019) and replaced all the emojis with their detailed descriptions<sup>11</sup>. Replacing URLs and emojis improved the results for all the models we have tested, so we give here only the results obtained after these replacements.
$\mathbf{BERT}^{R}$  _own_emb + base. Following (Parikh et al., 2019), we also experiment stacking multiple embeddings. We tailored a pre-trained BERT model for which we used the whole non annotated dataset (i.e., 205,000 tweets). The original BERT model uses a WordPiece tokenizer, which is not available in OpenSource. Instead, we used a SentencePiece tokenizer in unigram mode. Training the model using the Google Cloud infrastructure with the default parameters for 1 million steps took approximately 3 days.
$\mathbf{BERT}^{R}$  features. We relied on state of the art features that have shown to be useful for the task of hate speech detection: Surface features (tweet length in words, the presence of personal
8We also experiment with standard feature-based models, but the results were lower.
<sup>9</sup>We also experimented with GRU following (Zhang and Luo, 2018), but the results were not conclusive.
10 In case a particular web page is not available anymore, the URL is removed from the tweet.
11We relied on a manually built emoji lexicon that contains 1,644 emojis along with their polarity and detailed description.
<sup>12</sup>We experimented with different configurations by incorporating different French pre-trained embeddings available: Glove (Pennington et al., 2014), FastText (Grave et al., 2018), Flair (Akbik et al., 2018) and CamemBERT (Martin et al., 2019) but none of the configurations were able to achieve results better than BERT<sub>base</sub>.
$^{13}$ https://github.com/google/sentencepiece
pronoun and third-person pronoun, punctuation marks, URLs, images, hashtags, @userMentions and the number of words written in capital), Emoji features $^{11}$  (number of positive and negative emojis), Opinion features (number of positive, negative and neutral words in each tweet relying on opinion (Benamara et al., 2014), emotion (Piolat and Bannour, 2009) and slang French lexicons. We also account for hedges (negation and modality), reporting verbs, imperative verbs, and verbs used for giving advice.
$\mathbf{BERT}^{R}$  gen. Sexism is often expressed by using gender stereotypes, i.e., ideas whereby women and men are arbitrarily assigned characteristics and roles determined and limited by their gender. In order to force the classifier to learn from generalized concept rather than words which may be rare in the corpus, we adopt several replacement combinations extending (Badjatiya et al., 2017)'s approach consisting in replacing some words/expressions that trigger sexist content by their generalized term. However, instead of using a flat list composed of most frequent words that appear in a particular class and then replace them by similarity relationships, we rather rely on manually built lists of words $^{14}$  often used in sexist language (hereafter <SexistVocabulary>): designations (around 10 words such as femme (woman), fille (girl), nana (doll), ...), insults (around 400 words/expressions extracted from GLAWI (Hathout and Sajous, 2016), a machine-readable French Dictionary); and 130 gender stereotyped words grouped according to the following taxonomy as usually defined in gender studies (see Section 2): physical characteristics (e.g. petite (little), bouche (mouth), robe (dress), ... for women; petit (little), gros (fat), ... for men), behavioural characteristics (e.g. bavarde (gossipy), jalouse (jealous), tendre (loving), ... for women; macho, viril (virile), ... for men), and type of activities (e.g. mère (mother), cuisine (cooking), infirmière (nurse), ... for women; football, médecin (doctor), ... for men). Only  $1\%$  of all these words have been used as keywords to collect the corpus.
In addition, we also built two other lists: names (952/832 female/male firstnames to detect named entities) and around 170 words/expressions for places as they are mainly useful for detection of reporting messages since they represent public spaces
where sexist acts may occur.(e.g. métro (subway), rue (street), bureau (office), ...).
We experimented with distinct generalization strategies: hypernym replacement gen(Hypernym) (e.g., little is replaced by <PhysicalCharacteristics>), gendered hypernym replacement gen(Hypernym_gendered) (e.g., dress is replaced by <femalePhysicalCharacteristics>) as well as generic replacement gen(SexistVocabulary) (e.g., both little and doll are replaced by the same tag <SexistVocabulary>), etc., where  $X$  in  $\mathbf{BERT}^{R}_{\mathbf{features} + \mathbf{X}}$  indicates the adopted replacement strategy.
6.1 BIN and 3-CLASS results
Table 2 presents the results for the best state of the art models for the task of sexism detection (CNN, BiLSTM with attention, CNN-LSTM) applied on the BIN task in terms of accuracy (A), macro-averaged F-score (F), precision (P) and recall (R) with the best results in bold. None of these models were able to achieve results better than  $\mathrm{BERT}_{\mathrm{base}}$ . For this reason, we chose  $\mathrm{BERT}_{\mathrm{base}}$  as our baseline and trained it on top of several vectorial representations, as explained in Section 5.
CLASSIFIER	A	F	P	R
CNN	0.684	0.601	0.635	0.571
CNN+LSTM	0.676	0.640	0.623	0.657
BiLSTMAttention	0.695	0.527	0.501	0.554
BERTbase	0.773	0.723	0.726	0.721
Table 2: Results for BIN classification.
As shown in Table 3, we observe that training BERT with stacked embeddings did not improve over  $\mathrm{BERT}_{\mathrm{base}}$ . Replacing URLs and emojis with respectively the words within the title link and emoji description boosts the results by  $1.7\%$  and  $1.2\%$  in terms of accuracy while adding linguistic features to the embeddings increases the results for both the BIN and 3-CLASS configurations. We, therefore, keep  $\mathrm{BERT}^{R}_{\mathrm{features}}$  as basis for the rest of the models. Concerning the generalization strategies, all replacements were productive and outperformed all the previous models, observing that gendered replacements are better. This shows that forcing the classifier to learn from general concepts is a good strategy for sexism content detection. In particular, we observe that the best replacement depends on the task: For BIN, it is place and gen
$^{14}$ Following (Badjatiya et al., 2017), we also experiment with automatic word lists but the results were not conclusive as frequent words were too generic and not representative of the problem we want to solve.
CLASSIFIER	BIN	3-CLASS
A	F	P	R	A	F	P	R
BERTbase	0.773	0.723	0.726	0.721	0.714	0.540	0.572	0.515
BERTR	0.790	0.762	0.767	0.759	0.726	0.567	0.609	0.531
BERTRown_emb + base	0.768	0.751	0.712	0.795	0.708	0.526	0.605	0.513
BERTRfeatures	0.795	0.787	0.819	0.761	0.754	0.588	0.625	0.556
BERTRfeatures + gen(Hypernym)	0.806	0.804	0.835	0.776	0.763	0.614	0.649	0.598
BERTRfeatures + gen(Hypernym_gendered)	0.809	0.807	0.840	0.777	0.767	0.635	0.663	0.620
BERTRfeatures + gen(Name)	0.790	0.796	0.830	0.766	0.755	0.620	0.656	0.606
BERTRfeatures + gen(Name_gendered)	0.815	0.806	0.841	0.775	0.760	0.643	0.665	0.630
BERTRfeatures + gen(SexistVocabulary_gendered)	0.801	0.807	0.836	0.781	0.764	0.635	0.654	0.627
BERTRfeatures + gen(Place)	0.826	0.813	0.848	0.782	0.769	0.655	0.673	0.646
BERTRfeatures + gen(Place+Hypernym)	0.803	0.799	0.836	0.766	0.758	0.622	0.654	0.610
BERTRfeatures + gen(Place+Hypernym_gendered)	0.819	0.811	0.846	0.779	0.771	0.652	0.689	0.630
BERTRfeatures + gen(Place+Name_gendered)	0.837	0.824	0.865	0.787	0.769	0.629	0.657	0.615
BERTRfeatures + gen(Place+Hypernym_gendered+Name_gendered)	0.819	0.818	0.857	0.783	0.764	0.634	0.662	0.618
dered names whereas for 3-CLASS it is place and gendered hypernym. In both cases, replacing only public spaces with the generic <location> was one of the best strategy with 0.826 and 0.769 accuracy for respectively BIN and 3-CLASS. Multiple replacements (cf. last line in the table) were however, less productive.
Table 4 further details the results per class for the best performing systems for each task (i.e., those in bold in Table 3). For the 3-CLASS, we observe that the results are lower for the sexist content (direct and descriptive) class, but this might also be a consequence of the low number of instances annotated as such<sup>15</sup>.
Table 3: Results for most productive models for BIN and 3-CLASS classification.
Task	Class	F	P	R
BIN	non sexist	0.874	0.894	0.855
sexist	0.773	0.836	0.719
overall	0.824	0.865	0.787
3-CLASS	non sexist	0.849	0.855	0.842
reporting	0.666	0.633	0.703
sexist	0.452	0.532	0.392
overall	0.655	0.673	0.646
CASC	non sexist	0.882	0.912	0.855
reporting	0.942	0.919	0.975
sexist	0.791	0.768	0.816
A = 0.831
overall	0.717	0.724	0.709
Table 4: Results per class for the three tasks.
6.2 CASC results
Cascading models are known for being very accurate and can be used in the context of moderation
as we cannot afford to take actions against users that are following the guidelines and policies. In the first stage we used the best performing model for sexist content vs. non sexist classification (i.e.,  $\mathrm{BERT}^{R}_{\mathrm{gen(Place + Name - gendered)}}$ ). The instances classified as containing a sexist content by the first model were further used as the testing set for the second model (the best performing model for the 3-CLASS classification task in terms of F-score, i.e.,  $\mathrm{BERT}^{R}_{\mathrm{gen(Place)}}$ ). In Table 4, the results corresponding to the non-sexist class of CASC classifier present the improvement brought by the second stage classifier, i.e., it was able to correct (predict as non-sexist) instances that were misclassified during the first stage. The last line of Table 4 presents the overall results obtained after the two stages of classification. The results show an improvement over the best system of 3-CLASS, proving the usefulness of a cascading approach with an increasing system complexity.
6.3 Discussion
A manual error analysis shows that misclassification cases are due to several factors, among which humor and satire (as in (9)) or the use of stereotypes (as in (10)), mainly because they are not expressed by a single word or expression but by metaphors. In the examples below, the underlined words highlight the leading cause of misclassification.
(9) Ma femme est hystorique. C'est comme hystérique, sauf que lorsqu'elle pète un cable elle me sort des valeurs dossiers.
(My wife is hysterical. That's like hysterical, except that when she's angry she pulls out old files)
<sup>15</sup>We tried augmenting the number of instances in these classes by replacing the words/phrases that belong to the sexist vocabulary and stereotyped words list (cf. Section5) with the top 10 word2vec neighbours (i.e., for each instance we obtain 10 more) but the results were not conclusive. More accurate data augmentation techniques can be investigated.
(10) je demande pas ce qu'elle a fait sous le bureau pour arriver à se plateau (I'm not asking what she did under the desk to be on this TV set)
In particular for reporting tweets, we found many misclassified messages without any reporting verb or quotes as in (11), but also messages denunciating sexism using situational irony as in (12).
(11) Royal les rendrait elle tous fous? Alain Destrem (UMP): Ségolène Royal en boubou bleu,ça me rappelle ma femme de menage !(Does Royal make them all crazy? Alain Destrem (UMP): Ségolène Royal wearing a blue boubou, it reminds me my cleaning woman!)
(12) Continuons à communier... Notre héros national avait des comptes en Suisse et n'était pas loin du #balancetonporc... Mais bon commu-nions, rassemblons nous...(Let's keep on be united... Our national hero had bank accounts in Switzerland and was not far from #SquealOnYourPig... But OK let's be united, let's get together...)
In this paper, we have presented the first approach to detect reports/denunciations of sexism from real sexist content that are directly addressed to a target or describes a target. We proposed a new dataset of about 12,000 French tweets annotated according to a new characterization of sexist content inspired from both speech act theory and discourse studies in gender. We then experimented with several deep learning models in binary, three classes and a cascade classifier configurations, showing that BERT trained on word embeddings, linguistic features and generalization strategies (i.e., place and hypernym replacements) achieved the best results for all the configurations, and that cascade classification allows to successfully correct misclassified non-sexist messages. These results are encouraging and demonstrate that detecting reporting assertions of sexism is possible, which is a first step towards automatic offensive content moderation. In the future, we plan to develop more complex models to be added in the next stages of the cascade classifier as well as automatically identify irony, gender stereotypes and sexist vocabulary.
This work is funded by the Institut Carnot Cognition under the project SESAME.
Alan Akbik, Duncan Blythe, and Roland Vollgraf. 2018. Contextual String Embeddings for Sequence Labeling. In COLING 2018, 27th International Conference on Computational Linguistics, pages 1638-1649.
Maria Anzovino, Elisabetta Fersini, and Paolo Rosso. 2018. Automatic Identification and Classification of Misogynistic Language on Twitter. In Natural Language Processing and Information Systems - 23rd International Conference on Applications of Natural Language to Information Systems, NLDB 2018, pages 57-64.
John L. Austin. 1962. How to Do Things with Words. Oxford University Press.
Pinkesh Badjatiya, Manish Gupta, and Vasudeva Varma. 2019. Stereotypical Bias Removal for Hate Speech Detection Task Using Knowledge-based Generalizations. In The World Wide Web Conference, pages 49-59.
Pinkesh Badjatiya, Shashank Gupta, Manish Gupta, and Vasudeva Varma. 2017. Deep Learning for Hate Speech Detection in Tweets. In Proceedings of the 26th International Conference on World Wide Web Companion, pages 759-760.
Azy Barak. 2005. Sexual harassment on the Internet. Social Science Computer Review, 23(1).
Valerio Basile, Cristina Bosco, Elisabetta Fersini, Debora Nozza, Viviana Patti, Francisco Manuel Rangel Pardo, Paolo Rosso, and Manuela Sanguinetti. 2019. SemEval-2019 Task 5: Multilingual Detection of Hate Speech Against Immigrants and Women in Twitter. In Proceedings of the 13th International Workshop on Semantic Evaluation, pages 54-63.
Farah Benamara, Véronique Moriceau, and Yvette Yannick Mathieu. 2014. Fine-grained semantic categorization of opinion expressions for consensus detection (in French). In DEFT 2014 Workshop: Text Mining Challenge, pages 36-44.
Claudia Bianchi. 2014. The speech acts account of derogatory epithets: some critical notes. *Liber Amicorum Pascal Engel*.
Laurence Biscarrat, Marlene Coulomb-Gully, and Cecile Méadel. 2016. One is not born a female CEO and...won't become one! Gender equality and the media - a challenge for Europe. Routledge, ECREA Book Series.
Tolga Bolukbasi, Kai-Wei Chang, James Zou, Venkatesh Saligrama, and Adam Kalai. 2016. Man is to Computer Programmer As Woman is to Homemaker? Debiasing Word Embeddings. In Proceedings of the 30th International Conference on Neural Information Processing Systems, pages 4356-4364.
Simone Bonnafous. 2003. "Femme politique": une question de genre? Rèseaux, 120.
Danielle Bousquet, Françoise Vouillot, Margaux Collet, and Marion Oderda. 2019. 1er état des lieux dusexisme en France. Technical report, Haut Conseil à l'Egalité entre les femmes et les hommes. http:// www.haut-conseil-egalite.govu.fr/IMG/ pdf/hce_etatdeslieux-sexisme-vf-2.pdf.
Deborah Cameron. 1992. *Feminism and Linguistic Theory*. Palgrave Macmillan.
Bianca Cepollaro. 2015. In defence of a presuppositional account of slurs. Language Sciences, 52.
Pat K Chew and Lauren K Kelley-Chew. 2007. Subtly sexist language. *Colum. J. Gender & L.*, 16.
European Commission COM. 2017. Communication from the Commission to the European Parliament, the Council, the European Economic and Social Committee and the Committee of the Regions. Tackling illegal content online, towards an enhanced responsibility of online platforms. Technical report, COM (2017), 555 final. https://eur-lex.europa.eu/legal-content/EN/TXT/HTML/?uri=CELEX:52017DC0555.
Marlene Coulomb-Gully. 2012. Présidente: le grand défi - femme, politique et medias. Paris, Payot/Éd. Rivages.
Haoyun Dai and Xiaodong Xu. 2014. Sexism in News: A Comparative Study on the Portrait of Female and Male Politicians in The New York Times. Open Journal of Modern Linguistics, 4.
Thomas Davidson, Dana Warmsley, Michael W. Macy, and Ingmar Weber. 2017. Automated Hate Speech Detection and the Problem of Offensive Language. In Proceedings of the Eleventh International Conference on Web and Social Media, pages 512-515.
Richard Delgado, Adrien Katherine Wing, and Jean Stefancic. 2015. Words That Wound: A Tort Action for Racial Insults, Epithets, and Name-Calling. In Law Unbound!, pages 223-228. Routledge.
Sunipa Dev and Jeff M. Phillips. 2019. Attenuating Bias in Word vectors. In The 22nd International Conference on Artificial Intelligence and Statistics, AISTATS 2019, pages 879-887.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference
of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186.
Mai ElSherief, Vivek Kulkarni, Dana Nguyen, William Yang Wang, and Elizabeth M. Belding. 2018. Hate Lingo: A Target-Based Linguistic Analysis of Hate Speech in Social Media. In Proceedings of the Twelfth International Conference on Web and Social Media, ICWSM 2018, pages 42-51.
Elisabetta Fersini, Debora Nozza, and Paolo Rosso. 2018a. Overview of the Evalita 2018 Task on Automatic Misogyny Identification (AMI). In Proceedings of the Sixth Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop (EVALITA 2018) co-located with the Fifth Italian Conference on Computational Linguistics (CLiC-it 2018).
Elisabetta Fersini, Paolo Rosso, and Maria Anzovino. 2018b. Proceedings of the Third Workshop on Evaluation of Human Language Technologies for Iberian Languages (IberEval 2018) co-located with 34th Conference of the Spanish Society for Natural Language Processing (SEPLN 2018). volume 2150 of CEUR Workshop Proceedings. CEUR-WS.org.
Anastasia Giannakidou and Alda Mari. 2021. (Non) Veridicality in grammar and thought. Mood, Modality and Propositional Attitudes. The University of Chicago Press.
Peter Glick and Susan T. Fiske. 1996. The Ambivalent Sexism Inventory: Differentiating Hostile and Benevolent Sexism. Journal of Personality and Social Psychology, 70(3).
Jennifer Golbeck, Zahra Ashktorab, Rashad O. Banjo, Alexandra Berlinger, Siddharth Bhagwan, Cody Buntain, Paul Cheakalos, Alicia A. Geller, Quint Gergory, Rajesh Kumar Gnanasekaran, Raja Rajan Gunasekaran, Kelly M. Hoffman, Jenny Hottle, Vichita Jienjtlert, Shivika Khare, Ryan Lau, Marianna J. Martindale, Shalmali Naik, Heather L. Nixon, Piyush Ramachandran, Kristine M. Rogers, Lisa Rogers, Meghna Sardana Sarin, Gaurav Shahane, Jayanee Thanki, Priyanka Vengataraman, Zijian Wan, and Derek Michael Wu. 2017. A Large Labeled Corpus for Online Harassment Research. In Proceedings of the 2017 ACM on Web Science Conference, pages 229-233.
Hila Gonen and Yoav Goldberg. 2019. Lipstick on a Pig: Debiasing Methods Cover up Systematic Gender Biases in Word Embeddings But do not Remove Them. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 609-614.
Edouard Grave, Piotr Bojanowski, Prakhar Gupta, Armand Joulin, and Tomas Mikolov. 2018. Learning Word Vectors for 157 Languages. In Proceedings of LREC.
Nabil Hathout and Franck Sajous. 2016. Wiktionnaire's Wikicode GLAWified: a Workable French Machine-Readable Dictionary. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16), pages 1369-1376.
Vijayasaradhi Indurthi, Bakhtiyar Syed, Manish Shrivastava, Nikhil Chakravartula, Manish Gupta, and Vasudeva Varma. 2019. FERMI at SemEval-2019 Task 5: Using Sentence embeddings to Identify Hate Speech Against Immigrants and Women in Twitter. In Proceedings of the 13th International Workshop on Semantic Evaluation.
Akshita Jha and Radhika Mamidi. 2017. When does a compliment become sexist? Analysis and classification of ambivalent sexism using Twitter data. In Proceedings of the Second Workshop on NLP and Computational Social Science, pages 7-16.
Sweta Karlekar and Mohit Bansal. 2018. SafeCity: Understanding Diverse Forms of Sexual Harassment Personal Stories. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2805-2811.
Rae Langton. 2012. Beyond Belief: Pragmatics in Hate Speech and Pornography. Speech and Harm: Controversies Over Free Speech.
Michelle M Lazar. 2007. Feminist critical discourse analysis: Articulating a feminist discourse praxis. Critical Discourse Studies, 4(2).
Kate Manne. 2017. Down girl: The logic of misogyny. Oxford University Press.
Alda Mari, Claire Beyssade, and Fabio Del Prete. 2012. Genericity, volume 43. Oxford University Press.
Louis Martin, Benjamin Muller, Pedro Javier Ortiz Suárez, Yoann Dupont, Laurent Romary, Éric Villemonte de la Clergerie, Djamé Seddah, and Benoit Sagot. 2019. CamemBERT: a Tasty French Language Model. arXiv e-prints, page arXiv:1911.03894.
Mari J Matsuda, Charles R Lawrence III, Richard Delgado, and Kimberly Williams Crenshaw. 1993. Words that wound: Critical race theory. *Assaultive Speech*, and the First Amendment, 5.
Jessica Megarry. 2014. Online incivility or sexual harassment? Conceptualising women's experiences in the digital age. Women's Studies International Forum, 47.
Sara Mills. 2008. Language and sexism. Cambridge University Press Cambridge.
Nedjma Ousidhoum, Zizheng Lin, Hongming Zhang, Yangqiu Song, and Dit-Yan Yeung. 2019. Multilingual and Multi-Aspect Hate Speech Analysis. In Proceedings of EMNLP-IJCNLP.