SlowGuess/NAACL_2021_OCR huggingface dataset:
Figure 2: Flexible Cross-stitched Parameter Sharing Architecture. The embedding representations  $(e_1^{(p)}, e_2^{(p)}, \dots, e_n^{(p)})$  and  $(e_1^{(a)}, e_2^{(a)}, \dots, e_n^{(a)})$  identify BERTweet word-level embeddings for the primary and auxiliary task respectively. The different arrows are used to indicate the alternate passes of the primary task (solid arrows) and auxiliary task (dotted arrows). Two controllable parameters  $\alpha^{(p)}$  and  $\alpha^{(s)}$  are used to control information flow from task-specific and shared encoder respectively, for the primary task.
Sexual Abuse Disclosures - #MeTooMA This dataset  $^{4}$  has 9,973 tweets and covers different mutually non-exclusive linguistic annotations related to the #MeToo movement (Gautam et al., 2020). The distribution and statistics about various labels are present in Table 1 and Section 3. We present an instance associated with each of the proposed tasks in Table 1. For our experiments, we focus only on tweets that are annotated as relevant to the #MeToo movement.
Emotions - SemEval18 This dataset<sup>5</sup> has been taken from SemEval-2018 Task-1 (Mohammad et al., 2018) and covers emotion-specific labels representing the mental state of the authors of the tweets. It consists of 10,986 tweets distributed across 11 emotion labels – (anger, disgust, anticipation, fear, joy, love, optimism, pessimism, sadness, surprise and trust), each being a binary label to indicate the presence of a particular emotion.
5.2 Task Specific Setting
Single Task Learning STL experiments optimize each of the tasks associated with identifying narratives related to sexual abuse disclosures within #MeToo movement (Section 3) and emotion
detection, independently. We experiment with two distinct embedding spaces - GloVe-Tawitter and BERTweet. Based on the superior performance of BERTweet with respect to GloVe-Tawitter, we preferred it for further experimentation and studies.
Homogeneous Multitask Learning For this setup, we test the simultaneous optimization of two different tasks - both related to sexual harassment disclosure narratives, with one of them being primary and another coupled as the auxiliary. The results were obtained for a total of 12 pairs.
Heterogeneous Multitask Learning In these sets of experiments, we evaluate the positive transfer of representations across datasets by considering the identification of narratives associated with sexual abuse disclosure as the primary task and emotion detection as the auxiliary task.
Preprocessing We pre-process tweet text by (i) normalizing user mentions and URLs, and (ii) translating the emoticon into text (Hutto and Gilbert, 2014). For tokenization, we use Tweet Tokenizer from NLTK.
4The publicly available dataset can be found at https: //doi.org/10.7910/DVN/JN4EYU.
5https://competitions.codalab.org/competitions/17751
Hyperparameters For our model hyperparameters were tuned on the validation set to find the best configurations. We use a pre-trained BERTweet model to extract 768-dimensional token-level embeddings. Grid search was performed to find the optimal value of hyperparameters and their range is summarized as: size of BiLSTM and dense layers \{128,256,512\}, embedding size  $d\in \{100,200,300\}$ , dropout  $\delta \in \{0.1,0.2,0.3,0.4,0.5,0.6\}$ , learning rate  $\lambda \in \{10^{-5},10^{-4},10^{-3},10^{-2},10^{-1}\}$ , weight decay  $\omega \in \{10^{-6},10^{-5},10^{-4},10^{-3}\}$ , optimizer  $\{Adam,Adadelta\}$ , batch size  $b\in \{32,64,128\}$  and epochs ( $< 100$ ). For the MTL experiments, we tune the weightage of the auxiliary task ( $\gamma \in [0.1,0.9]$  with intervals of 0.1) for each task pair.
For each task associated with identifying narratives pertaining to the #MeToo movement in the MTL setup, its value is considered as the one where the model performance improved the most and for both the tasks. For instance, we find the optimal value of  $\gamma$  for hate speech (as the auxiliary task) to be 0.4 in all Homogeneous task cases and of emotion detection to be 0.2 for the Heterogeneous tasks. For the MTL experiments,  $\alpha^p$  and  $\alpha^s$  are learnable and tuned on the validation loss. The encoders consist of two stacked BiLSTM's with hidden size  $= 128$ . BiLSTM classifier has hidden size  $= 256$ , and the number of units in the penultimate dense layer is 128. Dropout is set to 0.3. For all our experiments, we use Adam optimizer (Kingma and Ba, 2014) and initialize model weights using Xavier initialization (Glorot and Bengio, 2010). We set the batch size to 128 and the learning rate to  $1e - 3$ .
Training All models were trained until convergence for both primary and auxiliary tasks. For our MTL experiments, the training process involves alternating between primary and auxiliary task steps, with each task having its own loss function. All experiments are run using stratified 5-fold cross-validation. We report the average macro F1 scores across the 5 folds to account for imbalance, as previously used in multi-label settings (Zhang and Zhou, 2013).
Task	ST	HS	SA	DI
ST	31.80	31.67	32.41	32.20
HS	31.82	31.78	31.64	31.80
SA	49.63	49.69	49.16	49.79
DI	23.54	23.42	23.20	23.41
Table 2: F1 macro score for pair-wise MTL (non-diagonal elements) and STL (diagonal elements - top left corner to bottom right corner). Rows denote the primary task and columns denote the auxiliary task in case of MTL. ST = Stance; HS = Hate Speech; SA = Sarcasm; DI = Dialogue. Bold denotes the highest score for that task.
Task	Homogeneous	Heterogeneous
Stance	32.41 ± 0.01 (SA)	32.62 ± 0.03
Hate Speech	31.82 ± 0.02 (ST)	32.01 ± 0.01
Sarcasm	49.79 ± 0.03 (DI)	49.50 ± 0.04
Dialogue	23.54 ± 0.03 (ST)	23.16 ± 0.06
Table 3: Best F1-scores obtained for Homogeneous MTL (Table 2) and Heterogeneous MTL experiments. Heterogeneous MTL experiments represent multitask learning performed with emotion identification as the auxiliary task. The best results highlighted in bold.
6.1 Single Task Learning
The aim of this paper is not limited to achieving the state of the art performance in terms of evaluation metrics but rather to conduct a thorough study to compare and contrast different methodologies for the benefit of the research community. As per our hypothesis and preliminary results on STL experiments on the #MeTooMA dataset, models trained using BERTweet embeddings perform far better than GloVe-TaTtter. This is largely true because BERTweet is specifically pre-trained on English tweets and is better suited to handle Twitter-specific data, typically having a short length, informal grammar, and irregular vocabulary (e.g., abbreviations and typographical errors) (Kireyev et al., 2009).
6.2 Single Task Learning vis-a-vis Homogeneous Multitask Learning
Learning the affective states in the #MeTooMA dataset is challenging due to the inherently subjective nature of the tweets coupled with limitations on the data's size. Multitask learning achieves significant performance gains in terms of macro F1 score, as shown in Table 2 for all task pairs. The diagonal results represented in green denote the baseline STL results whereas ones highlighted in shades of blue represent results for pair-wise
<sup>7</sup>We used Keras with Tensorflow backend for implementing the models.
Tweet Text	STL	Homogeneous MTL	Heterogeneous MTL
T1 - [name] says that nobody should be ashamed. Don't be scared and let it bury and corrode your soul. It gives hope through the pain, please visit [URL]. Lets speak up. #FightBack	Support	Support	Support
T2 - Saying that #MeToo movement could save lives, would be a grave mistake, those individuals deserve it, such swindling. This is the proof of a disabled mindset among those guys. #Resist #RiseUp.	Neither	Support	Oppose
T3 - When I ran my side production for the movie, people said that only p*nsi*s and h*m*s worked there because I had zero tolerance for sexual harassment in my unit. This is for all screeching people.	Neither	Gen Hate	Gen Hate
T1 - I believe that this ideology must be broken! Society now has created stigma and its time we move forward to a new way of thinking for all girls. Please spread the word. #metoo #Healing.	Neither	Support	Support
T5 - [name] says #MeToo is a trap, set by the left wingers. Don't take the bait. The right shouldn't worry least-bit, especially because of the involvement of [name] in these circumstances. #BringTheTruth	Neither	Refutation	Neither
T6 - [name] embodies #MeToo movement today, writing and spreading fake messages loud and clear. Push false narrative and wrong ideology among the youth. See this [URL]. #Liar #YesAllWomen.	Neither	Justification	Neither
Table 4: Qualitative analysis of the performance obtained by MTL architecture on some samples. The color intensity of each word corresponds to the token-level attention score given by BERTweet. Green denotes correct prediction and Yellow denotes incorrect prediction. Tweets have been paraphrased to prevent user identification.
Homogeneous MTL with row identifying primary task and columns denoting auxiliary task. The higher performance of Homogeneous MTL can be inferred to be indicative of better generalization when pairs of tasks are jointly modeled. Interestingly, these tasks show their best performance with the selective counterparts in the Homogeneous MTL setup. Stance detection is strongly coupled with Sarcasm labeling, and the same is seen to be true for Hate Speech classification and Stance identification. This selective out-performance of specific pairs of tasks can be attributed to the high correlation between the tasks themselves (Frenda, 2018; Gautam et al., 2020). For instance, the offensive text is often strongly coupled with sarcasm, as wit is a common linguistic denominator for understanding the intended meaning of phrases related to anger (Badlani et al., 2019). We further detail this through examples in Section 6.4.
6.3 Heterogeneous Multitask Learning
Results in Table 3 demonstrate that the Heterogeneous MTL setup achieves higher performance than Homogeneous MTL under similar settings in two out of four task pairs $^{8}$  - Stance and Hate Speech detection by the margins of  $+0.21$  and  $+0.19$  respectively. For the other two tasks, the performance of Heterogeneous MTL is very close if not better than Homogeneous MTL. These findings are in line with the claim supporting the generalizability across tasks in the #MeTooMA dataset, which is
highly correlated to emotion recognition. This is indicative of positive knowledge transfer between the two domains. Such joint optimization boosts the overall performance of both primary and auxiliary tasks through parameter sharing to learn common representations that may be mutually beneficial to both related tasks.
To emphasize our proposed approach, we perform a qualitative study by handpicking examples from the dataset. We analyze token-level attention assigned to individual terms by BERTweet, where color intensity corresponds to the attention score. These results are shown in Table 4. We infer that Homogeneous and Heterogeneous multitask learning shows superior performance in every instance compared to STL. Learning effective features across the joint formulation of pair-wise tasks in Homogeneous MTL is evident from  $T_{4}$ , where BERT's self-attention allots a higher weight to words such as ideology, stigma, and forward in line with the actual label as Support.
Similarly for  $T_{5}$ , highlighted terms such as trap and bait are indicative of the opposing nature of the tweets, hence identified as belonging to Refutation. On the other hand, due to positive knowledge transfer from the emotion recognition task, Heterogeneous MTL obtains better performance in several cases. Words such as grave, mistake and swindling in  $T_{2}$  connoted a negative emotion, hence accordingly being identified as belonging to the Oppose category. Similarly, terms such as
<sup>8</sup>We show only the best combinations of the Homogeneous task in the table for brevity.
hope and pain were given higher token-level attention in  $T_{1}$  emphasizing a positive emotion and thus can be correlated with belonging to the Support category. An interesting observation is the presence of named entities in  $T_{5}$  and  $T_{6}$ , resulting in the incorrect prediction via Heterogeneous MTL. Therefore, a limitation of the single task learning and Heterogeneous MTL is the inability to mitigate the effect of named entities or specific events in the text to influence the knowledge transfer and create negative shared representations.
7 Ethical Concerns and Discussion
Analyzing social media data of individuals discussing sexual harassment disclosures and exploitation in public spheres necessitates the need to safeguard the ethics and privacy of individuals (Tusinski Berg, 2019). We address these:
Generalization We acknowledge that the limitations of the experiments might get amplified due to the highly subjective nature of this challenging problem. Therefore it would not be fair to conduct a population-centric analysis based on inferences from this work.
Confidentiality Individual consent was not sought from social media users as the data was publicly available. Disclosure of sexual harassment information on public forums may have been met with public backlash and apathy. Therefore the social reputation of the accuser and the accused would be at a peril (McDonald, 2019). Hence, the authors were aware not to make any automated interventions, as any attempts to contact individuals could be seen as personally intrusive and might also repeal their social information (Fiesler and Proferes, 2018).
Bias & Discrimination Social support discussions on social media platforms gave victims the liberty to describe their instances of sexual exploitation and abuse (Manikonda et al., 2018a). The authors are aware of the potential inevitable sampling biases that may be present in the data. Importance has to be placed on mitigating the bias against certain minority groups, which might get amplified due to the sensitive nature of social discussions (Hellwig and Sinno, 2017).
In this work, we have proposed a flexible cross-stitched multitask learning framework for the de
tection of narratives linked with sexual abuse disclosure on social media. Our methodology takes advantage of the affective features from emotions and related tasks to encourage knowledge transfer and attain auxiliary knowledge. Qualitative and quantitative results demonstrate how joint optimization of Stance detection and Sarcasm identification benefit each other, indicating their relatedness and dependence on each other. Similarly, we observe that tasks like Hate-Speech classification and Stance labeling benefit from each other and from emotion detection, thus reinforcing the benefit of joint linguistic learning between the related tasks. In the future, we aim to explore how this joint learning paradigm can be effectively leveraged for improving performance on downstream tasks like emotion analysis, identifying suicidal tendencies among abuse survivors. Application from this work also has utility for problems such as identification of patterns of reported sexual harassment narratives, hate speech detection, the spread of rumors and fake news, and entity extraction for digital vigilantism (Yuce et al., 2014; Hosterman et al., 2018).
Rajiv Ratn Shah is partly supported by theInfosys Center for AI and Center for Design and New Media at IIIT Delhi.
Md Shad Akhtar, Deepanway Ghosal, Asif Ekbal, Pushpak Bhattacharyya, and Sadao Kurohashi. 2018. A multi-task ensemble framework for emotion, sentiment and intensity prediction. arXiv preprint arXiv:1808.01216.
Maria Anzovino, Elisabetta Fersini, and Paolo Rosso. 2018. Automatic identification and classification of misogynistic language on twitter. In International Conference on Applications of Natural Language to Information Systems, pages 57-64. Springer.
Isabelle Augenstein, Tim Roktaschel, Andreas Vlachos, and Kalina Bontcheva. 2016. Stance detection with bidirectional conditional encoding. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 876-885, Austin, Texas. Association for Computational Linguistics.
Rohan Badlani, Nishit Asnani, and Manan Rai. 2019. Disambiguating sentiment: An ensemble of humour, sarcasm, and hate speech features for sentiment classification. W-NUT 2019, page 337.
David Bamman and Noah A Smith. 2015. Contextualized sarcasm detection on twitter. In Ninth International AAAI Conference on Web and Social Media.
Valerio Basile, Cristina Bosco, Elisabetta Fersini, Debora Nozza, Viviana Patti, Francisco Manuel Rangel Pardo, Paolo Rosso, and Manuela Sanguinetti. 2019. SemEval-2019 task 5: Multilingual detection of hate speech against immigrants and women in twitter. In Proceedings of the 13th International Workshop on Semantic Evaluation, pages 54–63, Minneapolis, Minnesota, USA. Association for Computational Linguistics.
Rich Caruana. 1997. Multitask learning. Machine learning, 28(1):41-75.
Hao Cheng, Hao Fang, and Mari Ostendorf. 2015. Open-domain name error detection using a multitask rn. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 737-746.
Arijit Ghosh Chowdhury, Ramit Sawhney, Rajiv Shah, and Debanjan Mahata. 2019. # youtoo? detection of personal recollections of sexual harassment on social media. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2527-2537.
Rosemary Clark-Parsons. 2019. "i see you, i believe you, i stand with you":# metoo and the performance of networked feminist visibility. *Feminist Media Studies*, pages 1-19.
Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. Journal of machine learning research, 12(ArtICLE):2493-2537.
Yin Cui, Menglin Jia, Tsung-Yi Lin, Yang Song, and Serge Belongie. 2019. Class-balanced loss based on effective number of samples. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 9268-9277.
Verna Dankers, Marek Rei, Martha Lewis, and Ekaterina Shutova. 2019. Modelling the interplay of metaphor and emotion through multitask learning. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2218-2229, Hong Kong, China. Association for Computational Linguistics.
Thanh Vu Dat Quoc Nguyen and Anh Tuan Nguyen. 2020. BERTweet: A pre-trained language model for English Tweets. arXiv preprint, arXiv:2005.10200.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for
Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186.
Zhengming Ding, Shao Ming, and Yun Fu. 2014. Latent low-rank transfer subspace learning for missing modality recognition. In Twenty-eighth AAAI conference on artificial intelligence.
Long Duong, Trevor Cohn, Steven Bird, and Paul Cook. 2015. Low resource dependency parsing: Crosslingual parameter sharing in a neural network parser. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 845-850, Beijing, China. Association for Computational Linguistics.
Long Duong, Hiroshi Kanayama, Tengfei Ma, Steven Bird, and Trevor Cohn. 2016. Learning crosslingual word embeddings without bilingual corpora. arXiv preprint arXiv:1606.09403.
Mai ElSherief, Vivek Kulkarni, Dana Nguyen, William Yang Wang, and Elizabeth Belding. 2018. Hate lingo: A target-based linguistic analysis of hate speech in social media. arXiv preprint arXiv:1804.04257.
Casey Fiesler and Nicholas Proferes. 2018. "participant" perceptions of twitter research ethics. *Social Media+ Society*, 4(1):2056305118763366.
Mary Anne Franks. 2019. Witch hunts: Free speech, # metoo, and the fear of women's words. U. Chi. Legal F., page 123.
Simona Frenda. 2018. The role of sarcasm in hate speech. a multilingual perspective. In e Doctoral Symposium of the XXXIVInternational Conference of the Spanish Society for Natural Language Processing (SEPLN 2018), pages 13-17. Lloret, E.; Saquete, E.; Martinez-Barco, P.; Moreno, I.
Akash Gautam, Puneet Mathur, Rakesh Gosangi, Debanjan Mahata, Ramit Sawhney, and Rajiv Ratn Shah. 2020. # metooma: Multi-aspect annotations of tweets related to the metoo movement. In Proceedings of the International AAAI Conference on Web and Social Media, volume 14, pages 209-216.
Xavier Glorot and Y. Bengio. 2010. Understanding the difficulty of training deep feedforward neural networks. Journal of Machine Learning Research - Proceedings Track, 9:249-256.
Pankaj Gupta, Hinrich Schütze, and Bernt Andrassy. 2016. Table filling multi-task recurrent neural network for joint entity and relation extraction. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 2537-2547.
Timothy Hellwig and Abdulkader Sinno. 2017. Different groups, different threats: public attitudes towards immigrants. Journal of Ethnic and Migration Studies, 43(3):339-358.
Alec R Hosterman, Naomi R Johnson, Ryan Stouffer, and Steven Herring. 2018. Twitter, social support messages, and the# metoo movement. The Journal of Social Media in Society, 7(2):69-91.
Chris Hutchings. 2012. Commercial use of facebook and twitter-risks and rewards. Computer Fraud & Security, 2012(6):19-20.
Clayton J Hutto and Eric Gilbert. 2014. Vader: A parsimonious rule-based model for sentiment analysis of social media text. In *Eighth international AAAI conference on weblogs and social media*.
Akshita Jha and Radhika Mamidi. 2017. When does a compliment become sexist? analysis and classification of ambivalent sexism using twitter data. In Proceedings of the second workshop on NLP and computational social science, pages 7-16.
Sweta Karlekar and Mohit Bansal. 2018. Safecity: Understanding diverse forms of sexual harassment personal stories. arXiv preprint arXiv:1809.04739.
Diederik Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. International Conference on Learning Representations.
Kirill Kireyev, Leysia Palen, and Kenneth Anderson. 2009. Applications of topics models to analysis of disaster-related twitter data. In NIPS workshop on applications for topic models: text and beyond, volume 1. Canada: Whistler.
Linda Lane and Ulla-Carin Hedin. 2020. The promise of the# metoo movement for preventing and reporting sexual harassment. Available at SSRN.
Janie Leatherman. 2011. Sexual violence and armed conflict. Polity.
Pengfei Liu, Xipeng Qiu, and Xuanjing Huang. 2016. Recurrent neural network for text classification with multi-task learning. arXiv preprint arXiv:1605.05101.
Pengfei Liu, Xipeng Qiu, and Xuanjing Huang. 2017. Adversarial multi-task learning for text classification. arXiv preprint arXiv:1704.05742.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach.
Minh-Thang Luong, Quoc V Le, Ilya Sutskever, Oriol Vinyals, and Lukasz Kaiser. 2015. Multi-task sequence to sequence learning. arXiv preprint arXiv:1511.06114.