SlowGuess/NAACL_2021_OCR huggingface dataset:
Ramit Sawhney†, Puneet Mathur‡, Taru Jain†, Akash Kumar Gautam†, Rajiv Ratn Shah†
† Department of Computer Engineering, IIIT-Delhi
{ramits, akash15011, rajivratn}@iiitd.ac.in
† University of Maryland, College Park
puneetm@cs.umd.edu
The #MeToo movement on social media platforms initiated discussions over several facets of sexual harassment in our society. Prior work by the NLP community for automated identification of the narratives related to sexual abuse disclosures barely explored this social phenomenon as an independent task. However, emotional attributes associated with textual conversations related to the #MeToo social movement are complexly intertwined with such narratives. We formulate the task of identifying narratives related to the sexual abuse disclosures in online posts as a joint modeling task that leverages their emotional attributes through multitask learning. Our results demonstrate that positive knowledge transfer via context-specific shared representations of a flexible cross-stitched parameter sharing model helps establish the inherent benefit of jointly modeling tasks related to sexual abuse disclosures with emotion classification from the text in homogeneous and heterogeneous settings. We show how for more domain-specific tasks related to sexual abuse disclosures such as sarcasm identification and dialogue act (refutation, justification, allegation) classification, homogeneous multitask learning is helpful, whereas for more general tasks such as stance and hate speech detection, heterogeneous multitask learning with emotion classification works better. $^{1}$
The #MeToo movement<sup>2</sup> was started as an initiative to empower women against long-standing issues related to sexual abuse at workplaces, public spaces, and private organizations (McKenna and Chughtai, 2020). The usage of a dedicated hashtag #MeToo on media platforms signified a social support system for women from different sections
While the Women helpline of [user] receives one call every two hours where the trauma traces back to sexual harassment at workplace, it gets 3 calls in an hour after the #MeToo movement began. And they say #MeToo movement is a hoax to extort money.
Sexual Harassment Disclosure Label: Justification, Support
I knew him as a good guy in the office but could never imagine how disgusting he could be. I am shocked but glad that someone had the courage to call the police on him! #UtterShocked.
Emotion Label: Disgust, Surprise
Figure 1: Examples showing the relationship between tweets annotated for sexual harassment disclosure (top) and emotion recognition (bottom). Colors highlight token level attention assigned by BERTweet.
of society. The movement initiated discussions on many socially stigmatized issues that were missing from the virtual space (Clark-Parsons, 2019). Such conversations invited various reactions on the web, involving support to the cause of the movement and even outright bullying. While many users took part in the vilification of the survivors, the movement also saw opposition by factions of the society that felt threatened by the impact of social media in raising awareness about the scale of everyday sexual harassment faced by women in workplaces and institutions (Tambe, 2018). In many instances, the public disclosures of survivor-narrated incidents involved widespread use of hate-language and online trolling, both against the victims and alleged oppressors (Franks, 2019). The #MeToo movement also led to people coming out with allegations, refutations, and justifications about traumatic experiences as they transitioned to active participants in the mainstream conversation (Gautam et al., 2020). A closer look at the online posts about the #MeToo movement revealed that sarcasm was often used as a thin veil in such discussions to humorously mask disapproval, wit, and personal attacks (Sandhu et al., 2019).
The complex narratives present in the conversations on stigmatized issues like sexual abuse create an opportunity for researchers to study how people
$^{1}$ Code & Implementation: https://github.com/midas-research/metoo-mtl-naacl
2https://metoomvmt.org/
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4881-4892 June 6-11, 2021. ©2021 Association for Computational Linguistics
express their opinions on a sensitive topic in an informal social setting. It also offers a chance to social media regulators for fostering social inclusion, community integration, and improving the individual perception of being supported by others. This paper aims at categorizing the posts related to the #MeToo movement on the basis of stance (support or opposition), hate-speech, sarcasm, and dialogue acts (allegation, refutation, or justification of sexual misconduct). We focus our analysis on a publicly available dataset that is created in the backdrop of mass instances of sexual harassment disclosures and includes nuanced labels to identify accompanying linguistic behaviors.
Existing literature has emphasized that the text's emotional attributes have a high correlation with dialogue narratives describing instances of sexual harassment (Lane and Hedin, 2020). Prior works (Anzovino et al., 2018; Sharifirad et al., 2018) have mostly focused on label specific detection of linguistic narratives related to sexual harassment disclosures in isolation by exploiting lexical features (Chowdhury et al., 2019; Karlekar and Bansal, 2018). However, subtle intricacies present in the discussion of sexual abuse disclosures often reflect the speaker's affective and psychological state, which are overlooked by feature-engineered models. For instance, part (a) of Figure 1 shows a tweet expressing support towards the #MeToo movement but in a tone that might be difficult for naive neural learning models to capture without context. Part (b) of Figure 1 presents a tweet in which the author has an initial positive outlook, which later reverses to disgust for the subject. The lack of context about the event and contrasting qualifications describing the oppressor makes the correct classification of the sexual harassment disclosure label extremely challenging for traditional classifiers without emotional labels' additional supervision.
Moreover, apart from their inherent complexity, conversations related to the #MeToo movement also pose a challenge of emotional ambiguity. This work is the first attempt at joint modeling of narratives related to sexual abuse disclosures and emotion classification to learn the patterns of their interaction via parameter sharing techniques offered by Multitask Learning (MTL). The affective features, which result from a joint learning setup through shared parameters, will encompass the text's emotional content that is likely to be predictive of narratives corresponding to sexual abuse disclo
sures. More specifically, we formulate an MTL framework for multi-label classification of narratives related to sexual abuse disclosures (stance, hate-speech, sarcasm, dialogue acts) and emotional classification in the context of the #MeToo movement. MTL (Caruana, 1997) allows two or more related tasks to be learned jointly. This facilitates the transfer of inductive bias and better generalization across related tasks on account of shared representations of linguistic features.
Contributions We experiment with MTL architectures employing a flexible cross-stitched parameter sharing method that benefits from both hard-parameter sharing and soft parameter sharing through a gated mechanism using a weighted summation (Section 4). Hard parameter sharing allows for sharing lower-level word representations, and soft parameter sharing permits the sharing of task-specific networks. We explore two flavors of multitask learning: (i) Homogeneous MTL - Intradomain MTL between related tasks of sexual abuse disclosure narratives, and (ii) Heterogeneous MTL - cross-domain MTL between pairs of tasks in emotion classification and narratives of sexual abuse disclosure (Section 5.2). Our results demonstrate that both Homogeneous and Heterogeneous MTL setups outperform the Single Task Learning (STL) technique across various tasks (Section 6). Further, we conduct a qualitative analysis of several samples to analyze the benefit of joint training of related tasks (Section 6.4), keeping in mind the ethical concerns of communities affected by this research (Section 7).
Sexual Harassment Disclosures on Social Media Several works have focused on identifying sexual violence (Leatherman, 2011), harassment and sexism (Wekerle et al., 2018; Manikonda et al., 2018b) in social media posts by analyzing factors such as linguistic themes, social engagement, and lexical attributes. Jha and Mamidi (2017) experimented with algorithms such as SVM and BiLSTM along with fastText to categorize hostility of sexist posts. (Parikh et al., 2019) proposed a multi-label CNN-based neural architecture along with word and sentence level embeddings for identifying variants of sexism present in online social platforms. Chowdhury et al. (2019) emphasized the use of linguistic themes, contextual meta-data, and semantic cues for evaluating human behaviors related to sex
ual abuse disclosures. All of these works have dealt with modeling sexual disclosure narratives as single-task learning problems and were restricted to label specific detection (Marwa et al., 2018; Sawhney et al., 2020).
Multitask Learning Frameworks for learning representations across two different sources within the same domain follow multitask learning (Caruana, 1997). The ability to utilize knowledge from various sources compensates for missing data and complements existing meta-data (Tan et al., 2013; Ding et al., 2014), thus allowing for effective sharing of task-invariant features (Caruana, 1997; Zhang and Wang, 2016; Zhang et al., 2018). MTL has been utilized for name error recognition (Cheng et al., 2015), tagging-chunking (Collobert et al., 2011), machine translation (Luong et al., 2015) and relation extraction (Gupta et al., 2016). Liu et al. (2017) used shared and private latent features leveraging multitask learning for different text classification tasks. Rajamanickam et al. (2020); Duong et al. (2016); Liu et al. (2016) proposed a joint framework for modeling abuse and emotion detection and showed improvements over STL and transfer learning. Akhtar et al. (2018) proposed a multitask ensemble architecture for jointly modeling emotion, sentiment, and intensity, which gave improvements over single-label classification.
We aim to analyze different perspectives of the complex narratives pertaining to the #MeToo movement on social media platforms. Specifically, given a tweet text, we formulate for it a multi-label multiclass classification problem with definitions taken from previous works (ElSherief et al., 2018)
- Stance Detection: Determining the opinion of the author of a tweet, regarding a particular target of interest (Augenstein et al., 2016). Stance detection is categorized into three classes: Support for when the author favors the #MeToo movement or it's cause; Opposition, representing opposing stance or indifference towards the movement; or Neither, when the text does not have a clear viewpoint (Mohammad and Turney, 2013).
- Hate Speech Identification: Detection of hate speech involves labeling the tweets as Directed Hate if the comment is targeted towards an individual or an entity, Generalized Hate if
it is targeted towards a community or a section of people or Neither otherwise (Basile et al., 2019).
- Sarcasm Detection: Given a tweet  $t_i$ , we aim to map it to either be Sarcastic or Not Sarcastic based on the presence of implicit sarcastic tone of the post (Bamman and Smith, 2015).
- Dialogue Act Classification: These are a function of a speaker's utterance during a conversation, for example, question, answer, suggestion, etc., and are classified into three classes, namely Allegation (when the author intends to allege an individual or group of sexual misconduct) (Hutchings, 2012), Justification (tweets where the author is justifying their actions), and Refutation (for when the author refutes any accusation with or without evidence) (Gautam et al., 2020).
Modeling Settings To validate MTL's performance across different domains, we also experiment with emotion detection as the auxiliary task. We aim to predict one or more of the several emotions representing the affective state of the authors - (anger, disgust, anticipation, fear, joy, love, optimism, pessimism, sadness, surprise and trust). We conceptualize three diverse problem settings and compare them to analyze MTL within and across domains. These are (i) Single Task Learning: Independent optimization of the four mentioned tasks associated with sexual abuse disclosure narrative classification, (ii) Homogeneous Multitask Learning: Simultaneous optimization of a pair selected from the four tasks associated with the sexual abuse disclosure posts, and (iii) Heterogeneous Multitask Learning: Classification of narratives associated with sexual abuse disclosure as the primary task and emotion detection as the auxiliary task.
4.1 Text Encoding
Building on the success of transformer-based models in NLP, we chose BERTweet (Dat Quoc Nguyen and Nguyen, 2020), a pre-trained language model trained on 850 million English tweets. BERTweet has been trained with the same training procedure as RoBERTa (Liu et al., 2019) and has the same model configuration as the BERT base architecture (Devlin et al., 2019). The key component in
Task	Label	#Samples	Text
Relevance	Relevant	7,249	Guys are pissed off at [name] for affecting the credibility of a sexual assault survivor. Only men and r*p* enablers are questioning the movement in today's times. #Attack #BringTheChange.
Stance	Support	3,074	Thank you [name] for your courage passion and fight for #MeToo. It gives [name] strength to overcoming all this in front of people. Hope this inspires others as well to bring more stories. #Survivor .
Opposition	743	The progressive video by [user] shows the lady as a stripper, but are upset when [name] calls this movement bogus. Can't believe lies especially when so much has happened. #Fake #MeToo.
Hate Speech	Directed	419	Life comes hard at [name]. Desperate for [name] approval, she tries to subvert the risk of her tawdry dating habits [URL] and disgusted company. Didn't even flinch once before saying this. #Fake.
Generalized	281	These are not involved at all in this .... #MeToo has nothing with hating people. Its just a strategy by feminist h*e to get their a*e rich by manipulating people and asking for money. #Feminists
Sarcasm	Sarcastic	220	Thankfully the #HimToo movement will encourage [name] to put his d*c* inside his pant, out of the fear of getting publicly criticized by others. #MeToo. Is this really story of the decade, LOL !!! #SpeakOut
Dialogue Acts	Allegation	578	Shut up now, [name], you said nothing when 10 women accused [name] of sexual harassment [URL] in the premises. Instead [name] remained silent among all this with eyes wide out. #IBelieveSurvivors [URL]
Justification	292	[name] embodies #MeToo movement, writing and spreading fake message loud and clear. Push false narrative and wrong ideology among the youth. This would get the job done. #BringOutTheTruth
Refutation	216	[name] says #MeToo is a trap, set by left wingers. Don't take the bait at all. The right shouldn't worry least, especially because of the involvement of [name]. This has happened far too many times in the past.
Table 1: Distribution of labels and examples for all tasks in #MeTooMA dataset. The tweets have been paraphrased for anonymity reasons and personally identifiable information has been censored. We want to caution the readers that examples in this paper, though censored for profanity might contain offensive language.
transformer-based models is the token level self-attention (Vaswani et al., 2017) that enables them to generate dynamic contextualized embeddings as opposed to static embeddings of GloVe (Pennington et al., 2014). Let  $(w_{1}, w_{2}, \ldots, w_{n})$  represent the sequence of tokens from a given tweet  $t$ . These tokens are pre-processed and passed through BERTweet<sup>3</sup>. We consider embeddings from the last layer of BERTweet and obtain an embedding  $e_{i}$  for a given tweet  $t_{i}$ . Embedding for each tweet is of dimension  $m \times k$ , where  $k$  represents the dimension size of BERT based model and  $m$  represents the maximum length for the tweets.
$$
e _ {i} = B E R T w e e t \left(t _ {i}\right) \tag {1}
$$
These representations from Equation 1 are passed through a stacked BiLSTM encoder. Dropout is then applied to these encoded representations  $h^{(t)}$  (Equation 4 represents general formulation for both the tasks). These are then passed to a BiLSTM decoder, followed by a dropout layer and then a linear output layer to get output  $o^{(p)}$  ( $p$  representing primary task) or  $o^{(a)}$  ( $a$  representing auxiliary task).
$$
\overrightarrow {h _ {t} ^ {(f)}} = B i L S T M ^ {(f)} \left(e _ {t}, h _ {t - 1} ^ {(f)}\right) \tag {2}
$$
$$
\overleftarrow {h _ {t} ^ {(b)}} = B i L S T M ^ {(b)} \left(e _ {t}, h _ {t + 1} ^ {(b)}\right) \tag {3}
$$
$$
h _ {t} = \left[ \overrightarrow {h _ {t} ^ {(f)}}, \overleftarrow {h _ {T - t} ^ {(b)}} \right] \tag {4}
$$
4.2 Single Task Learning
We treat the task of categorizing narratives related to sexual abuse disclosure – Stance, Hate Speech,
Sarcasm and Dialogue Acts, independently. Each STL model is given an input representation  $e$  (Equation 1). Within the proposed tasks for classifying sexual abuse disclosure narrative for the tweets related to the #MeToo movement (Section 3), we use sigmoid activation for Sarcasm detection (whose classification outputs are binary) and softmax activation for all other tasks for the final output layer.
Model Optimization To account for the imbalance present among the labels, we use class-balanced focal loss as the optimization loss function (Cui et al., 2019), as formulated in Equation 5. Given a sample class  $i$  containing  $n_i$  samples in total, it adds a weighting factor of  $\frac{(1 - \beta)}{(1 - \beta^{n_i})}$  with parameters  $\beta \in [0,1)$ , where  $n_y$  is the number of samples in the ground truth class  $y$ . The proposed class-balanced term is model agnostic.  $p$  represents predicted class probabilities and  $\mathcal{L}$  represents the choice of the loss function (binary cross entropy for Sarcasm and categorical cross entropy for others).
$$
C B (p, y) = \frac {1 - \beta}{1 - \beta^ {n _ {y}}} \mathcal {L} (p, y) \tag {5}
$$
As for the multilabel emotion classification task, the unnormalized output (assuming one or more of 11 different emotions) is subjected to a Sigmoid activation, and the network is optimized using binary cross-entropy (BCE) as:
$$
L _ {B C E} = - \frac {1}{N} \sum_ {i = 1} ^ {N} y _ {i}. \log (p \left(y _ {i}\right)) + \left(1 - y _ {i}\right). \log \left(1 - p \left(y _ {i}\right)\right) \tag {6}
$$
where  $N$  is the number of training samples,  $y$  and  $p(y)$  denotes true and predicted labels respectively.
Implementation used for BERTweet is available here
4.3 Multitask Learning
For our MTL approach, we use two optimization objectives: one for the primary task, which can be any of the proposed tasks for classifying tweets related to #MeToo movement (Section 3), and other for the auxiliary task, which can be either a task related to classifying sexual abuse disclosure for #MeToo movement (Homogeneous MTL) or emotion classification task (Heterogeneous MTL). The two objectives are weighted by a parameter  $\gamma$ , which controls the importance placed on the auxiliary task ( $1 - \gamma$  for the primary task).
Multitask learning frameworks are generally built using either of these two approaches: hard parameter sharing or soft parameter sharing. In a hard parameter sharing model (Caruana, 1997), both the primary and auxiliary tasks have a shared encoder followed by separate task-specific network branches, and the shared encoder is updated by both the tasks alternately. On the other hand, in the soft parameter sharing approach, tasks have different encoders with independent parameters, and the distance between their parameters is regularized using a regularization constraint (Duong et al., 2015; Yang and Hospedales, 2016), to encourage the parameters to be similar.
Flexible Cross-Stitched Parameter Sharing Architecture: We design our model so that the task-agnostic textual feature representations benefit from hard sharing while the regularization of the task-specific features can be learned according to task pair settings. We call our approach flexible cross-stitched parameter sharing, presented in Figure 2. Specifically, we train two separate models (one for each task) in tandem while also having a shared encoder that is updated by both of them and weighted joint learning of primary task decoder parameters that are tuned specifically for the task. This allows both the models to have their own set of parameters while also encouraging knowledge transfer via the shared encoder weights.
For each training pass of the primary task, the input representation  $e^{(p)}$  is passed through (a) stacked BiLSTM encoder and (b) stacked shared BiLSTM encoder. This results in two contextualized word representations  $(h_1^{(p)}, h_2^{(p)}, \ldots, h_n^{(p)})$  and  $(h_1^{(s)}, h_2^{(s)}, \ldots, h_n^{(s)})$ , where superscript  $(p)$  is used to denote the representations resulting from encoder in the primary task model and superscript  $(s)$  is used to denote the ones from shared encoder. We
calculate the weighted summation of these two representations -  $\tilde{h}^{(p)}$ , using two learnable parameters,  $\alpha^{(p)}$  and  $\alpha^{(s)}$  (where  $\alpha^{(p)} + \alpha^{(s)} = 1$ ), as formulated in Equation 7 to regulate the information resulting from the two encoders (Figure 2).
$$
\tilde {h} ^ {(p)} = \alpha^ {(p)} h ^ {(p)} + \alpha^ {(s)} h ^ {(s)} \tag {7}
$$
Such an approach to aggregate information flow from two encoders has facilitated success in prior Multitask learning settings as well (Rajamanickam et al., 2020; Dankers et al., 2019). As for our auxiliary task, we pass the embeddings  $e^{(a)}$  through only the shared encoder  $(h^{(a)} = h^{(s)})$ , followed by a dropout layer. We use this architecture for Heterogeneous MTL experiments. For Homogeneous MTL ones, we employ hard parameter sharing model due to statistical out-performance in this scenario. This technique consists of a single stacked encoder that is shared and updated by both tasks related to identifying narratives related to sexual abuse disclosures within #MeToo movement, followed by task-specific branches. The shared representations from the encoder are passed through the dropout layer.
These output representations (in the case of both Homogeneous and Heterogeneous experiments) are passed through respective BiLSTM decoders and dropout layers to get the final representation  $m^{(p)}$  and  $m^{(a)}$ , respectively for both the tasks. The auxiliary network branch is optimized using either Equation 5 (Class Balanced Focal Loss) or Equation 6 (Binary Cross Entropy), depending upon whether the auxiliary task is associated with identifying sexual abuse disclosure narratives or emotions. These output representations  $m^{(p)}$  and  $m^{(a)}$  are passed through a linear output layer to get unnormalized outputs  $o^{(p)}$  and  $o^{(a)}$  respectively. Sigmoid activation function is used for Sarcasm detection and the emotion classification task, and Softmax activation for others.
MTL framework traditionally improves generalization by leveraging the domain-specific information due to the relatedness of the tasks present in the training signals (Caruana, 1997); hence we use two publicly available datasets mined from Twitter: