Standardizing Distress Analysis: Emotion-Driven Distress Identification and Cause Extraction (DICE) in Multimodal Online Posts
Authors: gopendra Vikram singh
Soumitra Ghosh
Atul Verma
Chetna Painkra
Asif Ekbal
Venue: EMNLP 2023 Main (2023)
Keywords: Hate speech
Social media
Multimodal online posts
Distress content
Causal phrases
Emotional information
Zero-shot strategy
TL;DR: Emotion-Driven Distress Identification and Cause Extraction (DICE)

Abstract: Due to its growing impact on public opinion, hate speech on social media has garnered increased attention. While automated methods for identifying hate speech have been presented in the past, they have mostly been limited to analyzing textual content. The interpretability of such models has received very little attention, despite the social and legal consequences of erroneous predictions. In this work, we present a novel problem of \textit{Distress Identification and Cause Extraction (DICE)} from multimodal online posts. We develop a multi-task deep framework for the simultaneous detection of distress content and identify connected causal phrases from the text using emotional information. The emotional information is incorporated into the training process using a zero-shot strategy, and a novel mechanism is devised to fuse the features from the multimodal inputs. Furthermore, we introduce the first-of-its-kind \textit{Distress and Cause annotated Multimodal (DCaM)} dataset of 20,764 social media posts. We thoroughly evaluate our proposed method by comparing it to several existing benchmarks. Empirical assessment and comprehensive qualitative analysis demonstrate that our proposed method works well on distress detection and cause extraction tasks, improving F1 and ROS scores by 1.95\% and 3\%, respectively, relative to the best-performing baseline. The code and the dataset can be accessed from the following link: \url{https://www.iitp.ac.in/~ai-nlp-ml/resources.html\#DICE}.

Standardizing Distress Analysis: Emotion-Driven Distress Identification and Cause Extraction (DICE) in Multimodal Online Posts
Gopendra Vikram Singh", Soumitra Ghosh", , Atul Verma, Chetna Painkra and Asif Ekbal
Department of Computer Science and Engineering, IIT Patna, India
(gopendra_ @copendra_1921e es ),aif) @ @ittp.ac.in,
atul. verma cccaaaaaaaccccceaaaaaaa55555 @gmail.com
Abstract
spreading hate remarks (Ribeiro al., 2018). Abu- sive speech based on race, religion, and sexual ori- entation is becoming more common (Karim et al., 2020). Automatic identification of hate speech and raising public awareness are critical tasks (Karim et al., 2020). Manually evaluating and validating large volume of web information, on the other hand, is time-consumin and intee-intensive.
Due its growing impact on public opinion, hate speech social media has garnered in- creased attention. While automated methods for identifying hate speech have been presented in the past, they have mostly been limited to analyzing textual content. The interpretability of such models has received very little atten- tion, despite the social and legal consequences erroneous predictions. In this work, we present a novel problem of Distress Identifi- cation and Cause Extraction (DICE) from mul- timodal online posts. We develop multi-task deep framework for the simultancous detec- tion of distress content and identify connected causal phrases from the text using emotional information. The emotional information is in- corporated into the training process using a zero-shot strategy, and a novel mechanism is de- vised to fuse the features from the multimodal inputs. Furthermore, we introduce the first-of- its-kind Distress and Cause annotated Multi- modal (DCaM) dataset of 20,764 social media posts. We thoroughly evaluate our proposed method by comparing it to several existing benchmarks. Empirical assessment and com- prehensive qualitative analysis demonstrate that our proposed method works well distress detection and cause extraction tasks, improv- ing F1 and ROS scores by 1.95% and 3%, re- spectively, relative to the best-performing hase- line. The code and the dataset can accessed from the following link: https://www iitp. ac. wiwiiinniiiii -aannnnppmmmmeeee mllsoo/usssssees......mmm hDm.#
You diavee te lonely
at Trump Haters dot com.
Meunwhile, in the Land of the #PinkSwaclkk....
He Jucked around and found out
HOrangeManBad Muh Nickel-per Post Khesticehariiors Don't really have be #Ulriiiee you!
Muh
survive this and hopeffully be better person
Thankfully
t Vegas VapeShop terime tlarma
Figure 1:
Sample Distressed posts from our DCaM dataset. Span annotation is highlighted in red.
Modern language models excel over traditional machine learning and neural network-based ap- proaches but lack transparency in output transfor- mation, posing limitations in domains, such as the military, medical research, and internet content monitoring. Robust models for monitoring dis- tressed content online require multimodal inputs. In our "DCaM" dataset, Figure highlights the sig- nificance of multimodality and span annotations in comprehending distress content. While both posts labeled as "distressed," the first post may not of- fer sufficient information based on textual content alone. However, the second post, with both picture and text, provides clarity, and the span annotation aids in analyzing the manifestation of distress.
1 Introduction
The exponential expansion of microblogging sites and social media not only empowers free expres- sion and individual voices, but also allows individ- uals to exhibit anti-social conduct (EISherief et al., 2018), such as cyberbullying, online rumours, and
This necessitates a shift in viewpoint away from performance-based models and toward inter- pretable models. We address model explainabil- ity by jointly learning the target classification of a multimodal social media post as Distressed or Non-
[*] These authors contributed equally to this work and are the joint first authors.

---

distressed and extracting the reasons for the classi- fication decision (for the Distressed class) from the textual input. The prime focus of this study is to comprehend the causes associated with any form of offensive content (hate, offensive, abusive, etc.). We club all the connotations of offensive content under the category distressed.
word embedding techniques (McKeown and Mc- Gregor, 2018). Recent models based on Transform- ers (Vaswani et al., 2017) have had extraordinary success. Since this is essentially a classification problem, BERT (Bidirectional Encoder Represen- tations from Transformers) (Devlin et al., 2018) has found widespread use in the field of hate speech identification. Ranasinghe al. (2019) showed that BERT-based model performed better than mod- els based on recurrent neural networks (RNNSs) Zaidan et al. (2007) first proposed the use of ra- tionales, where human annotators highlight text that supports their classification decision. This work was enhanced by Yessenalina et al. (2010) to provide self- eeff-genrrating rationales. An encoder- generator system for quality rationales without annotations
The main contributions are summarized below:
1. We propose the novel task of Unified Distress Identification and Cause Extraction (DICE) from multimodal online posts.
2. We develop a multi-tast deep framework for the simultaneous detection of distress content and identify connected causal phrases from the text using emotional information.
3. We devise zero-shot strategy to dynamically incorporate emotional information into train- ing and propose a novel fusion mechanism to infuse the features of multimodal inputs.
presented in Lei et al. (2016).
Mathew et al. (2021) used dataset rationales fine-tune BERT address bias and explainability.
Recent research has shifted towards accommo- dating multimodal content, with a focus on de- tecting hate speech and objectionable material various media. Gandhi et al. (2019) developed computer vision-based techmique for identify- ing offensive and non- offennsnev images in large datasets. Kiela et al. (2020) introduced a novel chal- lenge for multimodal hate speech detection in Face- book memes. Rana and Jha (2022) employed the Hate Speech Recognition Video Dataset to identify emotion-based hate speech in a multimodal context. Karim et al. (2022) presented a dataset for detect- ing hate speech in Bengali memes and text. Fersini et al. (2022) discussed SemEval-2022 Task 5, fo- cusing on identifying misogynous memes through text and images, including sub-tasks for recogniz- ing misogynous content and categorizing types of misogyny. Hee al. (2022) investigated multi- modal hateful meme detection models and their ability to capture derogatory references in both images and text. Additionally, Cao et al. (2022) introduced PromptHate, a model that leverages pre- trained language models with specific prompts and examples for hateful meme classification.
4. The first Distress and Cause annotated Mul- timodal (DCaM) corpus is created consisting over 20,764 social media posts.
5. Resources are ooun-soureed to aid research.
The rest of the paper is organized as follows. Section 2 summarises some previous works in this area. We discuss the dataset preparation in Sec- tion 3. Section 4 addresses our proposed methodol- ogy in depth, followed by the results and analysis in Section 5. Finally, we conclude our discussion in Section 6 and define the scope of future work.
2 Related Work
Several approaches have been suggested to identify online hate speech (Burnap and Williams, 2016; Zhang et al., 2018; Qian et al., 2018). The current interest in hate speech research has led to the avail- ability of datasets in several languages (Sanguinetti al., 2018; Ousidhoum et al., 2019) and differ- ent computational ways to counteract online hate (Mathew et al., 2019; Aluru et al., 2020). Text-, user-, and network-based traits and characteristics that identify bullies have been extracted in (Chatza- kou et al., 2017). Deep learning Lundberg and Lee (2017); Founta al. (2019) has been used extensively to identify hate speech keyword identi- fication, sexism, bullying. trolling, and racism.
Even though multimodal studies on offensive content have gotten a lot of attention, this study is the first to look at how to find distressed content on social media and figure out what caused it. Ad- ditionally, this work presents the first Distress and Cause annotated Multimodal (DCaM) corpus of social media posts to the research community.
Recent research on identifying hate speech has made use of deep learning techniques, including neural networks (Han and Eisenstein, 2019) and

---

Datasets
Total
Language
Multinodal?
Rationales?
Wascem and Hovy (2016)
Racist, Sexist, Normal
16,914
English
Davidson et (2017)
Hate Speech, Offensive, Normal
24,802
Fingiish
Founta et ul. (2018)
Abusive, Hateful, Normal, Spam
80, 80,000
English
X
Ousidhoum et al. (2019)
Lubels for five different uspocts
13,000
English. French, Arabic
X
X
Mathew et al. (2021)
Hate Speoch, Offensive, Normal
20,148
English
X
DCaM (ours)
Distressed (Hate-Offensive-eeeee Abbsivs.
20,764
English
Non-distressed
(causes)
Table Comparisons different distress datasets.
3 Dataset
by five annotators* (DI task), and then majority vot- ing applied decide the final label.
We discuss the data collection and annotation de- tails in the following subsections.
There are kinds of annotations in our dataset. First, whether the post is Distressed or Non- distressed post. Second, if the text is considered as Distressed by majority of the annotators, we ask the annotators to highlight parts of the text that in- clude terms that might be a plausible basis for the provided annotation. These span annotations help us delve further into the manifestations of hatred or offensive speech.
3.1 Data Collection
We collect our dataset from sources where previ- ous studies (Davidson et al., 2017; Zannettou et al., 2018; Mathew et al., 2021) on hate speech have been conducted: Twitter and Gabl. The data was scraped from the top 5 trending topics on Twitter using selenium² to reduce the effects of sample bias. As for Twitter, cent of all collected tweets between October 2022 and December 2022. Using the textual mode of scraped tweets, we generated a list of the most fre- quent words, which we then used as tags to gather the posts from Gab. Please refer to Appendix Sec- tion A.1 for details on data collection from Gab, including keywords used for the DCaM dataset (see Table 8). To compile this data, we scoured Gab for posts between November and December 2022. Posts that have been deleted and reposted are not considered. We also remove links from posts to ensure that annotators can access all rele- vant information. A number of distress datasets are compared in Table 1.
selected the top 10 per-
Twitter
Gab
Total
Distressed
3248
5210
8458
Non Non-distressed
7066
5240
12306
Total
10314
10450
20764
Table 2: Dataset details
Whenever ire new tridge or soor moccase... AAliApriRak SSamiiul MiichaChadlo
Bradley MAGA Har He Panched a Police Office (Tsscal) ATDS #Assalli Arrngnessivss
3.2 Data Annotation
ENELY NEWS
To ensure the dataset consists of only English posts, we used the TextBlob library for language detec- tion and included only those identified as English. Additionally, nnn-English posts were flagged and excluded during annotation. Annotators were in- formed about the presence of hate offensive con- tent beforehand. Annotation guidelines from Poria et al. (2021); Ghosh et al. (2022c) were provided to assist annotators understanding the classification and span anmotation tasks. Each post was annotated
New York Police busted creep who Unidenified gunman caught on camera
secuilly assailted thone wameei incide
shoning man the back withou
Manhu
Figure 2: Samples from our dataset
For the Distressed Identification task, the Krip- pendorff's s a for the inter-annotator agreement is 0.66 which is much higher than other hate speech
'https://twitte: com/, httpp://gab.. com/ 't
seleniuu/ 'The annotation guidelines are discussed in Section A.2 of the Appendix
"2 Ph.D. Linguistics dogree holders, 2 Ph.D. students, and Undergraduate student from the Computer Science discipline

---

Disinsse denification
Exmittio
ImagessEmbedding
Embending
Softran
Sigrold
vategoriial peitetty
Dimension
W Residual sightts
Cless
Ponalty
Converging
Comvy
Conva
Conre
Anger Sed
6bx
"Diggntt
lomo
O O
Recembining
The slices
ResNiXt
Embedaing
sPr
Asto Encader
WFF
Dacoder
Clustering
Embeddingg
Ritation
Space
Self- Atfention
Unssen
O O
Text Embbedding
Retatoon
Vocab Graph
Deep SentiBank
ResNeXi
BERT
Graph Embedding
(b) Depiction of Clustering Module (top) and Interaction between Vocab graph and BERT embedding (bottom).
(a) Overall Architecture
Figure 3: Illustration of the proposed Depression Identification and Cause Extraction (DICE) framework
datasets (Ousidhoum et al., 2019; Mathew al., 2021). Following the work in (Poria et al., 2021; Ghosh et al., 2022c), we marked at most 3 causal spans for distressed post in the dataset. The fi- nal causal span is marked using the span-level ag- gregation upproach detailed in (Gui et al., 2016). We use the macro-Fl measure assess inter-rater agreement based on previous work on span extrac- tion (Poria al., 2021; Ghosh et al., 2022c), and achieve an Fl-score of 0.73, suggesting that the annotations are of high quality. Table contains further information about the dataset obtained. Fig- ure 2 shows samples of our dataset. The average number of tokens highlighted per distressed post is 8.55, and the average token per post is 25.43.
4.1 Problem Formulation
Given a post P= [si, S Sp1 composed of sequence of sentences (s), and each utterance be further decomposed into a sequence of words. p indicates the number sentences the post. The objective is to determine the post distressed or not (0 or 1) and to extract every plausible causal span that supports the prediction.
4.2 Proposed DICE Framework
Textual Encoder. Our textual encoder uses BERT followed by an ontology-based word graph. BERT extracts local information from text. On- tology is the backbone of knowledge graphs (KGs) (Song al., 2022), which give meta-data descrip- tions to guide the creation and completion of knowl- edge graphs. Additionally, relation descriptions contain semantic information that can be used to represent relations. During Graph Neural Net- work (GNN) message transmission, we embed text within ontology nodes. First, all the nodes are em- bedded using node embedding and text embedding follows:
4 Methodology
In this section, we illustrate our proposed DICE framework, which is a multitask system for Depres- sion Identification and Cause Extraction from mul- timodal social media posts. The system employs zero-shot strategy dynamically incorporate emo- tional information into training and presents a novel fusion mechanism to infuse the features from the multimodal inputs. The overall architecture of the proposed method is shown Figure 3a.
h h. and h4
Σ a: n=1
(1)

---

where W.F is word text embedding (BERT), W is graph embedding, z; depicts node (represent- ing word). ho concept in ontology. Fig- ure 3b illustrates the interaction between the vocab graph and BERT embedding to establish relation- ships. Our method enriches the lext-embedding and graph-embedding space, enabling the identifi- cation of previously unseen relationships between graph embeddings of the head and tail.
pair consisting of I and Tí. Cross attention employed combine the textual and visual features to create the Text infused visual features (Tv.). Taking into account the spatial properties of the channel-wise features, the query vectors (Q) are generated by convolution with kerenlls on each channel of Z; and then averaging (avg pooling) the feature maps as illustrated in Figure 4. Similarly, construct the Visual infused textual features (Vr) by exchanging I& and Tí. In particular, the key vectors (K) are produced by convolution with N*kernels on each channel of Zj and then averaging (average pooling) the feature maps.
gh)
Ta
where, Ta is aggregate relationship, g(*) is aggre- gate function, and is neighboring nodes for the missing node.
Cross-Attention. First, take the query vector from one modality (say image, I) and the key/value pair from the other (say text, T). To examine how text affects the image vector, we feed the query (4g) and textual key/value to self-attention (sel Alt()))
We use ResNet' to capture fa- cial expressions and visual surroundings for rich emotional indicators from the image in the input post. We separated the embedding dimensions and image data into groups to simplify the problem and make better use of the complete embedding space. Each learner will create a unique distance metric us- ing just subspace of the original embedding space and portion of the training data. By segmenting the network's 's embedding layer into /) consecutive slices, are able isolate D unique learners in- side the embedding space. After learner solutions converge, we aggregate them to obtain the whole embedding space. The merging is accomplished by recombining the slices of the embedding layer that correspond to the D learners. To ensure unifor- mity in the embeddings produced by various learn- ers, then perform fine-grained tuning across the entire dataset. The merged embeddings may be hampered by the gradients, which resemble white noise and would hinder training performance. This is called the "shattered gradients problem". To ad- dress this, residual weights (Balduzzi et al., 2017) provide the gradients with some spatial structure, which aids in training, as shown in Figure 3b.
Image Encoder.
Ta Query(T)
Tg. T. Key(T), Velue),Value(TT) Value(T) STA sel All To.
(3)
We filter noise from the output of the self-attention using the forget gate (a) and concatenate it with the linear layer's residual (c.f. Figure 4).
GT Conccat(iama
(Sr)))
(4) Finally, we pass the representations of all the modalities (i.e., text, and image) through another self-attention to know how much the image vector will be impacted by text [Crossm S SA(Gr,, I,)] Please note that bolded I in Cross,r represents the impacted modality (i.e., 1). Similarly, we compute Cross,r and concatenate Pross-attentie multimodal features.
them to obtain the
Final Fusion. Although. the Ty and VT can in- dependently conduct image-text multimodal recog- nition, to further enhance the model's performance, we upply self- attention to fuse the two aforemen- tioned feature vectors.
Inter-modal Fusion (IMF). The IMF module exchanges information and aligns entities across modalities (text and image) to learn joint inter- modality representations. Figure illustrates the mechanism of inter-modal fusion.
Text infused visual features (and vice-versa). We use an external word embedding model to build high-level representations (T;') for image-text
Class Penalty. The inter-modal fusion unit ceives class penalty value to help the model un- derstand the link between a unified distress label and the input post. This improves the prediction of start and end tokens. The equations below origi-
//titt com/ jww./jjoaaanojssss.o
ResNet-

---

Self Attention
Textual aided (Injected) Visual
Visual sided (injected) Textual Features
Cross Attention
Query
Attention
Cross
Key
Pooling
Query
X Kornets
Pooling
Textual Weights
Aarnat
Textual
Weights
Weights
Text Embedding
Text Embedding
Text
Value
Visual Embedding
Image
Visual Embedding
Figure 4: Inter-modal Fusion
nally represent softmax and sigmoid:
where Lul and Lu2 is updated loss functions for softmax and sigmoid, respectively, 0 denotes the angle between weight W and feature / and denotes the amplifier function.
b. L =
exp Wlt
log
(5)
1=1
===
Emotion Features. We consider Ekman's 's (Ek- man, 1992) emotion classes and initialize them with the BERT (Devlin et al., 2018) vectors to rep- resent their semantic features.
L by
exp Wl: b;
Where, Rd the feature of tth sample; by is batch size; b, and by denote the bias; and W C IRden denotes the weight matrix. Information extraction tasks are notoriously difficult to find the decision boundary for the start and end markers of span, and a basic softmax/sigmoid classifier cannot manage this distinction. Some samples may be misclassified due to the classification boundary's ambiguity. This may require a faster convergence rate. We use the Insightface loss technique (Deng et al., 2019) to normalize the feature 4 and weight matrices. W to assess feature similarity based on the angle difference by which it maps the vector more closely. To converge the feature, it adds penalty value the angle.
Reconstruction Loss. An auto-encoder recon- structs adjective-noun pair (ANP) features and produces latent features while maintaining emotion information in the learned latent space to match label and ANP feature structures. By optimizing the following loss function, the auto-encoder input (A) and output (A) must be sufficiently close to identify its parameters.
C.e
t)) A(IMM..))))
Also, optimizing this loss results in lower- dimensional input features and high-accuracy feature reconstruction.
1
bx
exp exp aloosa(o00000
Adversarial loss. Our objective is to maintain the discriminative capacity of the combined fea-
Lul
log
b
exp
exp expo (0)
"To begin, we employ mid-level semantic representations of ANP features for the creation of an intermediary latent space. When provided with training image, we opt for the application of the pre-trained ANP detector, DeepSentiBank (Chen et al., 2014). to extract the ANP feature. To establish a proficient latent space conducive to concise representation of the original affective features, we emhrace the utilization of an auto-cncoder model.
1=1
(7)
1
1
L.2
bs
expafcas((+aa) (++)) (cos)()

---

Distress Identificttion
Cause Extraction
Models
KC
Inf
F
Modality
(%)
ACC. (%)
PM
JF
BERT-HateXplain
2.98
2.73
3.12
(T)
86.54
40.51
0.82
SpanBERT
3.26
3,47
3,55
DICE
70.15
31.28
ViL-BERT CC
83.75
84.78
CMSEKI
3.41
3.43
3.63
Visual BERT COCO
85.27
86.30
DICE (Proposed)
3.69
4.07
3,98
DICE (T+1)
87.71
89.53
41.31
45,48
0.69
0.85
0.88
(b) Results of human evaluation. Here, KC: Knowl- edge Consistency, Inf: Informativeness, F: Fluency
(a) Results across different modalities. Here, Text, Image
Table 3: Results of the DICE framework the DCaM dataset. Values in bold are the maximum scores attained.
ture of the text and visual i.e. LIM.A f)), and combine with the rich emotional structural data contained in feature dleemo)) This is accomplished by using an adversarial restriction that seeks to trick a discriminator network 'D such that the out- put A(IMF(a. t)) features are comparable as the ANP features:
CE tasks, we employ binary cross-entropy loss.
L Σ Ww
(9)
Here, represents the two tasks, DI and CE. The weights (Ww) updated using back propapropngttion for specific losses for each task.
Cade Ey(log Ey D(O(y))
Experiments and Results
Where (v) defines the combined feature of text and image iee. MF (a,t); and h(y) defines the la- tent feature space. The generator network (auto- encoder) minimizes this loss to learn how to gener- ate emotionally rich labels that closely match the ANP features, ensuring accurate label generation. Zero-shot loss. Suppose (x)) defines the com- bined feature of text and image i.e. MF(a,t), and <lemo) defines the semantic feature of the label. The objective here is to reduce the distance be- tween these two using the following function:
This section discusses the results and the analysis. Due to space constraints, we discuss the experimen- tal setup in Section A.3 and the evaluation metrics in Section A.5.1 in the Appendix.
5.1 Baselines
Our framework combines distress identification and cause extraction into a single automated sys- tem, utilizing classification and span detection. Due to the lack of suitable multimodal baselines with similar objectives, existing automated sys- tems were used for evaluation. We compare our proposed DICE approach and the presented DCaM dataset against various baselines, includ- ing BiRNN-Atth (Liu and Lane, 2016), CNN-GRU (Zhang et al., 2018) 8), GiRNN-Hate et al., 2021), BERT (Liu et al., 2019a), BERT- HateXplain (Mathew et al., 2021), SpanBERT (Liu et al., 2019b), and CMSEKI (Ghosh et al., 2022b), To thoroughly evaluate our approach on multi- modal inputs, employed two widely-used mul- timodal baselines, ViLBERT CC (Lu al., 2019) and Visual BERT COCO (Li et al., 2019), to assess the distress identification task in our datasct. We discuss the baselines briefly in Section A.4 of the Appendix.
L = (0( 2l
The zero-shot loss enhances the generation of ac- curate and emotionally rich labels by aligning the combined feature of text and image with the seman- tic feature of the emotion classes.
(Mathew
Joint Loss. The model is trained using unified loss function defined below:
Ljoint = Lodv + Lst + Lre
Emotion Label Prediction. For given post (text+image), our model will classify the labels using simple nearest neighbour (NN) search. Let suppose that the post and labels are fed into the embeddings obtain (MM F(a,t)) and ő(lemo).
((a, t)) - <(lemo)l13
5.2 Results and Analysis
4.2.1 Calculation of Final Loss
Table shows the results of the proposed DICE framework on the introduced DCaM dataset.
As illustrated equation 9, the model is trained using a unified loss function. For both the DI and
Specifically, we show the modality-varying results

---

Distress Identification (DI)
Cause Extraction (CE)
Models
F1 ( (%)
ACC. (%)
FM
PM
HD
JF
ROS
BiRNN-Altn (Liu and Lane, 2016)
75.47
76.71
26.21
30.41
0.51
0.69
0.74
CNN-GRU (Zhang et al., 2018)
76.13
78.11
27.13
30.92
0.53
0.70
0.75
BERT (Liu et al., 2019a)
81.78
82.41
31.32
36.38
0.58
0.75
0.78
BiRNN-HaleXplain (Mathew et ul., 2021)
77.71
78.58
28.51
32.16
0.54
0.72
0.76
BERlaHatcXplain (Mathew et al., 2021)
82.69
83.19
33.41
37.73
0.60
0.77
0.79
SpanBERT (Joshi al., 2020)
82.83
83.66
33.98
38.22
0.61
0.77
0.79
CMSEKI (Ghosh et al., 2022a)
84.17
85.31
36.39
38.22
0.64
0.80
0.81
DICE (Proposed)
86.12
86.54
38.74
40.51
0.66
0.82
0.84
Table Results from the DICE model and various baselines. Here, the bolded values indicate maximum scores.
in Table 3a. The bi-modal (Text+Image) configu- ration yields the best results, followed by the uni- modal network. The textual modality outperforms the others when compared independently, as texts have less background noise than visual sources. For the similar tasks, our results are consistent with prior studies (Hazarika et al., 2018).
any external information, outperforming other com- parable systems. However, it falls short by 2.88% FI for the DI task and ROS points for the CE task when compared to our DICE framework. Further- more, the DICE method managed to outperform the sophisticated state-of-the-an multimodal language models, ViL-BERT CC and Visual BERT COCO. The results analysis reveals that BERT, SpanBERT, and BERT-HateXplain exhibit notably lower perfor- mance in the task of cause extraction for offensive content. This observation underscores the inherent difficulty that even powerful language models face when comes to discerning crucial aspects, such as identifying causes, within offensive content.
Human evaluation. A qualitative human review was conducted on 300 randomly selected posts from the test dataset to assess the model'ss identified causes. The assessment used three well-defined measurements (Singh al., 2022), with scores ranging from 0 to 5 based on Fluency, Knowledge Consistency, and Informativeness?. Scores of were given to the most incorrect responses, while the best responses received a score of 5. In Table 3b, it can be seen that, compared to the various baselines, the proposed framework has done well for all the manual evaluation measures. Our sug- gested approach results in higher Knowledge Con- sistency score, ensuring that the extracted causal spans are consistent with annotated causal spans. The Informativeness and Fluency of our proposed framework is likewise of high quality. These re- sults demonstrate our model's 's strong ability to un- derstand offensive information and produce results comparable to human annotators.
Setup
F1 (%)
[T+1]-ry
84.68 (-3.03)
[T+I]av
83.92 (-3.79)
[T+I]-ns
85.74 (-1.97)
[T+I]-IM
82.76 (-4.95)
[T+I]-IMF-DS
82.11 (-5.60)
81.40 (-6.31)
(T+F)-IMF-DS+AE--W
79.98 (-7.93)
DICE (Proposed)
87.71
Table 5: Results of ablation experiments. The fall in scores are shown brackets. IMF: Inter-Modal Fu- sion, AE: Autoencoder, DS: DeepSentiBank, VG: Vo- cab Graph, Tv: Text infused visual features, VT: Visual infused textual features
Comparison with Existing Works. Table demonstrates that CMSEKI the best-perfrrming baseline, which not unexpected considering that it grasps the input information using common- sense knowledge from external knowledge sources. However, the DICE model beats CMSEKI on all measures, especially by 1.95% F1 for the DI task and 3 ROS points for the CE task. SpanBERT is the baseline that does not employ
Ablation Study. To examine the importance of the different modules in DICE framework, we re- move the constituent components, one at time, and report the results in Table 5. Specifically, we conduct five ablation experiments: first, we replace the proposed Inter-modal fusion (IMF) mechanism by linear concatenation to fuse multimodal fea- tures ([T+I].ıMr) Next, we independently eval-
WW discuss the definition of each metric in Appendix A.5.2

---

Modil
Label
1. Human Aunolator
Coloraoo scholl bus driser facos criminal charges for plapping 1ogearoold stadent in the fure For not wearing a mask
Distresed
BERT GERT-HascXplan
Colorado school bus driver facos inriminal icriminol-chaggs for slapping 10-yor-oodd student the face wearing mask
Distrossed
SpanRERT
Colando school bus faces criminal phurgges for slopping 10-yaar-old student the
wearing misk
Distresal
CMSEKI
Colorado school driver criminal charges for slapping 10-year-old student the
not weuting
Distressed
Proposed
Colorado school bus driver faces criminal charges foe alepping W-year-oldd itudent in the face for not wearing a mask
Distressed
Annotutor
you need Eviolence to defend against
ideas, your aren' Sterroria they are eseffDefonce) Food For Thought.
Distressed
HeKI--ueexllin
If you seed
Rviolence to defond kgainst Neewish ideus, your ideas uren't Rtezzorim the are #Seli1Detencee Food For Thought.
Distressed
SponBERT
If you need
Pviolence to defond against klewish ideas. your ideas aren't Aterzorism they are #SeliDetinnee. Food For Thought.
Non- Distressed
CMSEKT
If you neal
Aviolence defondl apains Rlewish ideus ideas oren't Rereinim they are #SellDefence, Food For Thought.
Distrosell
Proposed
you Aviolence defond apainst #lewish ideas, your ideas aren Aerrarism they #SefiDefonce. Food For Thought.
Distrossed
Table 6: Sample predictions from the various systems
uate the impact of one modality on the other by
Qualitative Analysis. We thoroughly examined the predictions made by the different systems. Con- sider the examples in Table 6. The top row dis- plays the tokens (or 'causes') that human anno- tators noted and that they consider representing the causes. for the post being Distressed. The next four rows show the extracted tokens from the various models. We observe that the proposed DICE model correctly categorizes the examples as distressed and also extracts good-quality causal spans. In the second example, we observe that although the SpanBERT model extracts partial causal span correctly, assigns the wrong label -Non-distressed), We also analyze the cases where the proposed model performs poorly. In the interest of space, we present the discussion in the Appendix (section A.6).
removing Tv and VT one by one. We observe from Table that removing the text-infused vi- sual features (Tv) has a more detrimental effect on the system's 's performance compared to remov- ing the visual infused text features (VT). Next, we remove DeepSentiBBnk sahi kya h(DS) along- side IMF ((T+I)-MMF+Ds, and, finally, we substi- tute the proposed IMF, DS and AE mechanism by linear concatenation to fuse multimodal features ([T+]]-MMP+ DD We observe notable fall in scores when either of these modules removed from the DICE approach, especially when we re- move the IMF+DS+AE module. This establishes that all components of the DICE model developed for multimodal data contribute to the success of the defined tasks in zero-shot environment.
Basic Model
Ludo
Lre
Lat
(%)
6 Conclusion
82.98 (-4.73)
86.01 (-1.72)
/
85.51 (-2.20)
In this work, we present novel problem of Dis- tress Identification and Cause Extraction (DICE) from multimodal online posts. We develop a multi- task, deep framework for detecting distress content and identifying associated causal phrases from text using emotional information. We devise zero-shot strategy dynamically incorporate emotional in- formation into training and propose a novel fusion mechanism infuse the features of multimodal inputs. Furthermore, we introduce the first Distress and Cause annotated Multimodal (DCaM) corpus, consisting of over 20,764 social media posts. We illustrate the effectiveness of our method by com- paring it to several state-of-the-an baselines. When compared to human performance, the present state- of-the-art models perform poorly, which serves emphasize the difficulty of the task at hand. We believe our work will advance multimodal reason- ing and comprehension while also assisting in the resolution of a significant real-world problem.
83.70 (-4.01)
87.71
Table 7: Affect of different loss functions. Basic model combines semantic features via zero-shot loss function
To investigate the significance of the loss func- tions in DICE, we remove them one by one and re- port the results in Table 7. the first ablated model, we remove all three loss functions (i.e., Ladv. Cre, and L.t). We remove the Lre loss function in the second model and the Lado adversarial function in the third. In the fourth model, we remove Lado and Cre. When any of these losses is eliminated from DICE, we see a performance decline when com- pared to the proposed method. The performance drop is the largest (4. (4.73%) when all three losses are climinated. Clearly, loss functions play a crucial role in training the entire model end-to-end.

---

Limitations and Future Scope
concept classification with deep convolutional neural networks. arXiv preprint arXiv:1410.8556
Due to the low prevalence of hate speech on social media (approximately 3% of messages are hateful), (Fortuna and Nunes, 2018)), we scrape posts by searching for hate words increase the likelihood of encountering hate-offensive content. This may have invited some undesired sampling bias while constructing the dataset. Additionally, emoticons and other non-standard symbols like S are often used in current online interactions. One potential research direction is to use these neglected visual features of text information to adapt to more realis-
Thomas Davidson, Dana Warmsley, Michael Macy, and Ingmar Weber. 2017. Automated hate speech de- tection and the problem of offensive language. Proceedings of the international AAAI conference on web and social media, pages 512-515.
Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. 2019. Areface: Additive angular margin loss for deep face recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4690-4699.
tic settings.
Jacob Devlin, Ming-Wai Wei Chang. Kenton Lee, and Kristina Toutanova, 2018. Bert: Pre-training of deep bidirectional transformers for language understand- ing. arXiv preprint arXiv:1810.04805.
Ethical Consideration
We created our resource using publicly accessible social media postings. We adhered the data use guidelines and did not infringe on any copyright problems. Our Institutional Review Board also reviewed and approved this research. We make the code and data accessible for research purposes through an appropriate data agreement mechanism.
Paul Ekman. 1992. An argument for basic emotions. Cognition & emotion, 603-4):169-200.
Mai EISherief, Vivek Kulkarni, Dana Nguyen, William Yang Wang, and Elizabeth Belding. 2018. Hate lingo: A target-based linguistic analysis of hate speech in social media. In Proceedings of the Inter- national AAAI Conference on Web and Social Media, volume 12.
References
Elisabetta Fersini, Francesca Gasparini, Giulia Rizzi, Aurora Saibene, Berta Chulvi, Paolo Rosso. Alyssa Lees, and Jeffrey Sorensen. 2022. Semeval-2022 task 5: Multimedia automatic misogyny identification. In Proceedings the 16th International Workshop on Semantic Evaluation SeemEval-2022) pages 533- 549.
Sai Saketh Aluru, Binny Mathew, Punyajoy Saha, and Animesh Mukherjee. 2020. Deep learning models for multilingual hate speech detection. arXiv preprint
arXiv:2004.06465.
David Balduzzi, Mareus Frean, Lennox Leary, JP Lewis. Kurt Wan-Duo Ma, and Brian McWilliams. 2017. The shattered gradients problem: If resnets are the answer, then what is the question? In International Conference on Machine Learning, pages 342-350.
Paula Fortuna and Sérgio Nunes. 2018. A survey on automatic detection of hate speech in text. ACM Computing Surveys (CSUR), (-0)
PMLR.
Antigoni Maria Founta, Despoina Chatzakou, Nicolas Kourtellis, Jeremy Blackburn, Athena Vakali, and Ilias Leontiadis, 2019. A unified deep learning chitecture for abuse detection. Proceedings of the 10th ACM conference web science, pages 105- 114.
Pete Burnap and Matthew Williams. 2016. Us and them: identifying cyber hate on twitter across mul- tiple protected characteristics. EPJ Data science, 5:1-15.
Rui Cao,, Roy Ka-Wei Lee, Wen-Haw Chong, and Jing Jiang. 2022. Prompting for multimodal hateful meme classification. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Process- ing, pages 321-332.
Antigoni Maria Founta, Constantinos Djouvas, De- spoina Chatzakou, Ilis Leontiadis, Jeremy Black- burn, Gianluca Stringhini, Athena Vakali, Michael Sirivianos, and Nicolas Kourtellis. 2018. Large scale crowdsourcing and characterization of twitter abusive behavior. In Twelfth International AAAI Conference Web and Social Media.
Despoina Chatzakou, Nicolas Kourtellis, Jeremy Black- burn, Emiliano De Cristofaro, Gianluca Stringhini, and Athena Vakali, 2017. Mean birds: Detecting aggression and bullying on twitter. In Proceedings of the 2017 ACM on web science conference, pages 13-22.
Shreyansh Gandhi, Samrat Kokkula, Abon Chaudhuri, Alessandro Magnani, Theban Stanley, Behzad Ah- madi, Venkatesh Kandaswamy, Omer Ovenc, and Shie Mannor. 2019. Image matters: scalable detec- tion of offensive and non-compliant content/logo product images. arXiv preprint arXiv:1905.0m meae:990550022 0223.4
Tao Chen, Damian Borth, Trevor Darrell, and Shih- Fu Chang. 2014. Deepsentibank: Visual sentiment

---

Soumitra Ghosh, Asif Ekbal, and Pushpak Bhat- tacharyya. 2022a. A multitask framework to detect depression, sentiment and multi-label emotion from suicide notes. Cognitive Computation, 14(1):110- 129.
Douwe Kiela, Hamed Firooz, Aravind Mohan, Vedanuj Goswami, Amanpreet Singh, Pratik Ringshia, and Davide Testuggine. 2020. The hateful memes chal- lenge: Detecting hate speech in multimodal memes. Advances in Neural Information Processing Systems, 33:261 1-2624.
Soumitra Ghosh, Asif Ekbal, and Pushpak Bhat- tacharyya. 2022b. A multitask framework to detect depression, sentiment and multi-labe emotion from suicide notes. Cogn. Comput., (4(1):110-122)
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd Inter- national Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Soumitra Ghosh, Swarup Roy, Asif Ekbal, and Pushpak Bhattacharyya. 2022c. Cares: Cause recognition for emotion in suicide notes. In European Conference on Information Retrieval. pages 128-136, Springer.
Tao Lei, Regina Barzilay, and Tommi Jaakkola. 2016. Rationalizing neural predictions. In Proceedings of the 2016 Conference on Empirical Methods in Natu- ral Language Processing, pages 107-117.
Lin Gui, Dongyin Wu, Ruifeng Xu, Qin Lu, and Yu Zhou. 2016. Event-driven emotion cause extrac- tion with corpus construction. In Proceedings of the 2016 Conference on Empirical Methods in Natu- ral Language Processing, pages 1639-1649, Austin, Texas. Association for Computational Linguisties.
Liunian Harold Li. Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. 2019. Visualbert: A sim- ple and performant baseline for vision and language. arXiv preprint wxxiv:190.003555
Xiaochuang Han and Jacob Fisenstein. 2019. Unsu- pervised domain adaptation of contextualized em- beddings for sequence labeling. Proceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing and the 9th International Juint Conference on Natural Language Procexsing (FMNLP-IJCNIP)), pages 4238-4248.
Bing Liu and Ian Lane. 2016. Attention tteention-based recurrent neural network models for joint intent detection and slot filling. arXiv preprint arXiv: 01444.
Yinhan Liu, Myle Ott. Naman Goyal, Jingfei Du Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019a. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint arxiv 1697.
Devamanyu Hazarika, Soujanya Poria, Amir Zadeh, Erik Cambria, Louis-Philippe Morency, and Roger Zimmermann. 2018 Conversational memory net- work for emotion recognition in dyadic dialogue videos. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technolo- gies, NAACI-HIT 2018, New Orleans, Louisiana, USA, June 1-6, 2018, Volume I (Long Papers), pages Association for Computational Linguis-
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019b. Roberta: robustly optimized BERT pretraining approach. CoRR, 1907.116999
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019. Vilbert: Pretraining task-agnostic visiolinguis- tic representations for vision-an and language tasks, Ad- vances in neural information processing systems, 32.
tics.
Ming Shan Hee, Roy Ka-Wei Lee, and Wen-Haw Chong. 2022. On explaining multimodal hateful meme de- tection models. In Proceedings of the ACM Web Conference 2022, pages 3651-3655.
Scott M Lundberg and Su-In Lee. 2017. A unified ap- proach to interpreting model predictions. Advances in neural information processing systems, 30.
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoye, and Omer Levy. 2020. Span- BERT: Improving pre- tre-trainin by representing and predicting spans. Transactions of the Association for Computational Linguistics, 8.
Binny Mathew, Punyajoy Saha, Hardik Tharad, Subham Rajgaria, Prajwal Singhania, Suman Kalyan Maity, Pawan Goyal, and Animesh Mukherjee. 2019, Thou shalt not hate: Countering online hate speech. Proceedings of the international AAAI conference on web and social media, volume 13, pages 369-380.
Md Karim, Sumon Kanti Dey. Tanhim Islam, Bharathi Raja Chakravarthi, et al. 2022. Multimodal hate speech detection from bengali memes and texts. arXiv preprint 2204 10196.
Binny Mathew, Punyajoy Saha. Seid Muhie Yimam. Chris Biemann. Pawan Goyal, and Animesh Mukher- jee. 2021. Hatexplain: A benchmark dataset for ex- plainable hate speech detection. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 14867-14875.
Md Rezaul Karim, Bharathi Raja Chakravarthi, John P McCrae, and Michael Cochez. 2020. Classification benchmarks for undeu-res bengali language based on multichannel convolutional-Istm network. In 2020 IEEE 7th International Conference on Data Science and Advanced Analytics (DSAA), pages 390- 399. IEEE
Rohan Kshirsagarl Tyrus Cukuvacl Kathleen McKe- own and Susan MeGregor. 2018. Predictive embed- dings for hate speech detection on twitter. EMNLP 2018, page 26.

---

Nedima Ousidhoum, Zizheng Lin, Hongming Zhang. Yangqiu Song, and Dit- Yan Yeung. 2019. Multi- lingual and multi-aspect hate speech analysis. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNL.-IJCNHL) pages 4675-4684.
Ainur Yessenalina, Yejin Choi, and Claire Cardie. 2010. Automatically generating annotator rationales to im- prove sentiment classification. In Proceedings of the ACL 2010 Conference Short Papers, pages 336-341.
Omar Zaidan, Jason Eisner, and Christine Piatko. 2007. Using "annotator rationales" to improve machine learning for text categorization. In Human language technologies 2007: The conference of the North American chapter of the association for computa- rional linguistics; proceedings of the main confer- ence, pages 260-267.
Soujanya Poria, Navonil Majumder, Devamanyu Haz- arika, Deepanway Ghosal, Rishabh Bhardwaj, Sam- son Yu Bai Jian, Pengfei Hong. Romila Ghosh, Ab- hinaha Roy, Niyati Chhaya, et al. 2021. Recognizing emotion cause in conversations. Cognitive Computa- tion, 13(5):1317-133
Savvas Zannettou, Barry Bradlyn, Emiliano De Cristo- faro, Haewoon Kwak, Michael Sirivianos, Gianluca Stringini, and Jeremy Blackburn. 2018. What is gab: A bastion of free speech or an alt-right echo chamber. In Companion Proceedings of the The Web Confer- ence 2018, pages 1007-1014.
Jing Qian, Mai ElSherief, Elizabeth Belding, and William Yang Wang. 2018. Hierarchical evae for fine-gran ameeggriieed hate speech classification. In Proceed- ings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3550-3559,
Ziqi Zhang, David Robinson, and Jonathan Tepper. 2018. Detecting hate speech on twitter using a convolution- based deep neural network. In Eu- ropean semantic web conference, pages 745-760. Springer.
Aneri Rana and Sonali Jha. 2022. Emotion based hate speech detection using multimodal learning. arXiv preprint arXiv:2202.062) 062/8.
Tharindu Ranasinghe, Marcos Zampieri, and Hansi Het- tiarachchi. 2019. Brums at hasoc 2019: Deep learn- ing models for multilingual hate speech and offensive language identification. In FIRE (working notes), pages 199-207.
A Appendix
We discuss the implementation details and present supporting details of the considered baselines and the human evaluation metrics. We also discuss a vivid qualitative analysis that compares our model's ss predictions with the best-pergorming baselines.
Manoel Horta Ribeiro, Pedro H Calais, Yuri A Santos, Virgílio AF Almeida, and Wagner Meira 2018. Characterizing and detecting hateful users on twitter. In Twelfth international AAAI conference web and social media.
states raped could racis st
terror r
others
ica
worl 1 dffamily
americo
Manuela Sanguinetti, Fabio Poletto, Cristina Bosco, Viviana Patti, and Marco Stranisci. 2018. An italian twitter corpus of hate speech against immigrants. In Proceedings of the eleventh international conference on language resources and evaluation (LREC 2018).
e
ukraine
state
Treesde
first white
te
group
crimes
Fforner
deobiden
texase
elonrusk
alliary prusitent
trump
cipray
never
today
system
attack
media
years
Dountry
Gopendra Vikram Singh, Mauajama Firdaus, Shruti Mishra, Asif Ekbal, et al. 2022. Knowing what to say: Towards knowledge grounded code-mixed re- sponse generation for open-domain conversations. Knowledgee Based Systems, 108900
trying
annarr
another
exposed
sm
Fatarent
forget
harassed
curcing
black
cer
mother
think
stillcrisis
Sinceearres arreeted
twitter
political
U
S
abuse
Stamis
party
e
every
woman
forcesterroriat
Ran Song, Shizhu He, Suncong Zheng, Shengxiang Gao, Kang Liu, Jun Zhao, and Zhengtao Yu. 2022. Ontology-guided and text-enhanced representation for knowledge graph zero-shot relational learning. In
murde
deree
quote
tweet
corr uurrupt
sexual
Bheakin
school
comine
raily Support
singh
ligto
watch
hindu
ICLR 2022 Workshop on Deep Learning on Graphs for Natural anguuge Processing.
po Lice
replying
pleasine
violence
Boang
scovid
right without
court
Tussia death
would
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information prommatsipssssssiggg systems, 30.
terroricss
Figure 5: Word Cloud from Distressed posts.
A.1 Word characteristies
Zeerak Waseem and Dirk Hovy. 2016. Hateful symbols or hateful people? predictive features for hate speech detection on twitter. In Proceedings of the NAACL student research workshop, pages 88-93.
We generate word clouds to graphically represent the word frequencies that appear more frequently in the Distressed and Non-distressed posts. The

---

bigger the term in the visual, the more often it ap- peared in user descriptions. Figures 5 and 6 show the word clouds generated from the 100 most fre- quent words of each class. The difference word choices for the distinct classes is evident from the figures. Table 8 shows some keywords used for crawling posts from Twitter and Gab to develop the DCaM dataset. Initially, we randomly crawled around 5000 posts each for period of 1 week from both Twitter and Gab and performed topic model- ing to fetch the trending topies. We randomly use a subset these topics to crawl posts for our dataset. From the collected posts, we create a bag of fre- quently occurring hashtags and use the generated set to crawl further posts. We take care of non- repetition in the collected posts by maintaining the post IDs. Lastly, to supplement the lack of offen- sive posts being crawled, we use the synonyms of the words 'hate', and "offensive" and use them as tags (like for the word "offensive" an example syn- onym could be 'insult' and gab URL that can be used: ht /ttpss//gal laa.a..ooa.............. hamm/tgs//aa insult) to extract posts during data scraping.
Trending Hashtags
#MusicBankinChie, #RichaChadha, #SateFlighhoursstrrnnnn
#Casteist BCCI, #TheWhitel.oong lotus, #lveesMPPnnBennallu,,
#LopezMartin, #mondaythoughts, #TradersWitDDlhhBBB.
#RiotGamesONE, Athursdayvibe,, /WaachingChcckerOPPrmee,
#NintendoSwicch202,, CAAPProiccacoomupiinn
#MumbaiiAttacks, MMMoodyMotivationnn #KashmirFiles,
#IndigoByRML #pac 2championship. #GoBlue,
#CujaratFlectioneeeaal Payeffy #TheCameAwands
Trending Topics
Thanksgiving, Galwan, Shame On You NBT, False News Not Required, Media SoldOut InSS
Fake News Factory. Modiji FastTrack SSRCase, Never Forget, Spiritual Revolution, Cause of Conspiracy, Hurdle For Missionaries, Israel, One Future, Presidency,
SSRCulprits Whitewash Shameful, APOLOGIZE TO LISA,
Hunter Biden, James Gunn, Christmas, Ukraine
Hate-Offensive Synonyms
toxic, illegal, slave, panic, victim, crisis, assault, protect, terror, protest, teach, justiec, sad, alone, broken, feelings, mood, stressed, depression, pain,
ugly, attack, shameless, stupid, poverty. revenge, hell, poor, liers, suffer, violence
Table 8: Sample keywords used for scarping posts construct DCaM.
people
season trailer binding Ruiter foa
g
images
that is, if there was no stated causal span. Two human experts-graduaat students with adequate task
aud1obook
a
suppor's
starring
blogspa
your Instagras
every post. We used the union of candidate spans from distinct annota- tors as the final causal span only when the size of their intersection was at least 50% of the size of the smallest candidate span. third annotator was brought in if the final span could not be determined from the previous spans. This third annotator was similarly told to choose shorter spans over larger spans where they could adequately depict the rea- son without losing any information.
igned
firs
business
domeste
tic
Barnesandnoble
bitchyte
bOOKS
overseas
seas
music
tnerid
design
we
verra
aovie
today
amazon
wa
watch
factor
india
ed
edition
Video
Sove
goog
1e
artist
poster
special
summmna
pape
pring
ba
aaruerr
print
K
trump
story
thank
painting
numans
happy
photography
follow
wa
Walmart
right
great
ebook
t
bluray
bay
years
wouldan
funny
pa
series
wor
Porldavan
the
VISIT
comedy
lab
l'
thath
neyne,
y
bars
barnes
ranes
From
nobel
1
A.3 Experimental Setup
website
chris
tmas
We use PyTorch", a Python-basee deep learning package, to develop our proposed model. We con- duct experiments with the BERT import from the huggingface transformers package. To establish the ideal value of the additive angle x, which affects performance, five values ranging from 0.1 to 0.5 were examined. The default value for x is 0.30. We amplification value as 64. All experiments are carried out on an NVIDIA GeForce RIX 2080 Ti GPU. We conducted grid search across 200 epochs. We find empirically that our Embedding
youtube
podcastiery
released
two
human
powe
intertainment
henderson
rueble
nomen
Lanooooooorrf
Figure 6: Word Cloud from Non-distressed posts.
A.2 Annotation Guidelines
Our annotation guidelines are rooted in the works of (Poria et al., 2021; Ghosh et al., 2022c). The an- notators were instructed to identify the set of causal spans that accurately depict the reasons for a post being tagged as distressed given an input post with that label. The annotators annotated a post with the No_ccuse tag if the cause of the post was latent,
$nttps://nytorch.org//
"nttps://huggingfe co htt.r//ac...oooooottaaaareeeee
index

---

size is 812 bytes. We use Adam (Kingma and Ba, 2015) for optimization. The learning rate is 0.05, and the dropout 0.5. The auto-latent encoder's dimension fixed at 812. The discriminator D con- sists of two completely linked layers and a ReLU layer and accepts 812-D input features. Stochastic gradient descent has a learning rate of le-4 and weight decay of 1e-3. with a momentum of 0.5. We perform 5 cross-validations of the DCaM dataset for training and testing purposes. We run our ex- periments for 200 epochs and report the averaged scores after runs the experiments to account for the non-determinism of Tensorflow GPU opera- tions.
tion values and the ground truth attention vector, detailed in (Mathew et al., 2021).
A.4.4 ViL-BERT CC
ViL-BERT CC (Lu et al., 2019) is variant of the ViL-BERT model that has been pre-trained on the Conceptual Captions (CC) dataset. Conceptual Captions a large-scale dataset containing image- caption pairs sourced from the web. By leveraging the rich and diverse data in CC, ViL-BERT CC is designed to understand and generate captions for images, enabling tasks such as image captioning. visual question answering, and image retrieval.
A.4.5 Visual BERT COCO
A.4 Baselines
Visual BERT COCO (Li et al., 2019) is a vari- ant of the Visual BERT model that has been pre-trained the Common Objects in Context (COCO) dataset. COCO is widely used dataset for object detection, segmentation, and captioning tasks. By pre-training on COCO, Visual BERT COCO learns to encode visual features and under- stand the context of images, enabling tasks such as object recognition, image captioning, and visual question answering. Visual BERT COCO enhances the model's s ability analyze visual content and perform various vision-related tasks.
We discuss the details of the considered baselines below. Similar to the DICE approach, to adapt the baselines to our multi-task scenario, we add a linear layer on top of the hidden-states output in the output layer of the CE task calculate span start and end logits. The output layer for the CE task employs sigmoid activation, in which the threshold value is set at 0.4.
A.4.1 BiRNN-Attention
The only difference between this model and the BiRNN model is the addition of an attention layer (Liu and Lane, 2016) after the sequential layer. In order to further train the attention layer outputs, we calculate the cross entropy loss between the atten- tion layer output and the ground truth attention.
A.4.6 BiRNN-HateXplain and BERT-HateXplain
We fine-tune the models made available by Mathew et al. (2021) on our DCaM dataset by changing the output layers as described earlier suit our task's objective.
A.4.2 CNN-GRU
Zhang et al. (2018) employed CNN-GRU to several hate speech
A.4.7 SpanBERT
achieve ofthe-of-the-art datasets. We add convolutional ID filters of win- dow sizes 2, 3, and 4, with 100 filters per size, to the existing architecture. We employ the GRU layer for the RNN component and max-pool the hidden layer output representation. This hidden layer is routed via fully connected layer to yield prediction logits.
SpanBERT (Joshi al., 2020) follows a differ- ent pre-training objective compared to traditional BERT system (e.g. predicting masked contiguous spans instead tokens) and performs better on question-ansse answering tasks. Following the work in (Ghosh et al., 2022c) where SpanBERT is used to solve mix of classification and cause extraction tasks, we fine-tune the SpanBERT base model on our DCaM dataset to meet our objective.
A.4.3 BERT
We fine-tune BERT (Liu et al., 2019a) by adding fully connected layer, with the output correspond- ing to the CLS token in the input. Next, to add attention supervision, we try to match the atten- tion values corresponding to the CLS token in the final layer to the ground truth attention. This is calculated using a cross-entropy between the atten-
A.4.8 Cascaded Multitask System with External Knowledge Infusion (CMSEKI)
We contrast the performance of our model with the state-of-the-ar CMSEKT system presented in
18 htttps: thub. http.////m/mmyaooo/aatttt...aaa

---

(Ghosh et al., 2022b). CMSEKI leverages common- sense knowledge in the learning process address multiple tasks simultaneously.
A.5.2 Human Eyaluation-basee Metrics
1. Fluency: This determines whether not the extracted span is fluent and natural. Natural and regular answers get a score of 5, whereas inarticulate ones receive a
A.5 Metric Definitions
The following metrics collectively provide a quan- titative assessment of how well our model performs in the task of extracting causal spans for manifesta- tions and determinants.
Knowledge consistency: This determines whether or not the produced answer has used the appropriate knowledge. the model gen- erates responses based on irrelevant informa- tion, it must get score of O, while the se- lection of pertinent knowledge must receive a score of 5.
A.5.1
Evaluation Metrics
Full Match (FM): This metric measures the percentage of predicted outputs that exactly match the ground truth outputs. In the con- text of span extraction, it would indicate the proportion of extracted causal spans that completely correct.
3. Informativeness: sess how informative the produced replies are. Here, score of 0 means that the replies are uninformative, and score of 5 means that they are.
This metric is used to as-
Partial Match (PM): This metric evaluates the similarity between the predicted outputs and the ground truth outputs, but it allows for some degree of variation. It takes into account cases where only portion of the prediction matches the ground truth. This can be useful when the extracted causal spans are almost correct but might have minor variations.
A.6 Error Analysis
Although our proposed DICE framework performs well in the majority of the test cases, still there are certain scenarios where it fails to make the correct predictions. We show some sample predictions from the test set in Table 9, In the first two in- stances, our model is able to partially predict the causal spans; however, in the first example, it fails categorize the post Distressed. is also be noted that the model extracted span in the sec- ond example seems to be more appropriate than the actual annotation by the human annotator. The model rightfully ignores the irrelevant information 'Video shows' and focuses on the relevant action part of the post. This illustrates our model's 's strong ability to comprehend offensive reasoning among diverse test cases. In the third and fourth examples, our model fails to extract any relevant cause from the given input. Moreover, in the third example, the model wrongly categorizes the post as Non- distressed. This can be due to the lack of sufficient context that hindered our model's comprehension ability for the given input.
Hamming Distance (HD): Hamming Distance is a measure of the difference between two strings of equal length. It counts the number of positions which the corresponding sym- bols in the two strings are different. In the context of causal extraction, it could represent the number of positions where the predicted and ground truth causal relationships differ.
Jaccard Similarity (JS): Jaceard Similarity is a measure of set similarity that calculates the ra- tio of the size of the intersection two sets to the size of their union. In the context of causal extraction, it would assess the similarity be- tween the sets of tokens (or other elements) in the predicted and ground truth sequences.
Rateliff-Obershely Similarity (ROS): The ROS is a sequence comparison metric that mea- sures the similarity between two sequences by identifying the common substrings between them. It calculates similarity score based the length of the longest common subse- quence between the sequences. This metric would quantify how much of the predicted causal spans match the ground truth causal spans in terms of shared subsequence patterns.

---

Post
Extracted Span
Predicted Label
Partially extracted causal spans
1. Rutgers Professor On White People: We Gotta Take These MIl'ers Out!."
Take These MF'ers Out!
Non-Distressed
#Rutgers iProfessor #WhitePeople
EXCLUSIVE:
Video shows terrifying ambush-style robbery, shooting in San Francisco @California #Crime
terrifying ambush-style robbery,
Distressed
shooting in San
Causal spans extracted
3. Apu reads a children's book about a transsexual Naxi.
Cause
Non-Distressed
4. Speaking with Raymond Ibrahim about #Christian Persecution
No_Cause
Distressed
Table 9; Error Analysis from the Proposed Systems. Color Coding: Blue- Correct, Red: Incorrect; Teal: Incomplete. Highlighted text in pink shows the human-ame annootted causal spans.


Reviews:
[{"review_id": "3YisOdijkB", "reviewer": "Reviewer_7DRN", "summary": "", "questions": "None.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Aligning with recent research (Ghosh et al., 2022; Singh et al., 2023) in cause extraction, we presented our findings using metrics such as Full match (FM), Partial match (PM), Hamming Distance (HD), Jaccard Similarity (JS), and Ratcliffe-Obershelp Similarity (ROS). We regret the omission of metric definitions and commit to incorporating them in the revised version pending acceptance.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The constructed distress and cause annotated is valuable.\n2. The proposed method is well-designed.", "reasons_to_reject": "1.  Some sections of the paper lack clarity, such as the absence of metric definitions (FM, PM, HD, JF, ROS) on page 7.", "questions_to_authors": "None.", "paper_topic_and_main_contributions": "introduces an online post-based method for identifying distress and extracting causes.\nThe primary contributions encompass a multimodal dataset annotated with distress and causes, along with a method for identifying distress and extracting causes.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "None.", "Typos_Grammar_Style_and_Presentation_Improvements": "None."}, "extra_text": {}}, {"review_id": "8Cw5aXNeiK", "reviewer": "Reviewer_rcDd", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. We appreciate the reviewer's input. A pivotal aspect of our research involves introducing the novel task of Unified Distress Identification and Cause Extraction (DICE) from multimodal online posts. While cross-attention and self-attention are used in various tasks, our proposed DICE framework introduces a novel inter-modal fusion (IMF) mechanism that dynamically aligns and exchanges information between modalities, facilitating a more comprehensive fusion of text and image features. Our method enhances the adaptability of modal interaction by combining external word embedding models and self-attention mechanisms in the IMF module. Additionally, the incorporation of sentiment information through zero-shot learning allows us to mitigate model complexity and dependency on annotated emotion inputs, setting our approach apart from conventional methods and enhancing its suitability for real-world applications.\n\n2. We apologize for not explicitly stating in the main text that the experimental setup (section A.3) and ablation experiments (section A.6) details are provided in the appendix. We will rectify this oversight in the final version, should it be accepted.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This paper is easy to read.\n- They construct a new dataset.", "reasons_to_reject": "- The method is not novel. 1) Using cross-attention and self-attention to fuse text and image is not new. 2) Using multi-task and incorporating sentiment information to do social media related-task is not new. \n- Lack of model ablation. I don't know which technique contributes most to the performance.\n- Lack of experiment settings. \n", "questions_to_authors": "", "paper_topic_and_main_contributions": "Background of the paper: \nWith the increasing influence of hate speech on social media on public opinion, methods for automatically identifying hate speech have attracted increasing attention. However, past automatic recognition methods have mostly been limited to analyzing textual content, and little attention has been paid to the interpretability of these models, despite the possible social and legal consequences of wrong predictions.\n\nPast solutions: \nPast methods mainly focus on the analysis of text content and lack the consideration of multimodal input. In addition, these methods have limitations in terms of interpretability.\n\nMotivation of the paper: \nGiven the limitations of past approaches, this study poses a novel problem of identifying distress and extracting causes from multimodal online posts. To address this issue, thet develop a multi-task deep framework and use sentiment information to incorporate sentiment information into the training process. Furthermore, they create the first multimodal dataset of distress and cause annotations, demonstrating the effectiveness of their proposed method by comparing with existing benchmarks", "excitement": 2, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "tzzzOp23cg", "reviewer": "Reviewer_RS7U", "summary": "", "questions": "1. What is the proportion of the samples whose cause is latent, i.e. No_cause sample? These samples can be difficult to detect.\n2. In Table 4, the F1 and Acc. of DICE are 86.12 and 86.54, which is same as the results of DICE with only textual-modal shown in Table 3. Does Table 4 only show the performance of each model in the textual modal? In the experimental stage, is the input of the ViL-BERT CC and Visual BERT COCO only text? Since these two baselines are multi-modal pre-training models, the introduction of bi-modal as an input will have obvious performance improvement.\n3. What is the performance of ViL-BERT CC and Visual BERT COCO in the task of CE?\n4. DICE is a general approach for multimodal tasks, so how well does it work on other tasks, e.g. multimodal sarcasm detection?\n5. As a multimodal task, the combination of text and image is necessary. However, 199~201 shows that the reason seems to only be based on the text itself, right?\n", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's thorough review, valuable feedback, and, acknowledge the need for clarity in our paper.\n\n1. Through our annotation process, we identified that over half of the posts exhibit latent causes, which were consequently labelled as \"no_cause.\" Given this significant portion of the dataset, our proposed DICE framework effectively accommodates data distribution, enabling the model to adeptly address such instances during inference by often predicting empty spans.\n\n2. We apologize for any confusion caused by the presentation of results in Table 4. Your observation is valid; indeed, ViL-BERT CC and Visual BERT COCO are multimodal baselines, and we trained them on bi-modal inputs. However, their results were mistakenly included in the text-based models section, leading to the confusion. The intention was to present their performance in Table 3, which primarily features multimodal results. In Table 4, we aimed to showcase the results of text-based models, including DICE and other methods, as the remaining baselines do not support multimodal inputs. We will ensure the correction of this inconsistency in the revised version upon acceptance. Thank you for your valuable feedback.\n\n3. Our DCaM dataset is comprised of image-text inputs, prompting us to incorporate two well-regarded multimodal baselines, ViLBERT CC and Visual BERT COCO, to ensure a comprehensive evaluation. While adapting these models for cause extraction tasks would necessitate modifications to their output layers, we deliberately refrained from making such changes. To maintain fairness in comparison, we ran these models using their original architecture while optimizing the hyperparameters, enabling us to report results specifically for the distress identification task. \n\n4. While our proposed DICE method is indeed designed to handle various multimodal tasks, our current study's focus was on distress identification and cause extraction from social media posts. We acknowledge the importance of testing the model's robustness on other tasks, including multimodal sarcasm detection, and we are eager to explore such avenues in our future research endeavors.\n\n5. Our dataset annotations encompass two aspects: the post's distress categorization (Distressed or Non-distressed), where both image and text are considered, and causal span annotation, which predominantly relies on the text. While the image plays a pivotal role in detecting distress, the main objective of the cause extraction task is to extract causal spans from the text, where the role of the image is not central. This distinction underlines the varying roles of text and image in our approach. We trust this clarifies your query.\n\n6. Our research was driven by a desire to delve into the intricacies of distress classification and cause extraction for distressed posts, with a primary focus on comprehending negative and harmful content efficiently. To streamline the analysis, we grouped various connotations of offensive content under the \"distressed\" category, as our primary goal was not to categorize among different forms of distress. Our study's central objective was to comprehend the causes associated with various forms of offensive content, including hate, offense, and abuse. Regarding your point about the distinction between hate speech and distressed content in our paper, while Fig 2 might depict a news article post, it's crucial to note that our dataset is deliberately designed to be inclusive of various post sources and types, ensuring a comprehensive representation. Our core objective revolves around identifying any harmful negative content within a post and uncovering its underlying causes. The introduction of the \"Distressed\" class is a deliberate choice to encompass a wide range of posts that hint at harmful events. This approach aligns perfectly with the overarching aim of our research. We hope this explanation helps clarify our perspective on the matter.\n\n7. We sincerely apologize for the inconsistencies between the symbols, module names in figures, and the main text. In our revised version, we will rectify these discrepancies to ensure clarity and alignment. Concerning DeepSentiBank (Chen et al., 2014), we acknowledge its significance as a visual sentiment analysis model using deep CNNs. While we omitted a detailed discussion on DeepSentiBank in the original submission due to its established nature, we will incorporate a brief explanation in the revised version, detailing its use for ANP features extraction in our approach. We also acknowledge the confusion regarding the Multi-Modal Fusion module in Fig. 3, which indeed corresponds to Inter-Modal Fusion (IMF) in the main text. To enhance clarity, we will rename it as IMF in the figure. We apologize for the oversight in incorrectly referring to Word2Vec instead of BERT for emotion class embeddings; this will be corrected in the revised version as well.\n\n8. a) We apologize for not explicitly stating in the main text that the discussion on the ablation experiments (section A.6) are provided in the appendix. We will rectify this oversight in the final version, should it be accepted.\nb) We acknowledge the importance of baseline experiments and will include CLIP and BERT+ResNet+MLP as suggested to enhance the robustness of our study in the revised version.\n\n9. We will ensure to incorporate the suggested references for a comprehensive literature coverage in the final version if accepted.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Unified Distress Identification and Cause Extraction of multimodal posts is an issue of concern. This paper could lead to a more active discussion of the issue among researchers.\n2. The dataset DCaM is on par with other datasets, and presents the cause of some stressed samples.\n3. The model DICE, especially Inter-modal Fusion module, is a general approach for multimodal tasks. It has good reusability.\n", "reasons_to_reject": "1. Some descriptions in this paper are inconsistent with the actual situation, which may lead to misunderstanding. The authors may have confused hate speech with distress content, believing that the dataset they built contained hate speech (Table 1). However, according to Fig. 2 and samples that have been open-source, most of the distressed samples in this dataset are news reporting violence and discrimination, which is harmless in itself. And it is fundamentally different from hate speech that spreads prejudice and insults others, mostly by user comments.\nTherefore, some of the discussions about comparisons with related datasets and methods of hate speech detection may be incorrect. For example, 206-209 mentions the high agreement of the dataset built in this paper and attributes it to the annotation process. In fact, it may simply be that news texts are more objective than user comments, resulting in less difficult and divisive labeling.\nIn a word, it is suggested that researchers should further clarify their research content, and it is necessary to describe the difference and connection between hate speech and distress content.\n\n2. Illustrations of the model DICE (i.e., Fig. 3 and Fig. 4) are not clear. And many symbols and modules in the figures cannot correspond to the text of methodology. For example, \na. In Fig.3, I can know that Deep SentiBank is introduced in DICE, however, the section of Methodology lacks related introduction. \nb. Is the Multi-Modal Fusion module in Fig. 3 be the Inter-Modal Fusion (IMF) in 294 of Section 3.2?\nc. In Fig.3, the semantic vectors of emotion classes is represented by BERT, however, 367-369 shows that Word2vec is introduced but not BERT.\n\n3. Existing experiments may not enough to prove the effectiveness of the model DICE:\na. The paper lacks some ablation experiments to prove the effectiveness of the modules, including vocab graph and emotion feature.\nb. The paper lacks the necessary baseline experiments, including CLIP, which is the most commonly used multimodal model at present, and BERT+ResNet+MLP, which is the backbone of DICE.\n\n4. The paper lacks some important references about multimodal hateful detection. I will list them later.\n", "questions_to_authors": "1. What is the proportion of the samples whose cause is latent, i.e. No_cause sample? These samples can be difficult to detect.\n2. In Table 4, the F1 and Acc. of DICE are 86.12 and 86.54, which is same as the results of DICE with only textual-modal shown in Table 3. Does Table 4 only show the performance of each model in the textual modal? In the experimental stage, is the input of the ViL-BERT CC and Visual BERT COCO only text? Since these two baselines are multi-modal pre-training models, the introduction of bi-modal as an input will have obvious performance improvement.\n3. What is the performance of ViL-BERT CC and Visual BERT COCO in the task of CE?\n4. DICE is a general approach for multimodal tasks, so how well does it work on other tasks, e.g. multimodal sarcasm detection?\n5. As a multimodal task, the combination of text and image is necessary. However, 199~201 shows that the reason seems to only be based on the text itself, right?\n", "paper_topic_and_main_contributions": "This paper proposes a novel and interesting task of Unified Distress Identification and Cause Extraction. This brings challenges to the understanding and reasoning ability of multimodal models in images and texts.\nThe paper builds relevant dataset, proposes a multi-task framework, and presents benchmarks of various baselines.\n", "excitement": 3, "reproducibility": 1, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": " Fig. 3 and Fig. 4 are not clear enough and should be replaced by vector diagram."}, "extra_text": {"Missing_References": "[1] The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes. (NIPS 2020)\n[2] SemEval-2022 Task 5: Multimedia Automatic Misogyny Identification. (SemEval 2022)\n[3] On Explaining Multimodal Hateful Meme Detection Models. (WWW 2022)\n[4] Prompting for Multimodal Hateful Meme Classification. (EMNLP2022)\n"}}]

Author rebuttal:

Meta-review: The paper presents a new multimodal dataset for distress analysis. The social media posts have been annoated for the distress level and the cause. The construction of the dataset is sound and might be a useful resource for the community. The methodology is mostly sound and it might need to clarify some aspects (e.g. metric definitions, detailed experimental settings and model ablation, distress vs hate speech). However, the reviewers did not find the work particularly exciting.
Decision: Accept-Main