Towards Identifying Fine-Grained Depression Symptoms from Memes
Year: 2023

Abstract:
The past decade has observed significant attention toward developing computational methods for classifying social media data based on the presence or absence of mental health conditions. In the context of mental health, for clinicians to make an accurate diagnosis or provide personalized intervention, it is crucial to identify fine-grained mental health symptoms. To this end, we conduct a focused study on depression disorder and introduce a new task of identifying fine-grained depressive symptoms from memes. Toward this, we create a highquality dataset (RESTORE) annotated with 8 fine-grained depression symptoms based on the clinically adopted PHQ-9 questionnaire. We benchmark RESTORE on 20 strong monomodal and multimodal methods. Additionally, we show how imposing orthogonal constraints on textual and visual feature representations in a multimodal setting can enforce the model to learn non-redundant and de-correlated features leading to a better prediction of fine-grained depression symptoms. Further, we conduct an extensive human analysis and elaborate on the limitations of existing multimodal models that often overlook the implicit connection between visual and textual elements of a meme.

heading: 1 Introduction
text: Mental health disorders have a profound impact on society. Almost 1 billion people worldwide suffer from mental health disorders, predominantly depression, anxiety, mood, and substance use disorders (WHO, 2022). Two of the most common mental health disorder, depression and anxiety account for the US $1 trillion in economic losses worldwide annually (Health, 2020). This cost is projected to rise to a staggering US $6 trillion by 2030 (Bloom et al., 2012). Apart from the economic burden, the social burden of mental health disorders is huge. Suicide is now the fourth leading cause of death among those aged 15 to 29 years old (Fleischmann
et al., 2021). However, considering its preponderance and global burden, depression continues to be significantly undertreated in all practice settings, where fewer than one-third of adults with depression receive effective treatment. Denial of illness and stigma are the two most common obstacles to appropriate diagnosis and treatment of depression (Sirey et al., 2001).
Recently, social media data has emerged as a powerful “lens” for tracking and detecting depression (De Choudhury et al., 2013; Yates et al., 2017). The vast majority of the existing works on depression have utilized the textual or multi-modal information available in the social media data to primarily classify the posts based on the perceived depressive behavior (depressive or non-depressive) (De Choudhury et al., 2014; Coppersmith et al., 2015; Gui et al., 2019). However, for healthcare professionals to intervene and provide effective treatment, it is crucial for them to understand the leading symptoms of that depressive behavior.
8890
Motivated by this, we aim to develop a practical decision support system that swift through social media posts and can provide healthcare professionals deeper insights into one’s depressive behaviors by capturing the fine-grained depressive symptoms. In the past, there have been few attempts (Yadav et al., 2020; Yazdavar et al., 2017) to capture the depression symptoms; however, they are confined to only textual information. Recently, a new form of communication has emerged in social media:
‘meme’. A meme usually consists of an expressive image embedded with a short block of text. It is designed to convey a complex idea or emotional state of mind, which is far easier to understand than a textual description of thoughts. They acknowledge a shared experience between the creator and the viewer and therefore have become a fast way of communication on social media. Outside the mental health domain, there are numerous studies on meme processing and understanding for emotion detection (Sharma et al., 2020; Pramanick et al., 2021a), cyberbullying (Maity et al., 2022), and hateful meme detection (Zhou et al., 2021; Pramanick et al., 2021b).
However, to our knowledge, none of the existing studies have yet leveraged the visual information available in memes specifically to capture the finegrained depression symptoms. There are two main reasons to consider the visual information: (i) according to a recent survey1, images appear in more than 42% of tweet; and (ii) textual information alone cannot capture the overall semantic meaning. For example, in Figure 1, considering only the text, “These would give me a peaceful scene in a land of trouble”, would not be sufficient to identify the depressive symptoms or even to distinguish whether the meme is depressive or not. However, it is evident from the image that the meme expresses suicidal thoughts/intent. Therefore, in order to obtain a holistic view of a post and accurately determine the depression symptoms, it is necessary to take into account the visual information available in social media posts.
To this end, we propose a new task – FineGrained Depression Symptom Identification from Memes and hypothesize that leveraging the multi-modal information available in the memes can more effectively help identify depression symptoms from social media posts. Towards this, we utilize clinically established 9-scale Pa-
1shorturl.at/nuFMQ
tient Health Questionnaire (PHQ-9) (Kroenke and Spitzer, 2002) depression symptoms categories to classify the depressive memes that we collected from two popular social media forums – Reddit and Twitter. In particular, we make the following contributions: (1) We create a high-quality dataset (RESTORE) consisting of 9, 837 depression memes, annotated with 8 fine-grained PHQ-9 symptom categories: Feeling Down, Lack of Interest, Eating Disorder, Sleeping Disorder, Concentration Problem, Lack of Energy, Low Self Esteem, and Self Harm. (2) We perform extensive experiments with 20 stateof-the-art monomodal and multimodal approaches to benchmark our dataset and introduce orthogonality constraints in a multimodal setting to incorporate multiple perspectives present in the meme. (3) We conduct a thorough human analysis and highlight the major findings and limitations of the monomodal and multimodal models. Our bestperforming model obtains an F1-Score of only 65.01, demonstrating the challenge involved with meme processing for depression symptom identification task, and we believe that our dataset will promote further research in this direction.

heading: 2 Related Works
text: Based on the data modalities, we categorize the existing works on depression detection as follows:
Language As highlighted in Fine (2006) study, people’s thoughts are frequently reflected in their language, and the linguistic cues, such as informal language usage, first-person referencing, and greater usage of negative emotion words, generally typify psychiatric disorders (Ramirez-Esparza et al., 2008; Resnik et al., 2015). Numerous research in computational linguistics has modeled the language usage in mental health-related discourse to predict mental health states (Tsugawa et al., 2015; Harman and Dredze, 2014) and infer risk to various mental disorders using social media data (Benton et al., 2017b; Coppersmith et al., 2016; Huang et al., 2015; Yadav et al., 2018, 2021). Most of the earlier works utilized a feature-driven approach (Resnik et al., 2015; Karmen et al., 2015) to detect depression. Recently with the availability of multiple benchmark datasets (Yates et al., 2017; Coppersmith et al., 2015), existing methods are designed using neural models (Orabi et al., 2018; Zhang et al., 2020). While most of these existing work studies depression at a coarser level, there
have been only a few efforts towards inferring depressive symptoms by analyzing the textual information in social media posts (Yadav et al., 2020).
Vision The visual information available in shared images offers valuable psychological cues for understanding a user’s depression status. Previous studies (Girard et al., 2014; Scherer et al., 2013; Zhu et al., 2017) conducted in a clinical setting have established that certain non-verbal behaviors such as downward gaze angle, dull smiles, and shorter average lengths of a smile characterize depressive behaviors. Recently, with the popularity of photo and video-sharing social networking services such as Instagram have piqued the interest of researchers in investigating people’s depressive behavior from their visual narratives. Reece and Danforth (2017); Manikonda and De Choudhury (2017) investigated the role of public Instagram profiles in identifying a depressive user.
Multimodal (Language+Vision+Speech) In recent years, there has been growing attention towards exploiting multimodal information such as speech, vision and text for depression detection (Valstar et al., 2013, 2014; Ringeval et al., 2018). Existing studies have devised several neural approaches to effectively combine the information from various modalities. For instance, Yin et al. (2019) utilized the hierarchical bidirectional LSTM network to extract and fuse the local video and audio features to predict the degree of depression. Gui et al. (2019) proposed a multi-agent reinforcement learning method for identifying depressive users. An et al. (2020) developed the topic-enriched multi-task learning framework that achieved stateof-the-art performance on multimodal depression detection tasks. In contrast to the above approaches, our study aims to find the fine-grained depression symptom from memes that have not yet been explored before.

heading: 3 Dataset Creation
text: In this section, we present a new benchmark dataset: RESTORE2 for identifying fine-grained depRessivE SympTOms fRom mEmes, that was created following a clinically-guided approach and includes contributions from medical informatics experts and psychologist at each phase.
2The dataset is available at https://forms.gle/ qe5fYgesBZr1SiUd7

heading: 3.1 Task Structure
text: Dataset Selection and Curation. We collect posts from two popular social media platforms: Twitter and Reddit. We chose these platforms as a data source because of their rising popularity among depressive users to publicly express their thoughts, moods, emotions, and feelings. This, coupled with the greater degree of anonymity, facilitates self-disclosure and allows users to be more truthful and open in sharing sensitive issues and personal life without fear of being embarrassed or judged. Thus these user-generated self-narratives provide low-cost, large-scale, non-intrusive data to understand depressive behavior patterns and outcomes outside the controlled clinical environment, both in real-time and longitudinally.
To ensure that we capture a broader spectrum of depressive behaviors, we use a domain-specific depression lexicon (Yazdavar et al., 2017). The lexicon contains depression-related terms from 8 symptom categories following the PHQ-93 questionnaire. We use the depression lexicon to collect tweets from Twitter public profiles that mention at least one of the words from the lexicon in their profile description. In a similar manner, we collect Reddit posts; however, we restrict ourselves to the following subreddits: “Mental Health”, “depression”, “suicide watch”, “depression memes”, “eating disorder”, and “sleeping disorder”.
Objective. Given a meme (containing image and an embedded text) and an 8 fine-grained PHQ-9 depression symptom categories, the goal is to identify all depression symptoms that are expressed in the meme.

heading: 3.2 Task Construction
text: Filtering Strategy. Since the focus of this study is to map the content in memes to the corresponding PHQ-9 symptoms categories, we filtered out the posts that do not have a meme. Further, we applied a series of filtering steps to remove any irrelevant memes: (i) the meme should contain both image and embedded text (refers to the text which is embedded in the meme); (ii) the meme text must be written in English; (iii) the embedded text in
3Though the PHQ-9 questionnaire includes 9 depression symptom categories, we excluded one symptom category, “Hyper/lower activity”, as this symptom can only be identified by the following behavior: “Moving or speaking so slowly that other people could have noticed?”. Since this behavior can’t be inferred by static social media data such as memes, we did not consider this symptom in our study.
the meme should be readable; (iv) the meme image should not be blurry and have a high resolution. Further, we filtered out those memes for which the OCR4 could not obtain the text. Following these filtering criteria, we obtain 11, 000 posts.
Expert Annotation. We devised an annotation guideline based on the clinically adopted PHQ-9 depression questionnaire, which is a tool to assess the severity of depression. A team of 4 annotators (experts in psychology and medical informatics) independently annotated the collected memes. Each annotator was provided annotation guidelines and an interface to map the content in memes to the closest PHQ-9 symptom. Specifically, for a given meme, the annotators were instructed to label the depression symptom categories: Lack of Interest, Feeling Down, Sleeping Disorder, Lack of Energy, Eating Disorder, Low Self-Esteem, Concentration Problem, and Self Harm, that are the closest match to the meme based on the textual or visual information available in the meme. Note that symptoms can be one or multiple per meme, which renders the task as multi-label classification. If the meme does not contain any of these symptoms, annotators were instructed to label the meme in the “Other” class, which was not considered in our final dataset. For this task, the inter-annotator agreement (Krippendorff’s alpha coefficient (Krippendorff, 2004)) is 81.55, which signifies a strong agreement amongst annotators. We provide examples for each symptom category corresponding to memes in Figure 2 and definition in Appendix-A.
4https://cloud.google.com/vision/docs/ocr

heading: 3.3 Benchmark Dataset
text: Our final dataset includes 4, 664 depressive memes, and the distribution of PHQ-9 symptoms corresponding to these memes, as well as the train, test, and validation split, are shown in Table-1. Based on the obtained PHQ-9 class distribution, we can notice that a few PHQ-9 symptom categories are prominent in our human-annotated set, such as ‘FD’, ‘ED’ and ‘SH’. In contrast, ‘LOI’,
‘SD’, ‘LOE’, and ‘CP’ symptoms rarely appear in our human-annotated dataset.
To enrich and balance a few PHQ-9 symptom categories, we developed the training set with a portion of automatic curation. In our automatic curation of training samples, we followed two strategies to expand the human-annotated training set. In the first strategy, we conducted keyword-based search using “eating disorder memes”, “feeling down memes”, “sleep disorder memes”, “lack of energy memes”, “low self esteem memes”, “concentration problem memes”, “self-harm” on the Google Image and selected only top image search results. The second strategy considers selecting the top image results from Pinterest with the queries: “insomnia memes”, “lack of interest memes”, and “sleep disorder memes”. To remove noise, we maintained strict filtering on the resolution of the meme and on the readability of the meme’s text. We also de-duplicate the memes if their sources are the same. Following this process, we obtained additional 5, 173 samples, which we used to enrich the training set. Also, it is to be noted that both the test and validation set only include manually annotated samples.

heading: 4 RESTORE Dataset Analysis
text: Visual Analysis. We conducted a visual analysis of the memes to study how depression symptoms are related to color features. We performed color analysis by computing the pixel-level average w.r.t HSV (hue, saturation, and value), in which lower hue scores imply more redness and higher hue scores suggest more blue. Saturation describes an image’s vibrancy. Value relates to the brightness of an image, with lower scores indicating a darker image. We observe that most of the memes, irrespective of symptom categories, are less colorful and have lower saturation values, suggesting negative emotions. These cases were prominent in “low self esteem”, “lack of interest”, and “self harm”, where users often share memes that were less vivid, darker (higher grayscale), and have a high hue. In contrast, the memes related to “eating disorder” are brighter and more colorful, mainly because of the presence of food in the memes.
Qualitative Language Analysis. To understand the psycho-linguistics patterns associated with each PHQ-9 symptom category, we employed the LIWC (Tausczik and Pennebaker, 2010) to measure various linguistic factors such as analytical reasoning, clout, originality, emotional tone, informal language markers, and pronouns. Our analysis reveals that “low self esteem” has the lowest analytic reasoning among all the depression symptoms, depicting a more intuitive and personal language. Surprisingly, “concentration problem” has the highest analytic reasoning, suggesting formal and logical thinking patterns. The clout feature, which measures individual confidence and clarity in speaking or writing, was found to be highest in the “feeling down” and lowest in the “eating disorder” category. A similar trend was observed with the authentic feature, which is one way of presenting themselves to others in an original way. Further, we notice that individuals expressing “self harm” behavior, “feeling down”, and “low self esteem” symptoms use more first-person pronouns.

heading: 4.1 Benchmark Methods
text: We benchmark the RESTORE dataset on the following methods:
Monomodal (Language) Methods. We experiment with four pre-trained language models: BERT (Devlin et al., 2019), ROBERTA (Liu et al., 2019), XLNET (Yang et al., 2019) and MENTALBERT (Ji et al., 2021), fine-tuned on the RESTORE training set. For each model, we obtained the hidden state representations and utilized the feedforward network with the sigmoid activation function to predict the multi-label depression categories. Additionally, we also fine-tuned the BERT model by adding the LIWC features to integrate psycholinguistic information into BERT explicitly. We project the LIWC features using a feedforward network and concatenate these projected features with the BERT [CLS] token representation. The concatenated features are used to predict fine-grained depression symptoms. We call this network as the BERT+LIWC model.
Monomodal (Vision) Methods. To evaluate the effectiveness of visual information, we experiment with seven popular pre-trained vision models: DENSENET (Iandola et al., 2014), RESNET152 (He et al., 2016), RESNEXT (Xie et al., 2017), CONVNEXT (Liu et al., 2022), REGNET (Schneider et al., 2017), EFFICIENTNET (Tan and Le, 2019), and VIT (Dosovitskiy et al., 2020). We fine-tuned these models on the RESTORE training set similar to the monomodal (language) models.
Multimodal Methods. We experiment with three state-of-the-art pre-trained multimodal models: VISUALBERT (Li et al., 2019), MMBT (Kiela et al., 2019), and CLIP (Radford et al., 2021), fine-tuned on the RESTORE training set. Additionally, we also experiment with the following models:
• Late Fusion: This model computes the mean prediction scores obtained from RESNET-152 and BERT model. • Early Fusion: This approach concatenates features obtained from RESNET-152 and BERT, which are passed to a feed-forward network to make predictions. • BERT+HSV: Here, we fine-tuned the BERT model by adding mean, max, and min values of HSV features of the image. Similar to BERT+LIWC, we concatenate HSV projected features with BERT [CLS] token representation to make predictions.

heading: 5 Proposed Approach
text: Existing multimodal approaches focus on generating text-image feature representations by detecting the objects in the image and learning an alignment between textual and visual tokens. However, a meme can convey multiple perspectives, and detecting the object alone may not be sufficient to generate a semantically-rich text-image representation. Therefore, to capture the image’s multiple views that could be beneficial in effectively distinguishing the depression symptoms, we introduce orthogonal feature generation in a multimodal setting. We begin by first encoding the meme image I and its embedded text T with the pre-trained RESNET-152 model and BERT model, respectively. We selected these models because of their simplicity and comparable performance to other languagevision models. To capture multiple perspectives of the image, we perform the 2-dimensional adaptive average pooling (adaptive-avg-pool) of output size S0 × S1 on RESNET-152 model output F that results in image representation hI ∈ RK×S0×S1 of K feature map. With this approach, we obtained feature representations h1I ∈ RK and h2I ∈ RK by setting S0 = 2 and S1 = 1 (based on the validation performance).
Orthogonal Feature Generation: We introduce orthogonal feature generation, where the features are regularized with orthogonality constraints. With this constraint, we generate new features that capture another perspective, which are nonredundant and de-correlated with the existing features. The resulting orthogonal features help the model fully utilize its capacity, improving the feature expressiveness. Formally, given the textual feature hT which corresponds to the BERT [CLS] token representation and image feature hI , we aim to generate the orthogonal feature h⊥ to h ∈ {h1I , h2I , hT } given another feature modality ĥ ∈ {h1I , h2I , hT } − {h}. Towards this, we first project the feature vector h into common vector space h̄ thereafter, we compute the vector component C and orthogonal projection as follows:
C = h̄T ĥ
h̄T h̄ h̄ and h⊥ = ĥ− C (1)
In this process, we obtained the orthogonal feature h⊥ to h̄ that also ensures (based on vector arithmetic) that it is non-redundant to ĥ.
Multimodal Fusion: In order to fuse both modalities, we devise a multimodal fusion strategy based on conditional adaptive gating. Specifically, we first compute the bimodal scalars g1 and g2 with the gating mechanism (Rahman et al., 2020) by considering textual representation as one modality and one of the image features as another modality. These scalar values denote relevant information in the image feature conditioned on the textual feature. In the next step, we compute the multimodal representation considering both the image representation and the previously computed bimodal scalars with respect to the textual feature. Formally,
hu = g 1W1fh 1 I + g 2W2fh 2 I (2)
where W1f and W 2 f are weight matrices for both the image representation. With this strategy, we obtained the multimodal feature f = hT + hu.
Depressive Symptoms Identification: Here, we first apply LayerNorm (Ba et al., 2016) operation on the multimodal feature f and orthogonal feature h⊥. The resulting feature is concatenated with the textual feature hT to form the final feature representation z. Finally, we apply the sigmoid operation on z to predict depression symptom categories.

heading: 6 Implementation Details
text: We utilized the pre-trained weights of BERTbase5, RoBERTa-large6, MentalBERT7 and XLNet-base8 from HuggingFace (Wolf et al., 2020). For the pre-trained vision models, we followed the torchvision API9 and obtained the pre-trained weights of the vision models. In particularly, we use resnet152, resnext101_32x8d, densenet161, efficientnet_b4, regnet_y_800mf, vit_l_32, and convnext_large pre-trained weights to fine-tune on the PHQ-9 depression symptom identification task. We use the HuggingFace implementation10 of VisualBERT to fine-tune the model on the PHQ-9 depression symptom
5https://huggingface.co/bert-base-uncased 6https://huggingface.co/roberta-large 7https://huggingface.co/mental/
mental-bert-base-uncased 8https://huggingface.co/xlnet-base-cased 9https://pytorch.org/vision/stable/models.html
10https://github.com/huggingface/transformers/ tree/main/examples/research_projects/visual_bert
identification task. For MMBT11 and CLIP12 also we follow the HuggingFace implementation and fine-tune the model on PHQ-9 depression symptom identification task. For the visual analysis of the RESTORE dataset, we use the open-cv python library13. We fine-tuned each model on the RESTORE training dataset for 10 epochs. The length of the maximum original text is set to 256 tokens. We normalized the images with pixel mean and standard deviation values before feeding them to the monomodal and multimodal networks. We evaluate the performance of each model on the RESTORE validation dataset and use the best (maximum micro F1-score) model to evaluate the performance on the RESTORE test dataset. To update the monomodal (vision) model parameters, we used AdamW (Loshchilov and Hutter, 2018) optimizer with the learning rate of 4e − 5. For the monomodal (language) and multimodal approaches, we used the AdamW optimization algorithm with a learning rate of 4e − 5. We set the batch size 64 to train all the benchmark models. We train the proposed network with batch size 16 and AdamW optimization algorithm (with the learning rate of 2e − 5) for 10 epochs. The dimension (K) of feature map obtained from RESNET-152 is 2048. For LIWC and HSV experiments, we set the size of the hidden unit as 20. We performed all the experiments on a single NVIDIA Tesla V100x GPU having 32GB memory. We observed the average runtime to train our framework is 11.55 minutes per epoch. The proposed model has ∼ 170 million parameters. All the libraries used in the experiment are licensed under the following:
• HuggingFace (3.5.0): Apache-2.0 License • NLTK (3.6.3): Apache-2.0 License • spacy (3.4.4): MIT License • LIWC (22): Academic License • open-cv (4.5.4): Apache-2.0 License • PyTorch (1.10.1): modified BSD license

heading: 7 Results and Observations
text: Main Results Table 2 provides the summary of the results of monomodal and multimodal approaches. The obtained results (first two blocks
11https://github.com/huggingface/transformers/ tree/main/examples/research_projects/mm-imdb
12https://huggingface.co/openai/ clip-vit-base-patch32
13https://docs.opencv.org/4.x/d6/d00/tutorial_ py_root.html
of the table) show that pre-trained language models are better at capturing depression symptoms than the pre-trained vision models. We hypothesize that the existing vision models are pre-trained on generic IMAGENET (Deng et al., 2009) classes. Thus, these models lack the deeper semantic understanding of images that are required to effectively encode the memes’ visual information in order to distinguish the depression symptoms categories precisely. While our finding reveals that the monomodal (language) model performs better than the monomodal (vision) model, we found that multimodal models having sophisticated fusion mechanisms such as VISUALBERT, and MMBT obtain significant improvement over the BERT on multiple symptom categories. This signifies that visual content is helpful in accurately classifying depression symptoms if used with a better mechanism to fuse the visual information with language features. Further, we observed that among all the competitive methods, our approach obtained the best performance in terms of the F1-score (cf. Table 3). For two classes (SH and LOI), MENTALBERT outperformed all the other models. We speculate that this
is because a major portion of the corpus used to pretrain the MENTALBERT was centered on suicide and stress. For the LOE class, basic MLP+HSV model performs best because memes of these categories have higher grayscale and lower brightness values, which were effectively captured by HSV features. Though some of these approaches perform well in a particular depression category, they could not translate their performance across all the categories. In contrast, our proposed model shows competitive performance across most categories, which signifies the superiority of our proposed approach. Ablation Study. To analyze the role of each component of the proposed method, we performed an ablation study and reported the results in Table 4 (top). We observe a performance drop of 1.2 and 2.6 points in the F1-score by removing multimodal fusion and orthogonal components. The significant performance drop confirms the importance of each component in predicting the symptoms category. We also analyze the role (Table 4, bottom) of imposing an orthogonal constraint on visual and
textual features and find that feature orthogonal to h1I given hT performs better compared to others.

heading: 7.1 Analysis and Observations
text: We conducted an in-depth human analysis of models’ predictions and came up with the following observations: (a) Language. We noticed the memes that were correctly classified have clear depressive words. For example, consider Fig 3 (a), here the LM correctly predicted it as ‘self-harm’ because of the presence of word ‘dead’ in the text. This type of case was relatively higher for the classes, ‘eating disorder’ and ‘sleeping disorder’. (b) Vision. The vision models were also able to make correct predictions when a certain object in the meme correlated with the particular symptom class. For example, in Fig 3 (b) due to the presence of the ‘cake’, most of the models correctly predicted it as ‘eating disorder’. (c) Implied Meaning. We observed that most of the models fail to infer an implicit sense of the memes. Fig 3 (c) shows an example of this error category made by all the models. Here, to correctly infer the depressive symptom, ‘lack of interest’, it is crucial to consider both the text and image which share complementary information. However, the multimodal models fail to judiciously fuse this complementary information leading to misclassification. The majority of the vision models predicted it as ‘eating disorder’, since the person is sitting on the dining chair and the models relates dining with
eating. (d) Figurative Speech. The usage of figurative speech is highly predominant in memes, mainly to compete with other memes and gain the attention and engagement of their followers. Our analysis reveals that both unimodal and multimodal models were not capable of dealing with figurative memes. For example, in Fig 3 (e), the word ‘loop’ is used in the metaphoric sense, and neither the vision nor the LM understand the sense of the word ‘loop’ or relate the ‘rope’ with the ‘self-harm’. (e) Artistic Texts. Another way of making the meme more appealing to others is by using a variety of styling options on the texts. This brings a unique challenge for the OCR system to correctly extract all the text. For example, in Fig 3 (d), the OCR extracted the word ‘changing’ instead of ‘hanging’ leading to misclassification. (f) Generic Images. We observed that few images which share the same aesthetic features do provide any symptom-specific visual cues. For example, in Fig 3 (g) and (h), if we just consider the image, we can only infer that person is feeling sad. It is in these cases the linguistic features are crucial in identifying the correct depression symptom class.

heading: 8 Conclusion
text: This work presents the first study towards identifying fine-grained depression symptoms from memes. We created a high-quality dataset – RESTORE, consisting of 9, 837 depressive memes annotated with PHQ-9 symptom categories and benchmark the dataset on 20 monomodal and multimodal models. Further, we introduce a novel method to incorpo-
rate various perspective in the meme that obtained best F1-Score over other approaches. Finally, our thorough human analysis of the model predictions indicates the model’s limitation while dealing with memes, which will be considered in the future.

heading: 9 Limitations
text: This paper aims to make advancements toward automatically identifying fine-grained depressive symptoms from memes shared on social media. Although we used only those memes shared by users who self-declared themselves as depressive, we did not conduct any further clinical assessment to judge whether the user was depressive or not, nor we clinically evaluated their depression severity. Therefore, deploying this system without expert advice could compromise patient safety and lead to undesirable outcomes. We further acknowledge that determining the depression symptoms based on the visual and textual cues present in the meme can be subjective, and therefore the created gold-standard dataset may contain explicit and demographic biases. In this study, we focused on training the models using only the social media data, leaving their performance unchecked if tested on other medical data sources. Finally, our study is not indented to provide any diagnosis; instead, we envision the methods we provide being used as aids by healthcare professionals.

heading: 10 Ethical Consideration
text: Given that the created dataset is derived from social media and is focused on a sensitive mental health topic, we follow various ethical concerns regard-
ing user privacy and confidentiality as inspired by (Benton et al., 2017a) to access and analyze the data. We adhere to the data usage privacy as provided by Twitter and Reddit to crawl the public profiles of their users. To ensure that we maintain the user’s privacy, we anonymized the user profiles prior to the annotations, and we did not keep any meta-data information that would disclose the user. Further, we did not make any efforts to interact, deanonymize, or connect users on their other social media handles. The ethics review board approved the study under Human Subjects Research Exemption 4 because it is limited to publicly available social media posts. We believe that the created data would be highly beneficial to the community and to avoid any misuse (Hovy and Spruit, 2016), we will share the data with other researchers who will not deanonymize any of the users and will follow all the ethical considerations as established in this study.

heading: A PHQ-9 Depression Symptom Categories
text: Following are the depression symptom categories definition as provided by the Mayo Clinic (Clinic, 2022):
1. Loss of Interest: A decline in interest or pleasure in the majority or all normal activities, such as sexual activity, hobbies, or sports. 2. Feeling Down: Feelings of sadness, tearfulness, emptiness, or hopelessness. 3. Sleeping Disorder: Sleep disturbances, including insomnia, sleeping too much, or trouble falling or staying asleep. 4. Lack of Energy: Tiredness and lack of energy, so even small tasks take extra effort. 5. Eating Disorder: Reduced appetite and weight loss or increased cravings for food and weight gain. 6. Low Self-Esteem: Feelings of worthlessness or guilt, fixating on past failures or self-blame. 7. Concentration Problem: Trouble thinking, concentrating, making decisions, and remembering things. 8. Self-Harm: Frequent or recurrent thoughts of death, suicidal thoughts, suicide attempts, or suicide.

heading: B RESTORE Dataset Analysis
text: B.1 PHQ-9 Symptom Co-occurrence. Given that a single meme can have multiple depressive symptoms, we analyzed what symptoms occur together in a similar context through a cooccurrence heatmap, depicted in Figure 4. As can be observed, most of the samples had a single symptom. Only a few symptoms such as “feeling down” are more likely to occur with other symptoms, frequently with “lack of self-esteem”, “selfharm” and “lack of energy”. This is because these symptoms share common overlapping expressions with more generic “feeling down” symptoms. We also noticed for few cases where user expressing self-harm behaviors suffers from low self-esteem issues. This similar trend was also observed for eating disorder. Surprisingly, we observed a few uncommon co-occurrences, for instance, “concentration problem" and “self harm".
We have also provided the distribution of the memes with faces detected using Face++ API in Fig. 5. The study reveals that memes with eating disorder category contain a maximum of 60% faces and sleeping disorder memes contains 28% faces minimum amongst all the depression symptom category.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
9
3 A2. Did you discuss any potential risks of your work? 9
3 A3. Do the abstract and introduction summarize the paper’s main claims? 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? 3,4,5
3 B1. Did you cite the creators of artifacts you used? 3,4,5
3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? 6

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? 6

heading: 3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any
text: information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? 10

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? 3,4
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. 3
C 3 Did you run computational experiments? 4,5,6,7

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? 6
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? 6

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? 7

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? 6
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? 3

heading: 3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots,
text: disclaimers of any risks to participants or annotators, etc.? 3, Appendix A
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Not applicable. Annotators are authors of the paper.

heading: 3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to
text: crowdworkers explain how the data would be used? 3,10
3 D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? 10
7 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Annotators are co-authors of this paper.

heading: Future Work
text: Finally, our thorough human analysis of the model predictions indicates the model’s limitation while dealing with memes, which will be considered in the future