Fusing Visual, Textual and Connectivity Clues for Studying Mental Health

Abstract: With ubiquity of social media platforms, millions of people are sharing their online persona by expressing their thoughts, moods, emotions, feelings, and even their daily struggles with mental health issues voluntarily and publicly on social media. Unlike the most existing efforts which study depression by analyzing textual content, we examine and exploit multimodal big data to discern depressive behavior using a wide variety of features including individual-level demographics. By developing a multimodal framework and employing statistical techniques for fusing heterogeneous sets of features obtained by processing visual, textual and user interaction data, we significantly enhance the current state-of-the-art approaches for identifying depressed individuals on Twitter (improving the average F1-Score by 5 percent) as well as facilitate demographic inference from social media for broader applications. Besides providing insights into the relationship between demographics and mental health, our research assists in the design of a new breed of demographic-aware health interventions.

0pt*0*0
0pt*0*0
0pt*0*0 0.95
1]Amir Hossein Yazdavar 1]Mohammad Saeid Mahdavinejad 2]Goonmeet Bajaj
 3]William Romine 1]Amirhassan Monadjemi 1]Krishnaprasad Thirunarayan
 1]Amit Sheth 4]Jyotishman Pathak [1]Department of Computer Science & Engineering, Wright State University, OH, USA [2]Ohio State University, Columbus, OH, USA [3]Department of Biological Science, Wright State University, OH, USA [4] Division of Health Informatics, Weill Cornell University, New York, NY, USA
[1] yazdavar.2@wright.edu
With ubiquity of social media platforms, millions of people are sharing their online persona by expressing their thoughts, moods, emotions, feelings, and even their daily struggles with mental health issues voluntarily and publicly on social media. Unlike the most existing efforts which study depression by analyzing textual content, we examine and exploit multimodal big data to discern depressive behavior using a wide variety of features including individual-level demographics. By developing a multimodal framework and employing statistical techniques for fusing heterogeneous sets of features obtained by processing visual, textual and user interaction data, we significantly enhance the current state-of-the-art approaches for identifying depressed individuals on Twitter (improving the average F1-Score by 5 percent) as well as facilitate demographic inference from social media for broader applications. Besides providing insights into the relationship between demographics and mental health, our research assists in the design of a new breed of demographic-aware health interventions.
Depression is a highly prevalent public health challenge and a major cause of disability worldwide. Depression affects 6.7% (i.e., about 16 million) Americans each year . According to the World Mental Health Survey conducted in 17 countries, on average, about 5% of people reported having an episode of depression in 2011 BIBREF0 . Untreated or under-treated clinical depression can lead to suicide and other chronic risky behaviors such as drug or alcohol addiction.
Global efforts to curb clinical depression involve identifying depression through survey-based methods employing phone or online questionnaires. These approaches suffer from under-representation as well as sampling bias (with very small group of respondents.) In contrast, the widespread adoption of social media where people voluntarily and publicly express their thoughts, moods, emotions, and feelings, and even share their daily struggles with mental health problems has not been adequately tapped into studying mental illnesses, such as depression. The visual and textual content shared on different social media platforms like Twitter offer new opportunities for a deeper understanding of self-expressed depression both at an individual as well as community-level. Previous research efforts have suggested that language style, sentiment, users' activities, and engagement expressed in social media posts can predict the likelihood of depression BIBREF1 , BIBREF2 . However, except for a few attempts BIBREF3 , BIBREF4 , BIBREF5 , BIBREF6 , these investigations have seldom studied extraction of emotional state from visual content of images in posted/profile images. Visual content can express users' emotions more vividly, and psychologists noted that imagery is an effective medium for communicating difficult emotions.
According to eMarketer, photos accounted for 75% of content posted on Facebook worldwide and they are the most engaging type of content on Facebook (87%). Indeed, "a picture is worth a thousand words" and now "photos are worth a million likes." Similarly, on Twitter, the tweets with image links get twice as much attention as those without , and video-linked tweets drive up engagement . The ease and naturalness of expression through visual imagery can serve to glean depression-indicators in vulnerable individuals who often seek social support through social media BIBREF7 . Further, as psychologist Carl Rogers highlights, we often pursue and promote our Ideal-Self . In this regard, the choice of profile image can be a proxy for the online persona BIBREF8 , providing a window into an individual's mental health status. For instance, choosing emaciated legs of girls covered with several cuts as profile image portrays negative self-view BIBREF9 .
Inferring demographic information like gender and age can be crucial for stratifying our understanding of population-level epidemiology of mental health disorders. Relying on electronic health records data, previous studies explored gender differences in depressive behavior from different angles including prevalence, age at onset, comorbidities, as well as biological and psychosocial factors. For instance, women have been diagnosed with depression twice as often as men BIBREF10 and national psychiatric morbidity survey in Britain has shown higher risk of depression in women BIBREF11 . On the other hand, suicide rates for men are three to five times higher compared to that of the women BIBREF12 .
Although depression can affect anyone at any age, signs and triggers of depression vary for different age groups . Depression triggers for children include parental depression, domestic violence, and loss of a pet, friend or family member. For teenagers (ages 12-18), depression may arise from hormonal imbalance, sexuality concerns and rejection by peers. Young adults (ages 19-29) may develop depression due to life transitions, poverty, trauma, and work issues. Adult (ages 30-60) depression triggers include caring simultaneously for children and aging parents, financial burden, work and relationship issues. Senior adults develop depression from common late-life issues, social isolation, major life loses such as the death of a spouse, financial stress and other chronic health problems (e.g., cardiac disease, dementia). Therefore, inferring demographic information while studying depressive behavior from passively sensed social data, can shed better light on the population-level epidemiology of depression.
The recent advancements in deep neural networks, specifically for image analysis task, can lead to determining demographic features such as age and gender BIBREF13 . We show that by determining and integrating heterogeneous set of features from different modalities – aesthetic features from posted images (colorfulness, hue variance, sharpness, brightness, blurriness, naturalness), choice of profile picture (for gender, age, and facial expression), the screen name, the language features from both textual content and profile's description (n-gram, emotion, sentiment), and finally sociability from ego-network, and user engagement – we can reliably detect likely depressed individuals in a data set of 8,770 human-annotated Twitter users.
We address and derive answers to the following research questions: 1) How well do the content of posted images (colors, aesthetic and facial presentation) reflect depressive behavior? 2) Does the choice of profile picture show any psychological traits of depressed online persona? Are they reliable enough to represent the demographic information such as age and gender? 3) Are there any underlying common themes among depressed individuals generated using multimodal content that can be used to detect depression reliably?
Mental Health Analysis using Social Media:
Several efforts have attempted to automatically detect depression from social media content utilizing machine/deep learning and natural language processing approaches. Conducting a retrospective study over tweets, BIBREF14 characterizes depression based on factors such as language, emotion, style, ego-network, and user engagement. They built a classifier to predict the likelihood of depression in a post BIBREF14 , BIBREF15 or in an individual BIBREF1 , BIBREF16 , BIBREF17 , BIBREF18 . Moreover, there have been significant advances due to the shared task BIBREF19 focusing on methods for identifying depressed users on Twitter at the Computational Linguistics and Clinical Psychology Workshop (CLP 2015). A corpus of nearly 1,800 Twitter users was built for evaluation, and the best models employed topic modeling BIBREF20 , Linguistic Inquiry and Word Count (LIWC) features, and other metadata BIBREF21 . More recently, a neural network architecture introduced by BIBREF22 combined posts into a representation of user's activities for detecting depressed users. Another active line of research has focused on capturing suicide and self-harm signals BIBREF23 , BIBREF24 , BIBREF25 , BIBREF26 , BIBREF2 , BIBREF27 . Moreover, the CLP 2016 BIBREF28 defined a shared task on detecting the severity of the mental health from forum posts. All of these studies derive discriminative features to classify depression in user-generated content at message-level, individual-level or community-level. Recent emergence of photo-sharing platforms such as Instagram, has attracted researchers attention to study people's behavior from their visual narratives – ranging from mining their emotions BIBREF29 , and happiness trend BIBREF30 , to studying medical concerns BIBREF31 . Researchers show that people use Instagram to engage in social exchange and storytelling about their difficult experiences BIBREF4 . The role of visual imagery as a mechanism of self-disclosure by relating visual attributes to mental health disclosures on Instagram was highlighted by BIBREF3 , BIBREF5 where individual Instagram profiles were utilized to build a prediction framework for identifying markers of depression. The importance of data modality to understand user behavior on social media was highlighted by BIBREF32 . More recently, a deep neural network sequence modeling approach that marries audio and text data modalities to analyze question-answer style interviews between an individual and an agent has been developed to study mental health BIBREF32 . Similarly, a multimodal depressive dictionary learning was proposed to detect depressed users on Twitter BIBREF33 . They provide a sparse user representations by defining a feature set consisting of social network features, user profile features, visual features, emotional features BIBREF34 , topic-level features, and domain-specific features. Particularly, our choice of multi-model prediction framework is intended to improve upon the prior works involving use of images in multimodal depression analysis BIBREF33 and prior works on studying Instagram photos BIBREF6 , BIBREF35 .
Demographic information inference on Social Media: 
There is a growing interest in understanding online user's demographic information due to its numerous applications in healthcare BIBREF36 , BIBREF37 . A supervised model developed by BIBREF38 for determining users' gender by employing features such as screen-name, full-name, profile description and content on external resources (e.g., personal blog). Employing features including emoticons, acronyms, slangs, punctuations, capitalization, sentence length and included links/images, along with online behaviors such as number of friends, post time, and commenting activity, a supervised model was built for predicting user's age group BIBREF39 . Utilizing users life stage information such as secondary school student, college student, and employee, BIBREF40 builds age inference model for Dutch Twitter users. Similarly, relying on profile descriptions while devising a set of rules and patterns, a novel model introduced for extracting age for Twitter users BIBREF41 . They also parse description for occupation by consulting the SOC2010 list of occupations and validating it through social surveys. A novel age inference model was developed while relying on homophily interaction information and content for predicting age of Twitter users BIBREF42 . The limitations of textual content for predicting age and gender was highlighted by BIBREF43 . They distinguish language use based on social gender, age identity, biological sex and chronological age by collecting crowdsourced signals using a game in which players (crowd) guess the biological sex and age of a user based only on their tweets. Their findings indicate how linguistic markers can misguide (e.g., a heart represented as <3 can be misinterpreted as feminine when the writer is male.) Estimating age and gender from facial images by training a convolutional neural networks (CNN) for face recognition is an active line of research BIBREF44 , BIBREF13 , BIBREF45 .
Self-disclosure clues have been extensively utilized for creating ground-truth data for numerous social media analytic studies e.g., for predicting demographics BIBREF36 , BIBREF41 , and user's depressive behavior BIBREF46 , BIBREF47 , BIBREF48 . For instance, vulnerable individuals may employ depressive-indicative terms in their Twitter profile descriptions. Others may share their age and gender, e.g., "16 years old suicidal girl"(see Figure FIGREF15 ). We employ a huge dataset of 45,000 self-reported depressed users introduced in BIBREF46 where a lexicon of depression symptoms consisting of 1500 depression-indicative terms was created with the help of psychologist clinician and employed for collecting self-declared depressed individual's profiles. A subset of 8,770 users (24 million time-stamped tweets) containing 3981 depressed and 4789 control users (that do not show any depressive behavior) were verified by two human judges BIBREF46 . This dataset INLINEFORM0 contains the metadata values of each user such as profile descriptions, followers_count, created_at, and profile_image_url.
Age Enabled Ground-truth Dataset: We extract user's age by applying regular expression patterns to profile descriptions (such as "17 years old, self-harm, anxiety, depression") BIBREF41 . We compile "age prefixes" and "age suffixes", and use three age-extraction rules: 1. I am X years old 2. Born in X 3. X years old, where X is a "date" or age (e.g., 1994). We selected a subset of 1061 users among INLINEFORM0 as gold standard dataset INLINEFORM1 who disclose their age. From these 1061 users, 822 belong to depressed class and 239 belong to control class. From 3981 depressed users, 20.6% disclose their age in contrast with only 4% (239/4789) among control group. So self-disclosure of age is more prevalent among vulnerable users. Figure FIGREF18 depicts the age distribution in INLINEFORM2 . The general trend, consistent with the results in BIBREF42 , BIBREF49 , is biased toward young people. Indeed, according to Pew, 47% of Twitter users are younger than 30 years old BIBREF50 . Similar data collection procedure with comparable distribution have been used in many prior efforts BIBREF51 , BIBREF49 , BIBREF42 . We discuss our approach to mitigate the impact of the bias in Section 4.1. The median age is 17 for depressed class versus 19 for control class suggesting either likely depressed-user population is younger, or depressed youngsters are more likely to disclose their age for connecting to their peers (social homophily.) BIBREF51 
Gender Enabled Ground-truth Dataset: We selected a subset of 1464 users INLINEFORM0 from INLINEFORM1 who disclose their gender in their profile description. From 1464 users 64% belonged to the depressed group, and the rest (36%) to the control group. 23% of the likely depressed users disclose their gender which is considerably higher (12%) than that for the control class. Once again, gender disclosure varies among the two gender groups. For statistical significance, we performed chi-square test (null hypothesis: gender and depression are two independent variables). Figure FIGREF19 illustrates gender association with each of the two classes. Blue circles (positive residuals, see Figure FIGREF19 -A,D) show positive association among corresponding row and column variables while red circles (negative residuals, see Figure FIGREF19 -B,C) imply a repulsion. Our findings are consistent with the medical literature BIBREF10 as according to BIBREF52 more women than men were given a diagnosis of depression. In particular, the female-to-male ratio is 2.1 and 1.9 for Major Depressive Disorder and Dysthymic Disorder respectively. Our findings from Twitter data indicate there is a strong association (Chi-square: 32.75, p-value:1.04e-08) between being female and showing depressive behavior on Twitter.
We now provide an in-depth analysis of visual and textual content of vulnerable users.
Visual Content Analysis: We show that the visual content in images from posts as well as profiles provide valuable psychological cues for understanding a user's depression status. Profile/posted images can surface self-stigmatization BIBREF53 . Additionally, as opposed to typical computer vision framework for object recognition that often relies on thousands of predetermined low-level features, what matters more for assessing user's online behavior is the emotions reflected in facial expressions BIBREF54 , attributes contributing to the computational aesthetics BIBREF55 , and sentimental quotes they may subscribe to (Figure FIGREF15 ) BIBREF8 .
Facial Presence: 
For capturing facial presence, we rely on BIBREF56 's approach that uses multilevel convolutional coarse-to-fine network cascade to tackle facial landmark localization. We identify facial presentation, emotion from facial expression, and demographic features from profile/posted images . Table TABREF21 illustrates facial presentation differences in both profile and posted images (media) for depressed and control users in INLINEFORM0 . With control class showing significantly higher in both profile and media (8%, 9% respectively) compared to that for the depressed class. In contrast with age and gender disclosure, vulnerable users are less likely to disclose their facial identity, possibly due to lack of confidence or fear of stigma.
Facial Expression:
Following BIBREF8 's approach, we adopt Ekman's model of six emotions: anger, disgust, fear, joy, sadness and surprise, and use the Face++ API to automatically capture them from the shared images. Positive emotions are joy and surprise, and negative emotions are anger, disgust, fear, and sadness. In general, for each user u in INLINEFORM0 , we process profile/shared images for both the depressed and the control groups with at least one face from the shared images (Table TABREF23 ). For the photos that contain multiple faces, we measure the average emotion.
Figure FIGREF27 illustrates the inter-correlation of these features. Additionally, we observe that emotions gleaned from facial expressions correlated with emotional signals captured from textual content utilizing LIWC. This indicates visual imagery can be harnessed as a complementary channel for measuring online emotional signals.
General Image Features:
The importance of interpretable computational aesthetic features for studying users' online behavior has been highlighted by several efforts BIBREF55 , BIBREF8 , BIBREF57 . Color, as a pillar of the human vision system, has a strong association with conceptual ideas like emotion BIBREF58 , BIBREF59 . We measured the normalized red, green, blue and the mean of original colors, and brightness and contrast relative to variations of luminance. We represent images in Hue-Saturation-Value color space that seems intuitive for humans, and measure mean and variance for saturation and hue. Saturation is defined as the difference in the intensities of the different light wavelengths that compose the color. Although hue is not interpretable, high saturation indicates vividness and chromatic purity which are more appealing to the human eye BIBREF8 . Colorfulness is measured as a difference against gray background BIBREF60 . Naturalness is a measure of the degree of correspondence between images and the human perception of reality BIBREF60 . In color reproduction, naturalness is measured from the mental recollection of the colors of familiar objects. Additionally, there is a tendency among vulnerable users to share sentimental quotes bearing negative emotions. We performed optical character recognition (OCR) with python-tesseract to extract text and their sentiment score. As illustrated in Table TABREF26 , vulnerable users tend to use less colorful (higher grayscale) profile as well as shared images to convey their negative feelings, and share images that are less natural (Figure FIGREF15 ). With respect to the aesthetic quality of images (saturation, brightness, and hue), depressed users use images that are less appealing to the human eye. We employ independent t-test, while adopting Bonferroni Correction as a conservative approach to adjust the confidence intervals. Overall, we have 223 features, and choose Bonferroni-corrected INLINEFORM0 level of INLINEFORM1 (*** INLINEFORM2 , ** INLINEFORM3 ).
** alpha= 0.05, *** alpha = 0.05/223
Demographics Inference & Language Cues: LIWC has been used extensively for examining the latent dimensions of self-expression for analyzing personality BIBREF61 , depressive behavior, demographic differences BIBREF43 , BIBREF40 , etc. Several studies highlight that females employ more first-person singular pronouns BIBREF62 , and deictic language BIBREF63 , while males tend to use more articles BIBREF64 which characterizes concrete thinking, and formal, informational and affirmation words BIBREF65 . For age analysis, the salient findings include older individuals using more future tense verbs BIBREF62 triggering a shift in focus while aging. They also show positive emotions BIBREF66 and employ fewer self-references (i.e. 'I', 'me') with greater first person plural BIBREF62 . Depressed users employ first person pronouns more frequently BIBREF67 , repeatedly use negative emotions and anger words. We analyzed psycholinguistic cues and language style to study the association between depressive behavior as well as demographics. Particularly, we adopt Levinson's adult development grouping that partitions users in INLINEFORM0 into 5 age groups: (14,19],(19,23], (23,34],(34,46], and (46,60]. Then, we apply LIWC for characterizing linguistic styles for each age group for users in INLINEFORM1 .
Qualitative Language Analysis: The recent LIWC version summarizes textual content in terms of language variables such as analytical thinking, clout, authenticity, and emotional tone. It also measures other linguistic dimensions such as descriptors categories (e.g., percent of target words gleaned by dictionary, or longer than six letters - Sixltr) and informal language markers (e.g., swear words, netspeak), and other linguistic aspects (e.g., 1st person singular pronouns.)
Thinking Style:
Measuring people's natural ways of trying to analyze, and organize complex events have strong association with analytical thinking. LIWC relates higher analytic thinking to more formal and logical reasoning whereas a lower value indicates focus on narratives. Also, cognitive processing measures problem solving in mind. Words such as "think," "realize," and "know" indicates the degree of "certainty" in communications. Critical thinking ability relates to education BIBREF68 , and is impacted by different stages of cognitive development at different ages . It has been shown that older people communicate with greater cognitive complexity while comprehending nuances and subtle differences BIBREF62 . We observe a similar pattern in our data (Table TABREF40 .) A recent study highlights how depression affects brain and thinking at molecular level using a rat model BIBREF69 . Depression can promote cognitive dysfunction including difficulty in concentrating and making decisions. We observed a notable differences in the ability to think analytically in depressed and control users in different age groups (see Figure FIGREF39 - A, F and Table TABREF40 ). Overall, vulnerable younger users are not logical thinkers based on their relative analytical score and cognitive processing ability.
Authenticity:
Authenticity measures the degree of honesty. Authenticity is often assessed by measuring present tense verbs, 1st person singular pronouns (I, me, my), and by examining the linguistic manifestations of false stories BIBREF70 . Liars use fewer self-references and fewer complex words. Psychologists often see a child's first successfull lie as a mental growth. There is a decreasing trend of the Authenticity with aging (see Figure FIGREF39 -B.) Authenticity for depressed youngsters is strikingly higher than their control peers. It decreases with age (Figure FIGREF39 -B.)
Clout:
People with high clout speak more confidently and with certainty, employing more social words with fewer negations (e.g., no, not) and swear words. In general, midlife is relatively stable w.r.t. relationships and work. A recent study shows that age 60 to be best for self-esteem BIBREF71 as people take on managerial roles at work and maintain a satisfying relationship with their spouse. We see the same pattern in our data (see Figure FIGREF39 -C and Table TABREF40 ). Unsurprisingly, lack of confidence (the 6th PHQ-9 symptom) is a distinguishable characteristic of vulnerable users, leading to their lower clout scores, especially among depressed users before middle age (34 years old).
Self-references:
First person singular words are often seen as indicating interpersonal involvement and their high usage is associated with negative affective states implying nervousness and depression BIBREF66 . Consistent with prior studies, frequency of first person singular for depressed people is significantly higher compared to that of control class. Similarly to BIBREF66 , youngsters tend to use more first-person (e.g. I) and second person singular (e.g. you) pronouns (Figure FIGREF39 -G).
Informal Language Markers; Swear, Netspeak:
Several studies highlighted the use of profanity by young adults has significantly increased over the last decade BIBREF72 . We observed the same pattern in both the depressed and the control classes (Table TABREF40 ), although it's rate is higher for depressed users BIBREF1 . Psychologists have also shown that swearing can indicate that an individual is not a fragmented member of a society. Depressed youngsters, showing higher rate of interpersonal involvement and relationships, have a higher rate of cursing (Figure FIGREF39 -E). Also, Netspeak lexicon measures the frequency of terms such as lol and thx.
Sexual, Body: 
Sexual lexicon contains terms like "horny", "love" and "incest", and body terms like "ache", "heart", and "cough". Both start with a higher rate for depressed users while decreasing gradually while growing up, possibly due to changes in sexual desire as we age (Figure FIGREF39 -H,I and Table TABREF40 .)
Quantitative Language Analysis:
We employ one-way ANOVA to compare the impact of various factors and validate our findings above. Table TABREF40 illustrates our findings, with a degree of freedom (df) of 1055. The null hypothesis is that the sample means' for each age group are similar for each of the LIWC features.
*** alpha = 0.001, ** alpha = 0.01, * alpha = 0.05
We leverage both the visual and textual content for predicting age and gender.
Prediction with Textual Content:
We employ BIBREF73 's weighted lexicon of terms that uses the dataset of 75,394 Facebook users who shared their status, age and gender. The predictive power of this lexica was evaluated on Twitter, blog, and Facebook, showing promising results BIBREF73 . Utilizing these two weighted lexicon of terms, we are predicting the demographic information (age or gender) of INLINEFORM0 (denoted by INLINEFORM1 ) using following equation: INLINEFORM2 
where INLINEFORM0 is the lexicon weight of the term, and INLINEFORM1 represents the frequency of the term in the user generated INLINEFORM2 , and INLINEFORM3 measures total word count in INLINEFORM4 . As our data is biased toward young people, we report age prediction performance for each age group separately (Table TABREF42 ). Moreover, to measure the average accuracy of this model, we build a balanced dataset (keeping all the users above 23 -416 users), and then randomly sampling the same number of users from the age ranges (11,19] and (19,23]. The average accuracy of this model is 0.63 for depressed users and 0.64 for control class. Table TABREF44 illustrates the performance of gender prediction for each class. The average accuracy is 0.82 on INLINEFORM5 ground-truth dataset.
Prediction with Visual Imagery:
Inspired by BIBREF56 's approach for facial landmark localization, we use their pretrained CNN consisting of convolutional layers, including unshared and fully-connected layers, to predict gender and age from both the profile and shared images. We evaluate the performance for gender and age prediction task on INLINEFORM0 and INLINEFORM1 respectively as shown in Table TABREF42 and Table TABREF44 .
Demographic Prediction Analysis:
We delve deeper into the benefits and drawbacks of each data modality for demographic information prediction. This is crucial as the differences between language cues between age groups above age 35 tend to become smaller (see Figure FIGREF39 -A,B,C) and making the prediction harder for older people BIBREF74 . In this case, the other data modality (e.g., visual content) can play integral role as a complementary source for age inference. For gender prediction (see Table TABREF44 ), on average, the profile image-based predictor provides a more accurate prediction for both the depressed and control class (0.92 and 0.90) compared to content-based predictor (0.82). For age prediction (see Table TABREF42 ), textual content-based predictor (on average 0.60) outperforms both of the visual-based predictors (on average profile:0.51, Media:0.53).
However, not every user provides facial identity on his account (see Table TABREF21 ). We studied facial presentation for each age-group to examine any association between age-group, facial presentation and depressive behavior (see Table TABREF43 ). We can see youngsters in both depressed and control class are not likely to present their face on profile image. Less than 3% of vulnerable users between 11-19 years reveal their facial identity. Although content-based gender predictor was not as accurate as image-based one, it is adequate for population-level analysis.
We use the above findings for predicting depressive behavior. Our model exploits early fusion BIBREF32 technique in feature space and requires modeling each user INLINEFORM0 in INLINEFORM1 as vector concatenation of individual modality features. As opposed to computationally expensive late fusion scheme where each modality requires a separate supervised modeling, this model reduces the learning effort and shows promising results BIBREF75 . To develop a generalizable model that avoids overfitting, we perform feature selection using statistical tests and all relevant ensemble learning models. It adds randomness to the data by creating shuffled copies of all features (shadow feature), and then trains Random Forest classifier on the extended data. Iteratively, it checks whether the actual feature has a higher Z-score than its shadow feature (See Algorithm SECREF6 and Figure FIGREF45 ) BIBREF76 .
Main each Feature INLINEFORM0 INLINEFORM1 
RndForrest( INLINEFORM0 ) Calculate Imp INLINEFORM1 INLINEFORM2 Generate next hypothesis , INLINEFORM3 Once all hypothesis generated Perform Statistical Test INLINEFORM4 //Binomial Distribution INLINEFORM5 Feature is important Feature is important
 Ensemble Feature Selection
Next, we adopt an ensemble learning method that integrates the predictive power of multiple learners with two main advantages; its interpretability with respect to the contributions of each feature and its high predictive power. For prediction we have INLINEFORM0 where INLINEFORM1 is a weak learner and INLINEFORM2 denotes the final prediction.
In particular, we optimize the loss function: INLINEFORM0 where INLINEFORM1 incorporates INLINEFORM2 and INLINEFORM3 regularization. In each iteration, the new INLINEFORM4 is obtained by fitting weak learner to the negative gradient of loss function. Particularly, by estimating the loss function with Taylor expansion : INLINEFORM5 where its first expression is constant, the second and the third expressions are first ( INLINEFORM6 ) and second order derivatives ( INLINEFORM7 ) of the loss. INLINEFORM8 
For exploring the weak learners, assume INLINEFORM0 has k leaf nodes, INLINEFORM1 be subset of users from INLINEFORM2 belongs to the node INLINEFORM3 , and INLINEFORM4 denotes the prediction for node INLINEFORM5 . Then, for each user INLINEFORM6 belonging to INLINEFORM7 , INLINEFORM8 and INLINEFORM9 INLINEFORM10 
Next, for each leaf node INLINEFORM0 , deriving w.r.t INLINEFORM1 : INLINEFORM2 
and by substituting weights: INLINEFORM0 
which represents the loss for fixed weak learners with INLINEFORM0 nodes. The trees are built sequentially such that each subsequent tree aims to reduce the errors of its predecessor tree. Although, the weak learners have high bias, the ensemble model produces a strong learner that effectively integrate the weak learners by reducing bias and variance (the ultimate goal of supervised models) BIBREF77 . Table TABREF48 illustrates our multimodal framework outperform the baselines for identifying depressed users in terms of average specificity, sensitivity, F-Measure, and accuracy in 10-fold cross-validation setting on INLINEFORM1 dataset. Figure FIGREF47 shows how the likelihood of being classified into the depressed class varies with each feature addition to the model for a sample user in the dataset. The prediction bar (the black bar) shows that the log-odds of prediction is 0.31, that is, the likelihood of this person being a depressed user is 57% (1 / (1 + exp(-0.3))). The figure also sheds light on the impact of each contributing feature. The waterfall charts represent how the probability of being depressed changes with the addition of each feature variable. For instance, the "Analytic thinking" of this user is considered high 48.43 (Median:36.95, Mean: 40.18) and this decreases the chance of this person being classified into the depressed group by the log-odds of -1.41. Depressed users have significantly lower "Analytic thinking" score compared to control class. Moreover, the 40.46 "Clout" score is a low value (Median: 62.22, Mean: 57.17) and it decreases the chance of being classified as depressed. With respect to the visual features, for instance, the mean and the median of 'shared_colorfulness' is 112.03 and 113 respectively. The value of 136.71 would be high; thus, it decreases the chance of being depressed for this specific user by log-odds of -0.54. Moreover, the 'profile_naturalness' of 0.46 is considered high compared to 0.36 as the mean for the depressed class which justifies pull down of the log-odds by INLINEFORM2 . For network features, for instance, 'two_hop_neighborhood' for depressed users (Mean : 84) are less than that of control users (Mean: 154), and is reflected in pulling down the log-odds by -0.27.
Baselines:
To test the efficacy of our multi-modal framework for detecting depressed users, we compare it against existing content, content-network, and image-based models (based on the aforementioned general image feature, facial presence, and facial expressions.)

Figure and table captions:
Figure 1: Self-disclosure on Twitter from likely depressed users discovered by matching depressiveindicative terms
Figure 2: The age distribution for depressed and control users in ground-truth dataset
Figure 3: Gender and Depressive Behavior Association (Chi-square test: color-code: (blue:association), (red: repulsion), size: amount of each cell’s contribution)
Table 3: Statistical significance (t-statistic) of the mean of salient features for depressed and control classes 20
Figure 4: The Pearson correlation between the average emotions derived from facial expressions through the shared images and emotions from textual content for depressed-(a) and control users-(b). Pairs without statistically significant correlation are crossed (p-value <0.05)
Figure 5: Characterizing Linguistic Patterns in two aspects: Depressive-behavior and Age Distribution
Table 4: Statistical Significance Test of Linguistic Patterns/Visual Attributes for Different Age Groups with one-way ANOVA 31
Figure 6: Ranking Features obtained from Different Modalities with an Ensemble Algorithm
Table 7: Gender Prediction Performance through Visual and Textual Content
Figure 7: The explanation of the log-odds prediction of outcome (0.31) for a sample user (y-axis shows the outcome probability (depressed or control), the bar labels indicate the log-odds impact of each feature)
Table 8: Model’s Performance for Depressed User Identification from Twitter using different data modalities

Questions about this paper:
How do this framework facilitate demographic inference from social media?
How is the data annotated?
Where does the information on individual-level demographics come from?
What is the source of the user interaction data? 
What is the source of the textual data? 
What is the source of the visual data? 

Answers:
Demographic information is predicted using weighted lexicon of terms.
The data are self-reported by Twitter users and then verified by two human experts.
From Twitter profile descriptions of the users.
Sociability from ego-network on Twitter
Users' tweets
Profile pictures from the Twitter users' profiles.