# Use of Social Media Data to Diagnose and Monitor Psychotic Disorders: Systematic Review

Alban Lejeune, Benoit-Marie Robaglia, Michel Walter, Sofian Berrouiguet, Christophe Lemey

► To cite this version:

Alban Lejeune, Benoit-Marie Robaglia, Michel Walter, Sofian Berrouiguet, Christophe Lemey. Use of Social Media Data to Diagnose and Monitor Psychotic Disorders: Systematic Review. Journal of Medical Internet Research, 2022, 24 (9), pp.e36986. 10.2196/36986. hal-04287000

HAL Id: hal-04287000

https://imt-atlantique.hal.science/hal-04287000v1

Submitted on 15 Nov 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
# Use of social media data to diagnose and monitor psychotic disorders: systematic review and perspectives

Alban Lejeune, Benoit ROBAGLIA, Michel WALTER, Sofian Berrouiguet, Christophe LEMEY

Submitted to: Journal of Medical Internet Research

on: February 02, 2022

**Disclaimer:** © The authors. All rights reserved. This is a privileged document currently under peer-review/community review. Authors have provided JMIR Publications with an exclusive license to publish this preprint on its website for review purposes only. While the final peer-reviewed paper may be licensed under a CC BY license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.
Table of Contents

<table><tr><td>Original Manuscript</td><td>4</td></tr><tr><td>Supplementary Files</td><td>23</td></tr><tr><td>Multimedia Appendixes</td><td>24</td></tr><tr><td>Multimedia Appendix 1</td><td>24</td></tr><tr><td>Multimedia Appendix 2</td><td>24</td></tr><tr><td>Multimedia Appendix 3</td><td>24</td></tr><tr><td>Multimedia Appendix 4</td><td>24</td></tr></table>
# Use of social media data to diagnose and monitor psychotic disorders: systematic review and perspectives

Alban Lejeune¹ MD; Benoit ROBAGLIA²; Michel WALTER¹,³ MD, PhD; Sofian Berrouiguet¹,³,⁴,⁵ MD; Christophe LEMEY¹,³,⁵ MD

¹Hôpital Bohars CHRU BREST Bohars FR

²Institut Polytechnique de Paris Palaiseau FR

³Université de Bretagne Occidentale BREST FR

⁴IMT Atlantique Lab-STICC UMR CNRS 6285, F-29238, BREST FR

⁵LaTIM, INSERM, UMR 1101 BREST FR

**Corresponding Author:**

Alban Lejeune MD
Hôpital Bohars CHRU BREST
Rte Ploudalmézeau,
Bohars
FR

## Abstract

**Background:** Schizophrenia is a disease associated with a high burden. Improvement in care is necessary.

**Objective:** The objective of our study is to analyse the current capabilities of artificial intelligence (AI) to use social media data as a diagnostic tool for schizophrenia.

**Methods:** A systematic review of the literature was conducted on several databases, using relevant keywords.

**Results:** 93 studies were identified, among which 7 studies were included. The included studies have shown encouraging results. Social media data could be used in several ways in care of patients with schizophrenia, including the monitoring of patients after a first episode of psychosis. Several studies had major limitations. The studies were heterogeneous in quality and in their objective. Most studies did not have access to health data for the training of the algorithm. We tried to identify several limitations in the methodology of the included studies. We made recommendation for future studies.

**Conclusions:** AI could be used to analyse social media data in order to bring new tools in the care of patients with schizophrenia.

(JMIR Preprints 02/02/2022:36986)

DOI: https://doi.org/10.2196/preprints.36986

## Preprint Settings

1) Would you like to publish your submitted manuscript as preprint?

✓ **Please make my preprint PDF available to anyone at any time (recommended).**

Please make my preprint PDF available only to logged-in users; I understand that my title and abstract will remain visible to all users.

Only make the preprint title and abstract visible.

No, I do not wish to publish my submitted manuscript as a preprint.

2) If accepted for publication in a JMIR journal, would you like the PDF to be visible to the public?

✓ **Yes, please make my accepted manuscript PDF available to anyone at any time (Recommended).**

Yes, but please make my accepted manuscript PDF available only to logged-in users; I understand that the title and abstract will remain visible to all users.

Yes, but only make the title and abstract visible (see Important note, above). I understand that if I later pay to participate in <a href="http://jmir.org" target="_blank">http://jmir.org</a>.
Original Manuscript
# Use of social media data to diagnose and monitor schizophrenia: systematic review and perspectives

## Systematic review

Alban Lejeune<fnref n="1" />, MD ; Benoit Robaglia<fnref n="2" /> ; Michel Walter<fnref n="1" />,<fnref n="3" />, MD, PhD ; Sofian Berrouiguet<fnref n="1" />,<fnref n="3" />,<fnref n="4" />,<fnref n="5" />, MD ; Christophe Lemey<fnref n="1" />,<fnref n="3" />,<fnref n="5" /> MD ;

## Author for correspondence:

Alban LEJEUNE

e-mail: Alban.lejeune@gmail.com

## Present address:

Hôpital de Bohars – CHRU DE BREST

## Abstract:

Background: Schizophrenia is a disease associated with a high burden. Improvement in care is necessary. Artificial intelligence has been used to diagnose several medical conditions as well as psychiatric disorders. This technology needs huge amount of data to be efficient. Social media data could be used to improve diagnostic capabilities.

Objectives: The objective of our study is to analyse the current capabilities of artificial intelligence (AI) to use social media data as a diagnostic tool for psychotic disorders.

Methods: A systematic review of the literature was conducted on several databases (PubMed, EMBASE, Cochrane, Psychinfo and IEEE Xplore) until 12/11/2021, using relevant keywords. We used PRISMA criteria to identify, select and critically assess the quality of the relevant studies while minimizing bias. We critically analysed the methodology of the studies to detect any bias. Results would be presented.

Results: 93 studies were identified, among which 7 studies were included. The included studies have shown encouraging results. Social media data could be used in several ways in care of patients with schizophrenia, including the monitoring of patients after a first episode of psychosis. Several studies had major limitations. The studies were heterogeneous in quality and in their objective. Most studies did not have access to health data for the training of the algorithm. We tried to identify several limitations in the methodology of the included studies. We analysed the machine learning methodology and we made recommendations for future studies.

Conclusion: Our study brought encouraging results. The use of social media data for the diagnostic of psychotic disorders is promising. However, most of the included studies had major limitations which do not allow to conclude on the accuracy in clinical setting. Future studies need to use a more accurate methodology in order to get unbiased results.

Financial Support: This research received no specific grant from any funding agency, commercial or not-for-profit sectors.

## Keywords :

Schizophrenia ; Artificial intelligence; machine learning; neural network ; social media

<footnote n="1">¹URCI Mental Health Department, Brest Medical University Hospital, Brest, France</footnote>

<footnote n="2">²Insitut Polytechnique de Paris, Palaiseau, France</footnote>

<footnote n="3">³EA 7479 SPURBO, Université de Bretagne Occidentale, Brest, France</footnote>

<footnote n="4">⁴IMT Atlantique, Lab-STICC, UMR CNRS 6285, F-29238, Brest, France</footnote>

<footnote n="5">⁵LaTIM, INSERM, UMR 1101, Brest, France</footnote>
# Introduction

## Background

Schizophrenia is a chronic mental disease affecting 20 million people worldwide [1]. Although treatment with medicine and psychosocial support is effective, people with schizophrenia are less likely to seek treatment. According to the WHO, efforts to transfer care from mental health institution to the community need to be accelerated [2]. Currently, schizophrenia is a disease associated with a high burden [3,4] and efforts should be taken to reduce this burden.

Artificial intelligence (AI) has emerged as a way to improve several medical task [5,6]. AI algorithms can identify patterns in a dataset to generate diagnostic tools. In other medical disciplines, AI has already shown a good accuracy for diagnostic purpose. It can match the current diagnostic capabilities in some specific fields. In psychiatry, AI could be used for diagnostic purposes, to support daily patient assessment or drug prescription [7]. AI has also been studied to improve diagnostic and classification capabilities [8]. It has been used in suicide risk detection [9] and to diagnose mood disorders [10,11].

Despite encouraging results, it is still unclear how AI should be implemented in clinical practice. The potential of this technology is not yet fully understood. AI could be used to bring completely new tools in care. We believe that social media might be used to create new diagnostic or monitoring tools. Indeed, AI requires a large database to extract a patient's profile [12]. Social media are very broad source of information. People can disclose personal information on social media, including health related information. Studies have used those data to identify broad human traits (such as intelligence or personality traits) [13]. Subtle features of everyday language could be analysed to predict mental diseases [14–16]. Prior works showed that social media data can be used for risk classification associated with mental health diseases, such as suicide risk [17]. AI can be used to detect posts associated with mental illness [18]. Therefore, we chose to study the use of AI applied to social media because we believe it could become a brand-new tool in the care of patients with psychotic disorders.

## Focus on artificial intelligence technologies

AI can be used to perform several different tasks. ML algorithms are generally classified as supervised or unsupervised learning. The main type of ML algorithms used in the included studies was supervised learning. Supervised learning algorithms allow patterns correlated to a result to be determined in a dataset [12]. The supervised algorithms are separated into two categories: regression and classification. Classification algorithms allow data to be classified into separate categories. Patterns can be used to classify patients in a given group. Decision tree (DT), support vector machine (SVM) and random forest (RF) can perform classification tasks. Regression algorithms are used to predict quantitative data. Logistic regression (LR) and LASSO regression are part of this class [6].

Artificial neural networks are powerful artificial intelligence tools built in reference to the cortical neural structure. They can perform supervised or unsupervised tasks. Neural networks (NN) are organized in a succession of layers, with each layer having its input on the output of the previous one. Information travels from the input neurons to the hidden layers before arriving at the outcome layer where the final decision is made [5]. In a neural network, each layer functions differently. This type of AI requires significant computing power and large databases [19].

The most used machine learning algorithm in our paper is the Support Vector Machine (SVM) [20]. The idea of this algorithm is to learn a linear separation (a hyperplane) of the data points in order to classify them. As there exists an infinity of hyperplanes satisfying this condition, the SVM algorithm learns a hyperplane with the maximum margin: the maximum distance between the classes. However, as most real-life datasets cannot be linearly separated, the SVM uses what is called the "kernel trick". This transformation projects the initial data points in a higher dimensional feature space where the new is linearly separable. However, the SVM suffers from many limitations: 1) Finding a good kernel function is very hard in practice ; 2) Training is long
on large datasets ; 3) The model is very sensitive to the hyperparameters.

Overfitting means fitting an AI model on data noise or error instead of the actual relationship. It represents one of the limitations of AI. Overfitting is either due to having a small data sample or too many parameters compared to the data [12]. Cross-validation is one of the techniques used to reduce overfitting. With this technique, the dataset is split into several groups, themselves divided into training data and validation data. For each group, the statistical model is therefore trained and then validated by a different dataset. This technique reduces the risk of having an overoptimistic estimate [21]. Other techniques are used to reduce overfitting, such as the use of drop-out rate. Dropout is a regularization technique for NN to reduce overfitting and improve generalization [22]. The idea is to randomly ignore neurons (and their connections) from the neural network during training. Thus, as the NN architecture is changing at every inference, the same input of data can produce a different output. The intuition is that it forces the units to be less co-dependent and to be more robust. The main difference between cross validation and dropout lies in the source of randomness: in cross-validation, the data is randomly split into train and validation set while in dropout, the neural units are randomly discarded.

We explained machine learning parameters used in this paper in *Supplementary material 1*.

## Prior work and goal of this study

Currently, there can be an important delay in the diagnostic of psychotic disorders. These delays can vary depending on where the patient live. When the patient remains untreated with psychotic symptoms, there can be important social consequences, including in some case a risk of violence [23]. Duration of untreated psychosis could have a significant impact on the patient psychosocial condition. Early detection and early treatment could help improving the care of patients with psychotic disorders [24]. Later in the evolution of the disease, being able to diagnose sooner a relapse could have significant impact on the life quality of the patients and reduce the burden for the caregivers.

AI has been studied for the diagnose of several psychiatric conditions, including schizophrenia. A Korean team used machine learning to identify patterns on CT scan and classify patients [25]. One of the most robust studies on the classification of psychotic disorders with machine learning come from an American team [8] which used a clustering algorithm to build three biotypes using clinical data and laboratory measures. Neuroimaging and social functioning measures were used for validation. Beyond its initial goal, this study showed that several new source of data can be used to improve diagnostic capabilities. In this paper, we choose to look at social media data as a new source of data. Symptoms of schizophrenia are very broad, and some of these symptoms could impact social media activity of the patients. For example, we can hypothesize that delusion and disorganized speech or behavior could be seen on social media posts. Alterations of language are being increasingly studied in schizophrenia [15,26] and could be used for the detection of psychotic disorders on social media. Several studies have been made to show the capabilities of AI to identify mental health diseases on social media, most of them having been published in the last few years. One brief report [27] reviewed the literature until December 2020. We hypothesise that social media data could be used to follow patients with schizophrenia or at-risk for psychosis patients in order to identify sooner a first psychotic episode or a relapse of psychotic disorders. Patients could have access to care before psychotic symptoms overtake their social functioning capabilities. Thus, we could reduce the burden of schizophrenia.

## Objective of this review:

The objective of our study is to analyse prior works on the use of social media data with machine learning in order to diagnose a psychotic episode. The diagnostic capabilities would be studied in a broad sense, including the diagnostic of relapse. Therefore, we performed a systematic review of the
literature and critically discussed the included articles and their methodology.

**Methods:**

**Search strategy:**

We used Preferred Reporting Items for Systematic reviews and Meta Analyses (PRISMA) criteria to identify, select and critically assess relevant studies while minimizing bias. We went through the bibliographic databases PubMed, EMBASE, Cochrane, Psychinfo and IEEE Xplore until 12/11/2021. We based the keywords list on three fields: schizophrenia, AI and social networks. The search strategy is described in the **Supplementary material 2**. To limit the selection bias, we did not apply any restriction in terms of population. Papers found by any other mean would be included if relevant with the inclusion criteria. Studies that were not written in English were excluded.

**Study selection:**

We included clinical trials and observational studies. The primary objective was to collect studies using AI to identify users with a psychotic disorder on social media. Given the low number of published studies on this subject, studies related to any psychotic disorder would be included. Studies were selected by two independent authors. We excluded studies using social media posts as control data to study language alterations in schizophrenia.

**Data collection process:**

Data were extracted from each paper independently by using a standard form. Data were extracted by one reviewer. The following information was collected: the main author's name and country of origin; year of publication; population; social media and technology used; features; inclusion/exclusion criteria; main objective; method; main endpoint; results; main limitations.

**Synthesis method:**

The results of the articles will be presented in the form of graphs and tables. Machine learning studies often use different parameters. The graphs will group studies using similar parameters. No secondary analysis of statistical data will be done. All results presented will be only those presented by the authors of the included studies. The results will be presented with the parameters used in the articles. Machine learning parameters used in this paper are explained in **Supplementary material 1**.

**Risk of bias and quality assessment:**

Quality of included studies will be assessed using PRISMA certainty tool. This paper will be reviewed using PRISMA check list. The machine learning methodology of the articles will be evaluated by an experienced author in artificial intelligence. The risk of bias will be critically assessed by the authors.
## Results:

### Flow chart

We made a PRISMA flow chart summarizing the steps of the review (Figure 1). The initial search identified 93 studies. Based on the titles and abstracts, we excluded 78 studies. We downloaded the 7 remaining studies for full-text review. We included the 7 studies that matched the inclusion criteria (Table 1).

<figure><img src="image_2.png" /></figure>
<caption>**Figure 1.** PRISMA Flowchart of study selection process</caption>

PRISMA flowchart

PRISMA flowchart

PRISMA FLOW CHART 3

PRISMA FLOW CHART 3

Included

Identification

Authors, year of publication

PsychINFO (n = 0), JEEJ Xplore

PRISMA FLOW CHART 3

PRISMA FLOW CHART 3

Included studies were conducted in the USA (5/7) and Korea (2/7). Included studies were published between 2015 and 2021 (Figure 2). Three of the included studies were written by Birnbaum et al. team.

Figure 2. Three of the included studies were written by Birnbaum et al. team.

Figure 2. Three of the included studies were written by Birnbaum et al. team.

Figure 2. Three of the included studies were written by Birnbaum et al. team.

Figure 2. Three of the included studies were written by Birnbaum et al. team.

Figure 2. Three of the included studies were written by Birnbaum et al. team.

Figure 2. Three of the included studies were written by Birnbaum et al. team.

Figure 2. Three of the included studies were written by Birnbaum et al. team.

Figure 2. Three of the included studies were written by Birnbaum et al. team.

Figure 2. Three of the included studies were written by Birnbaum et al. team.

Figure 2. Three of the included studies were written by Birnbaum et al. team.

Figure 2. Three of the included studies were written by Birnbaum et al. team.

<figure><img src="image_3.png" /><figcaption>**Figure 2.** Number of studies included by year of publication</figcaption></figure>

## Studies design, PRISMA Quality assessment and sample size

The included studies had a retrospective design. The quality of the studies has been assessed using PRISMA criteria (Figure 3 and Supplementary file 1). The quality of the seven included studies was heterogeneous, with a mean PRISMA score of 32,3. The included studies had several risks of bias. The main bias was a classification bias with 5 of the 7 included studies not using the clinical diagnostic. The secon Samples size were mostly small, six of the seven included studies had a sample size smaller than 5 392 participants. Samples size varied between 51 and 265 396 participants.
PRISMA Quality Assessment

<figure><img src="image_4.png" /><figcaption>**Figure 3.** PRISMA Quality Assessment. Score ranking from 0 to 48, the higher the score the better the overall quality.</figcaption></figure>
Social media and technologies used:

Several AI technologies have been used. The two most commonly used algorithms were support vector machine (SVM) and random forest (RF) (Figure 4). None of the included studies used data augmentation. Cross-validation technique to prevent overfitting were used in 5 studies. One study used a dropout rate of 0.25 to prevent overfitting. The social media in which the studies were performed were Facebook, Twitter and Reddit. The studies used mainly linguistic features, but also activity related features (Table 2).

<figure><img src="image_5.png" /><figcaption>**Figure 3.** Main AI types used. SVM: support vector machine ; RF : random forest ; LR : logistic regression ; NN: neural network ; NB : native bayes ; XGB : extreme gradient boosting</figcaption></figure>
<table><thead><tr><th></th><th>Authors, year, country</th><th>Inclusion</th><th>Objective</th><th>Method</th><th>Social media and AI Technology</th><th>Outcome</th><th>Main limitations</th></tr></thead><tbody><tr><td>A Collaborative Approach to Identifying Social Media Markers of Schizophrenia by Employing Machine Learning and Clinical Appraisals</td><td>Birnbaum et al., 2017, USA [22]</td><td>Users with selfdisclosed diagnoses of schizophrenia on twitter between 2012 and 2016. 671 users from the primary dataset were randomly selected diagnoses of schizophrenia. The control group was made with a random sample of Twitter users collected from individuals without any mentions of "schizophrenia" or "psychosis" in their timeline.</td><td>To explore the utility of social media as a viable diagnostic tool in identifying individuals with schizophrenia.</td><td>Twitter profiles from the training dataset were reviewed by two physicians to determine the probability of belonging to a patient with schizophrenia. The users were then classified in three groups, "yes", "maybe" or "no". The machine learning model was then tested on unseen data of 100 users and its results were compared to those of the two physicians.</td><td>Twitter. Several algorithms including the Gaussian naïve Bayes, random forest, logistic regression, and support vector machines were trained. The best performing on cross validation was selected (random forest). Use of 10-fold-cross-validation.</td><td>Random forest yielded an AUC of 0.88.</td><td>The research team only had access to publicly available Twitter profiles. The clinical diagnostic of included subjects was unknown.</td></tr><tr><td>A deep learning model for detecting mental illness from user content on social media</td><td>Kim et al., 2020, Korea [23]</td><td>Data from 228,060 users with 488,472 posts from january 2017 to december 2018 in total were employed for the analysis.</td><td>To answer the question: Can we identify whether a user's post belongs to mental illnesses on social media ?</td><td>Collection of post data on mental health related subreddit groups. The study has collected information from 248,537 users, who wrote 633,385 posts in the seven subreddits. Six models were created for each mental disorder. Each model was created with the posts of the associated subreddit group.</td><td>Reddit. XGBoost and convolutional neural network (CNN) were employed. Use of dropout rate of 0.25 to prevent over-fitting issues.</td><td>In r/schizophrenia, accuracy of XGBoost was 86,75% vs. 94,33% with CNN.</td><td>The clinical diagnostic of included subjects was unknown.</td></tr><tr><td>Detecting relapse in youth with psychotic disorders utilizing patient-generated and patient-contributed digital data from Facebook</td><td>Birnbaum, 2019, USA [24]</td><td>Participants between the ages of 15 and 35 years old who had been diagnosed with a primary psychotic disorder screened for eligibility. Recruitment occurred between March 2016 and December 2018. 51 included participants with recent onset psychosis.</td><td>The study sought to identify and predict early relapse warning signs in social media activity collected from a cohort of individuals receiving psychiatric care for schizophrenia and other primary psychotic disorders</td><td>The authors collected 52,815 Facebook posts across 51 participants with recent onset psychosis and applied anomaly detection to explore linguistic and behavioral changes associated with psychotic relapse.</td><td>Facebook. Three one-class support vector machine (SVM) models for three different data configurations (3 differsents time periods: 1-month, 2-months, 3-months). The 1-month showed the highest specificity, which lead to build an ensemble one-class support vector machine algorithm.</td><td>The ensemble model had the highest specificity (0,71) but low sensitivity (0,38). The 3-months model had a good sensitivity (0,90) but low specificity (0,04).</td><td>Characterization of monthly periods of relative health and relative illness. The illness trajectory of psychotic disorder does not fall only in two distinct categories, as the symptoms can fluctuate over time.</td></tr><tr><td>Identifying signals associated with psychiatric illness utilizing language and images posted to Facebook</td><td>Birnbaum, 2020, USA [25]</td><td>3,404,959 Facebook messages and 142,390 images across 223 participants with schizophrenia spectrum disorders (SSD), mood disorders (MD), and healthy volunteers (HV). Participants between the ages of 15 and 35 years old were recruited from Northwell Health's psychiatry department.</td><td>The main objective was to evaluate whether it was possible to distinguish between SSD, MD, and HV based on Facebook data alone.</td><td>The authors analyzed features uploaded up to 18 months before the first hospitalization using machine learning and built classifiers that distinguished SSD and MD from HV, and SSD from MD.</td><td>Facebook. Use of Random Forest. Use of 5-fold cross validation.</td><td>Classification achieved AUC of 0.77 (HV vs. MD), 0.76 (HV vs. SSD), and 0.72 (SSD vs. MD).</td><td>Data from facebook was retrospectively collected.</td></tr><tr><td>Mining Twitter Data to Improve Detection of Schizophrenia</td><td>McManus, USA, 2015 [26]</td><td>Cohort of Twitter users who self-identify as having schizophrenia (cases) and user who do not self-identify as having any mental disorder (controls). 96 cases and 200 controls. A user was defined as case if two or more of the following held: the user self-identifies in user description; the user self-identifies in status updates; the user follows @schizotribe, a known Twitter community of users with schizophrenia.</td><td>To distinguish individuals with schizophrenia from control individuals using Twitter data.</td><td>In order to distinguish Twitter users with schizophrenia from controls, the authors extracted a set of features from each user's profile and posting history (28 numerical features).</td><td>Twitter. Several models: Naive Bayes (NB), artificial neural networks (ANNs), and support vector machines (SVMs). 5-fold cross validation on the training data. In addition to the raw feature vectors, the authors tested two transformations of the feature vectors for each of the models: log scaling of the delay between tweets and Principal Components Analysis (PCA).</td><td>The best performing model was a SVM with PCA transformed features (accuracy of 0,893). The two best performing models, based on the F1 score, both involved PCA transformed features.</td><td>Users are self-identified as patients with schizophrenia.</td></tr><tr><td>Schizophrenia Detection Using Machine Learning Approach from Social Media Content</td><td>Bae, Korea, 2017 [27]</td><td>Large corpus of social media posts collected from online Reddit sub-communities for schizophrenia (n = 12,156) in 100 control groups</td><td>This study aims to determine whether machine learning could be effectively used to detect signs of</td><td>Authors collected post from subreddit. They only included original posts and excluded the comments. They collected titles and bodies of posts along with</td><td>Reddit. Posts from the control group were randomly downsampled to create a balanced dataset (n = 13,156 posts in 100 control groups)</td><td>AUC: RF 0.97, SVM 0.91, LR 0.90, NB 0.87</td><td>The authors do not have evidence that users of the schizophrenia</td></tr></tbody></table>
<table><caption>**Table 1.** Extracted information from the included studies</caption><tr><td></td><td></td><td>(n = 247,569) comprising non-mental health related subreddits (fitness, jokes, meditation, parenting, relationships, and teaching).</td><td>schizophrenia in social media users by analyzing their social media texts.</td><td>user IDs. This resulted in 60,009 original schizophrenia posts from 16,462 users as well as 425,341 posts of the control group from 248,934 users.</td><td>group). The authors evaluated four different algorithms, support vector machine, logistic regression, naive Bayes, and random forest. Use of 10-fold cross validation.</td><td>(r/schizophrenia) are clinically diagnosed.</td></tr><tr><td>Quantifying the language of schizophrenia in social media.</td><td>Mitchell et al., USA, 2015 [28]</td><td>Corpus of users diagnosed with schizophrenia from publicly available Twitter data: 174 users with an apparently genuine self-stated diagnosis of a schizophrenia-related condition. Random twitter users for control. Equal number of schizophrenia users and community controls</td><td>To examine how linguistic signals may aid in identifying and getting help to people suffering from schizophrenia.</td><td>Each self-stated diagnosis included in this study was examined by an authors to verify that it appeared to be a real statement of a schizophrenia diagnosis, excluding jokes, quotes, or disingenuous statements. For each user, the authors obtained a set of their public Twitter posts via the Twitter API, collecting up to 3200 tweets</td><td>Twitter. Use of 10 fold cross validation. 2 ML methods: Support Vector Machines (SVM) and Maximum Entropy (MaxEnt).</td><td>SVM model reached 82,3% accuracy.</td><td>Clinical diagnostic was unknown.</td></tr></table>
<table><caption>**Table 2.** Features used in the included studies</caption><thead><tr><td>Authors, year, country</td><td>Features</td></tr></thead><tbody><tr><td>Birnbaum et al., 2017, USA [28]</td><td>The authors employed feature scaling to standardize the range of features. The LIWC features were within a normalized range of 0 to 1. The n-gram features represented frequency counts that required standardization. The min-max rescaling technique was used to scale the n-gram features to the range of 0 to 1. They employed feature selection methods to eliminate noisy features. Filter method was used where features are selected on the basis of their scores in various statistical tests for their correlation with the outcome variable. Adopting the ANOVA F test reduced the feature space from 550 features to k-best features (where k=350) by removing noisy and redundant features.</td></tr><tr><td>Kim et al., 2020, Korea [29]</td><td>Natural language toolkit (NLTK) implemented in Python to tokenize users' posts and filter frequently employed words (stop words). Porter Stemmer (a tool used to explore word meaning and source), was employed on the tokenized words, to convert a word to its root meaning and to decrease the number of word corpus.</td></tr><tr><td>Birnbaum, 2019, USA [30]</td><td>Facebook timeline data grounded in the symptomatic and functional impairments associated with psychotic disorders. These include 3 types of features. The first was word usage and psycholinguistic attributes related to affective, social, and personal experiences. The second was linguistic structural attributes, such as complexity, readability, and repeatability related to thought organization and cognitive abilities. The third was online activities relating to social functioning and diurnal patterns (friending, posting, and check-ins).</td></tr><tr><td>Birnbaum, 2020, USA [31]</td><td>Image and linguistic features.</td></tr><tr><td>McManus, USA, 2015 [32]</td><td>Features for describing emoticon use and schizophrenia-related words. The authors used the Natural Language Toolkit (NLTK) for Python to perform tokenization and lemmatization, before extracting textual features and NumPy for generating the final numeric feature vectors. The final 28 numerical features included: number of Twitter followers and number of followed users, proportion of tweets using schizophrenia-related words, emoticon usage, posting time of day, and posting rate. Two transformations of the feature vectors for each of the models: log scaling of the delay between tweets and Principal Components Analysis (PCA).</td></tr><tr><td>Bae, Korea, 2021 [33]</td><td>The linguistic features were extracted using the LIWC package and the liwcalike function from the quanteda package. Structural and psychological components of the text based on psychometrically validated dictionary, word stems, and emotions assigned to a range of categories were assessed. 22 LIWC features for each post: linguistic processes (word count and words more than six letters), function words (personal pronouns, first person singular, first-person plural, second person, third person singular, third person plural, and impersonal pronouns), time orientations (past focus, present focus, and future focus), and psychological processes (positive emotion, negative emotion, anger, fear, joy, disgust, sadness, anticipation, trust, and surprise). Linguistic features between the schizophrenia and the control (nonschizophrenia) groups were compared. The D'Agostino and Pearson's test (= 0.05) were conducted to test whether each of the linguistic features was normally distributed. As data followed a normal distribution, a two-tailed t-test was performed to determine whether the linguistic features differed between groups. The threshold of statistical significance was adjusted using the false discovery rate method to correct for multiple comparisons. All p-values &lt; 0.05.</td></tr><tr><td>Mitchell et al., USA, 2015 [34]</td><td>All NLP features were either automatically constructed or unsupervised, meaning that no manual annotation is required to create them. It is important to note that although these features were inspired by the literature on schizophrenia, they were not direct correlates of standard schizophrenia markers. Perplexity (ppl), Brown-Cluster Dist, LIWC, CLM, LIWC+CLM, LDA Topic Dist (TDist), CLM+TDist+BDist+ppl, CLM+TDist, LIWC+TDist.</td></tr></tbody></table>

Objective of the studies and performance of algorithms

Main results:

https://preprints.jmir.org/preprint/36986

[unpublished, peer-reviewed preprint]
Most studies aimed at identifying users with schizophrenia on social media. One study aimed at identifying and predicting early relapse after a hospitalization for schizophrenia [30].

The results were informed in multiple parameters, including AUC, accuracy and sensitivity/specificity. The AUC of the included studies went from 0,76 to 0,97 (Table 3). This AUC can be judged as between good to excellent. However, only one of those studies [31] used data from clinically diagnosed patients. They obtained an AUC of 0,76 The studies which informed results with the accuracy parameter obtained an accuracy from 81% to 96%. One study [30] informed the results of the predictive models with a sensitivity/specificity couple (Table 4). This study sought to identify and predict relapse of schizophrenia. The authors collected Facebook data from a small sample of patients with a diagnostic of schizophrenia who had a relapse in the following months. They used this data to build a ML model which could be used to analyse patients' data in real time. They obtained several sensitivity/specificity couple. The 3-month ensemble model brought a good sensitivity (90%) although the specificity was low (40%). This is an example of the totally novel tools which could be made using AI. A high sensitivity tool could allow physicians to detect a relapse earlier and offer earlier car to their patients. The 1-month model had a higher specificity (0,71) but low sensitivity (0,38).

<table><caption>**Table 3.** Performance in AUC of the different algorithms. SVM : support vector machine ; RF : random forest ; LR : logistic regression ; NB : native bayes ;</caption><thead><tr><th></th><th>SVM</th><th>RF</th><th>LR</th><th>NB</th></tr></thead><tbody><tr><td>Birnbaum et al., 2017, USA [28]</td><td></td><td>0,88</td><td></td><td></td></tr><tr><td>Birnbaum, 2020, USA [31]</td><td></td><td>0,76</td><td></td><td></td></tr><tr><td>Bae, Korea, 2021 [33]</td><td>0,91</td><td>0,97</td><td>0,9</td><td>0,87</td></tr></tbody></table>

<table><caption>**Table 4.** Performance in accuracy and sensitivity/specificity of the different algorithms. LR : logistic regression ; NN : neural network ; XGB/GBT : extreme gradient boosting/gradient boosted tree ; BN : Bayesian network ; RF : random forest ; DT : decision tree</caption><thead><tr><th></th><th>Accuracy</th><th>Sensitivity/specificity</th></tr></thead><tbody><tr><td>Birnbaum et al., 2017, USA [28]</td><td>81%</td><td></td></tr><tr><td>Kim et al., 2020, Korea [29]</td><td>86,75% (XGB), 94,33% (CNN)</td><td></td></tr><tr><td>Birnbaum, 2019, USA [30]</td><td></td><td>38%/71% ; 90%/40% (SVM)</td></tr><tr><td>McManus, USA, 2015 [32]</td><td>89,3% (SVM)</td><td></td></tr><tr><td>Bae, Korea, 2021 [33]</td><td>86% (NB), 89% (LR), 91% (SVM), 96% (RF)</td><td></td></tr><tr><td>Mitchell et al., USA, 2015 [34]</td><td>82,3% (SVM)</td><td></td></tr></tbody></table>

### Data used:

Most of the studies did not have access to clinical diagnostic or health data. They instead use an evaluation of users' profile by psychiatrists to access which user would be classified as having schizophrenia. The included studies used the content of the posts to train and test the models. They also used activity derived markers such as friending, check ins, number of followers, etc. Used features were chosen in order to represent the symptoms of schizophrenia described in the literature. They were particularly focus toward the identification of disorganized symptoms and cognitive abilities (Table 2).

### Discussion
## Principal results

On a statistical basis, the included studies reported good to excellent performance. Indeed, the several metrics they reported are in the top of their respective range (AUC, accuracy, sensitivity/specificity). However, this accuracy has high chances to be biased as most studies did not have access to clinical diagnostic to train the models. Most studies used the evaluation of social media profiles by trained physicians to classify patients in different groups. Thus, we cannot conclude yet on the performance of AI to detect patients with schizophrenia on social media. Two of the included studies that did have access to clinical diagnostic [30,31] showed the most conservative results (Table 3, Table 4). The included studies were heterogeneous, and some of them introduced very interesting new perspectives. After a first psychotic episode, AI and social media could be used to monitor the clinical state of the patients and detect sooner a relapse. This strategy has been studied by Birnbaum et al. team [30]. We also hypothesise it could be useful with cohort of UHR patients. Social media provide a constant flow of data, which could in theory allow the monitoring of large cohort of patients, and detect early signs of a psychotic episode. This tool could be integrated in the care of these patients with their consent.

## Critical assessment of the machine learning methodology in the included studies

The problem we are considering in this paper is a binary classification problem (whether a user profile on social media has schizophrenia) mainly based on textual data (the user's posts). In this section, we are going to introduce the key challenges we need to tackle given the collected data, analyse the methodology used in the literature and present our Machine Learning methodology to solve this problem.

## Key challenges to perform the statistical study

The first major obstacle that the statistician needs to tackle is the imbalanced dataset [35]. Indeed, in the included studies, the number of included controls is several times higher than the included cases. Thus, a “naïve” binary classifier (RF, SVM, LR, etc.) should not be used. It would indeed tend to overestimate the dominant class over the minor one. Moreover, the small sample size in the included studies suggests that there is a chance that the probability distribution in clinical practice is different than the one in the training set used for the experiments.

The second challenge imposed by the data structure is the textual data. Indeed, unlike most problems, we have to deal with what is called unstructured data, opposed to structured data where the features are well organized in a table. Here, the data we consider is text (posts of the users) so it is by definition unstructured. Therefore, Natural Language Processing techniques are called for to extract relevant features to be able to run a Machine Learning classifier. This is often a very delicate stage as the researcher often introduces an inductive bias when he decides what features to extract from the data. State-of-the art neural network algorithms like the BERT algorithm [36] is able to automatically extract features from textual data in what is call a “pre-trained stage”. It is then trained to perform the desired classification task during the “fine-tuned stage”.

## Analysis of the methodology conducted in the literature

Most previous works introduced in the last section extract their features using NLP modules such as NLTK [29,32] the LIWC package [33] or even older methods like the n-gram [28]. The major drawback of these “Bag-of-words” techniques [37] or Term Frequency-Inverse Document Frequency (Tf-IDF) methodologies [38] is that they often vectorize the textual data only based on the words (and their statistics in the sentences) without taking into account the grammar, the semantic relations between them and their context. Thus, these feature extraction methods fall short when it comes to capture semantics syntactic information or the sentiment of words [39].

## Perspectives for future studies: our recommendations for Machine Learning methodology

At the light of the challenges we mentioned and the limitations of previous works, we present in this section our approach to identify patients with schizophrenia based on social network activity.

First of all, we need to address the imbalanced dataset problem. Among the numerous approaches, we selected to use one of the most used method in practice called SMOTE [40]. The idea is to balance the dataset by creating synthetic samples from the minority class so that both classes end up being more balanced. Specifically, SMOTE selects an example from the minority class and its k neighbors (typically k=5) and creates a synthetic example as the convex combination of these 2 data points. This procedure can
produce as many synthetic samples for the minority class as needed and it guarantees that these created examples are realistic as they close in feature space to the existing ones.

However, for this data augmentation technique to work, the feature space needs to be continuous which is not the case of textual data. To alleviate this issue, we need to use what is called word embeddings. In NLP, a word embedding is a continuous representation of a word, usually a real-valued vector, that encodes the meaning of the word in a feature space [41]. Thus, 2 words with close meaning like “ill” and “disease” will be closer in the feature space than 2 words like “ill” and “car”. Yet, the remaining difficulty is to create a relevant embedding space. A popular method used in NLP is to use a pre-trained language model like BERT [36] already introduced before. This algorithm has been pre-trained by Google on the concatenation of the 2 largest datasets: BookCorpus [42], gathering 11,038 unpublished books and English Wikipedia gathering 6,427,217 articles. Not only does these high-quality embeddings allow us to use the SMOTE method to augment the minority class, but they also allow us to represent the textual data in an informative feature space. The latter will be used as input to our classification algorithm.

Finally, we need to define the binary classifier for our problem. As we have already used the BERT algorithm to create the word embeddings of the posts, the natural way would be to use it also as a binary classifier. To do so, we need to “fine-tune” it in our own dataset adding a linear layer to the existing neural network architecture. This approach for text classification has demonstrated state of the art results on 8 widely studied text classification dataset [43].

## Limitations of this study

We performed a systematic review of studies using machine learning to identify schizophrenia on social media. In regards of our hypothesis, the main limitation of our review is the small number of included studies. At the time of submission of this paper, the published studies on this subject are limited and we are not aware of any new study that would meet the inclusion criteria. Our review aim at evaluating the potential of this technology as a new tool in the care of patients with schizophrenia. Therefore, we used broad inclusion criteria to include more papers. The included studies do not have all the same objective. Thus, their results can't be compared. However, it shows the variety of use of this technology.

Most included studies (5/7) did not use the clinical diagnostic. They used instead an evaluation of the mental state of the included subjects with their public profile history and the content of their post. It is unlikely that this method is efficient in identifying accurately patients with schizophrenia. Future studies should use health data including medical diagnostic in order to be able to develop an accurate model.

Furthermore, some of the included studies showed limitations in their methodology and in their choice of ML algorithms. We analysed those limitations and used them to make recommendations for future projects.

## Ethical reflection and privacy

These tools could be useful in several ways to improve the care of patients. We could monitor social media activity of patients to detect sooner psychotic relapse. These tools could also be used to detect sooner a first psychotic disorder in patients followed for high-risk of psychosis.

The use of machine learning to predict mental health disease raise ethical questions. In which context should we use such tools? Patients should comply with the use of these tools and their data in their care. The use of machine learning would be appropriate only if patients can consent to it. Furthermore, many countries are setting a regulatory framework on AI [44,45]. AI tools will have to comply with regulation laws in order to be used in clinical setting.

To be accurate, machine learning algorithms need to be trained on unbiased and massive data. To prevent third-parties to use these technologies for other purposes, the safety of medical data is essential.

## Conclusions

AI brings new perspectives in research on schizophrenia. It could be used to monitor the clinical condition of patients at-risk for psychosis or to detect relapse of schizophrenia by following patients on social media. There are currently only a few papers published on this subject, and most of them do not allow to accurately estimate the potential of this technology. However, this technology could become a new tool in the care for patients with schizophrenia, and in the end reduce the burden. It should be developed and used in regards with ethical consideration and law regulation.

## Registration and protocol
The study was not registered.

**Financial Support**

This research received no specific grant from any funding agency, commercial or not-for-profit sectors.

**Conflicts of interest**

The authors declare none.

**Abbreviations**

AI: Artificial Intelligence

AUC: Area under the curve

DT: Decision tree

LR: Logistic regression

ML: Machine Learning

NB: Native Bayes

NLP: Natural Language Processing

NN: Neural network

SVM: Support vector machine

UHR: Ultra high risk

**Supplementary materials.**

Supplementary file 1. Machine learning parameters

Supplementary file 2. Search strategy

Supplementary file 3. Table summarizing the included studies and PRISMA certainty assessment

Supplementary file 4. PRISMA check list

**References:**

1. James SL, Abate D, Abate KH, et al. Global, regional, and national incidence, prevalence, and years lived with disability for 354 diseases and injuries for 195 countries and territories, 1990–2017: a systematic analysis for the Global Burden of Disease Study 2017. *The Lancet*. 2018;392(10159):1789–1858. doi:10.1016/S0140-6736(18)32279-7

2. WHO. Schizophrenia. https://www.who.int/news-room/fact-sheets/detail/schizophrenia

3. Charlson FJ, Ferrari AJ, Santomauro DF, et al. Global Epidemiology and Burden of Schizophrenia: Findings From the Global Burden of Disease Study 2016. *Schizophr Bull*. 2018;44(6):1195-1203. doi:10.1093/schbul/sby058

4. Cloutier M, Sanon Aigbogun M, Guerin A, et al. The Economic Burden of Schizophrenia in the United States in 2013. *J Clin Psychiatry*. 2016;77(06):764-771. doi:10.4088/JCP.15m10278

5. Sidey-Gibbons JAM, Sidey-Gibbons CJ. Machine learning in medicine: a practical introduction. BMC Med Res Methodol. 2019;19(1):64. doi:10.1186/s12874-019-0681-4

6. Deo RC. Machine Learning in Medicine. Circulation. 2015;132(20):1920-1930. doi:10.1161/CIRCULATIONAHA.115.001593

7. Lin E, Lin CH, Lane HY. Precision Psychiatry Applications with Pharmacogenomics: Artificial Intelligence and Machine Learning Approaches. IJMS. 2020;21(3):969. doi:10.3390/ijms21030969

8. Mothi SS, Sudarshan M, Tandon N, et al. Machine learning improved classification of psychoses using clinical and biological stratification: Update from the bipolar-schizophrenia network for intermediate phenotypes (B-SNIP). *Schizophrenia Research*. 2019;214:60-69. doi:10.1016/j.schres.2018.04.037

9. Lejeune A, Le Glaz A, Perron PA, et al. Artificial intelligence and suicide prevention: A systematic review. *Eur Psychiatr*. 2022;65(1):e19. doi:10.1192/j.eurpsy.2022.8

10. Ma Y, Ji J, Huang Y, et al. Implementing machine learning in bipolar diagnosis in China. *Transl Psychiatry*. 2019;9(1):305. doi:10.1038/s41398-019-0638-8
11. Gao S, Calhoun VD, Sui J. Machine learning in major depression: From classification to treatment outcome prediction. *CNS Neurosci Ther*. 2018;24(11):1037-1052. doi:10.1111/cns.13048

12. Mooney SJ, Pejaver V. Big Data in Public Health: Terminology, Machine Learning, and Privacy. *Annu Rev Public Health*. 2018;39(1):95-112. doi:10.1146/annurev-publhealth-040617-014208

13. Mori K, Haruno M. Differential ability of network and natural language information on social media to predict interpersonal and mental health traits. *J Pers*. 2021;89(2):228-243. doi:10.1111/jopy.12578

14. Rezaii N, Walker E, Wolff P. A machine learning approach to predicting psychosis using semantic density and latent content analysis. *npj Schizophr*. 2019;5(1):9. doi:10.1038/s41537-019-0077-9

15. Le Glaz A, Haralambous Y, Kim-Dufor DH, et al. Machine Learning and Natural Language Processing in Mental Health: Systematic Review. *J Med Internet Res*. 2021;23(5):e15708. doi:10.2196/15708

16. Tariq S, Akhtar N, Afzal H, et al. A Novel Co-Training-Based Approach for the Classification of Mental Illnesses Using Social Media Posts. *IEEE Access*. 2019;7:166165-166172. doi:10.1109/ACCESS.2019.2953087

17. Ophir Y, Tikochinski R, Asterhan CSC, Sisso I, Reichart R. Deep neural networks detect suicide risk from textual facebook posts. *Sci Rep*. 2020;10(1):16685. doi:10.1038/s41598-020-73917-0

18. Gkotsis G, Oellrich A, Velupillai S, et al. Characterisation of mental health conditions in social media using Informed Deep Learning. *Sci Rep*. 2017;7(1):45141. doi:10.1038/srep45141

19. Kriegeskorte N, Golan T. Neural network models and deep learning. *Curr Biol*. 2019;29(7):R231-R236. doi:10.1016/j.cub.2019.02.034

20. M. A. Hearst, S. T. Dumais, E. Osuna, J. Platt and B. Scholkopf, "Support vector machines," in *IEEE Intelligent Systems and their Applications*, vol. 13, no. 4, pp. 18-28, July-Aug. 1998, doi:10.1109/5254.708428

21. Bey R, Goussault R, Grolleau F, Benchoufi M, Porcher R. Fold-stratified cross-validation for unbiased and privacy-preserving federated learning. *Journal of the American Medical Informatics Association*. 2020;27(8):1244-1251. doi:10.1093/jamia/ocaa096

22. Srivastava, Nitish, et al. Dropout: a simple way to prevent neural networks from overfitting." The journal of machine learning research 15.1 (2014): 1929-1958). https://jmlr.org/papers/volume15/srivastava14a.old/srivastava14a.pdf

23. Látalová K. Violence and duration of untreated psychosis in first-episode patients. *Int J Clin Pract*. 2014;68(3):330-335. doi:10.1111/ijcp.12327

24. Albert N, Weibell MA. The outcome of early intervention in first episode psychosis. *International Review of Psychiatry*. 2019;31(5-6):413-424. doi:10.1080/09540261.2019.1643703

25. Jo YT, Joo SW, Shon S, Kim H, Kim Y, Lee J. Diagnosing schizophrenia with network analysis and a machine learning method. *Int J Methods Psychiatr Res*. 2020;29(1). doi:10.1002/mpr.1818

26. de Boer JN, Brederoo SG, Voppel AE, Sommer IEC. Anomalies in language as a biomarker for schizophrenia: *Current Opinion in Psychiatry*. 2020;33(3):212-218. doi:10.1097/YCO.0000000000000595

27. Feldman J, Hamlyn A, Rice T. Social media in screening and monitoring for early intervention in psychosis. *Schizophrenia Research*. 2021;238:70-72. doi:10.1016/j.schres.2021.09.023

28. Birnbaum ML, Ernala SK, Rizvi AF, De Choudhury M, Kane JM. A Collaborative Approach to Identifying Social Media Markers of Schizophrenia by Employing Machine Learning and Clinical
Appraisals. *J Med Internet Res.* 2017;19(8):e289. doi:10.2196/jmir.7956

29. Kim J, Lee J, Park E, Han J. A deep learning model for detecting mental illness from user content on social media. *Sci Rep.* 2020;10(1):11846. doi:10.1038/s41598-020-68764-y

30. Birnbaum ML, Ernala SK, Rizvi AF, et al. Detecting relapse in youth with psychotic disorders utilizing patient-generated and patient-contributed digital data from Facebook. *npj Schizophr.* 2019;5(1):17. doi:10.1038/s41537-019-0085-9

31. Birnbaum ML, Norel R, Van Meter A, et al. Identifying signals associated with psychiatric illness utilizing language and images posted to Facebook. *npj Schizophr.* 2020;6(1):38. doi:10.1038/s41537-020-00125-0

32. McManus K. Mining Twitter Data to Improve Detection of Schizophrenia. doi:PMID: 26306253

33. Bae YJ, Shim M, Lee WH. Schizophrenia Detection Using Machine Learning Approach from Social Media Content. *Sensors*. 2021;21(17):5924. doi:10.3390/s21175924

34. Mitchell M, Hollingshead K, Coppersmith G. Quantifying the Language of Schizophrenia in Social Media. In: *Proceedings of the 2nd Workshop on Computational Linguistics and Clinical Psychology: From Linguistic Signal to Clinical Reality*. Association for Computational Linguistics; 2015:11-20. doi:10.3115/v1/W15-1202

35. Haibo He, Garcia EA. Learning from Imbalanced Data. *IEEE Trans Knowl Data Eng.* 2009;21(9):1263-1284. doi:10.1109/TKDE.2008.239

36. Devlin J, Chang MW, Lee K, Toutanova K. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. *arXiv:181004805 [cs]*. Published online May 24, 2019. Accessed December 29, 2021. http://arxiv.org/abs/1810.04805

37. Yin Zhang et al. Understanding Bag-of-Words Model: A Statistical Framework. https://cs.nju.edu.cn/zhouzh/zhouzh.files/publication/ijmlc10.pdf

38. Anand Rajaraman, Jeffrey D. Ullman, Stanford Univ. *Mining of Massive Datasets*. ISBN-10 : 9781107077232

39. Naseem U, Razzak I, Khan SK, Prasad M. A Comprehensive Survey on Word Representation Models: From Classical to State-Of-The-Art Word Representation Language Models. *arXiv:201015036 [cs]*. Published online October 28, 2020. Accessed January 18, 2022. http://arxiv.org/abs/2010.15036

40. Chawla NV, Bowyer KW, Hall LO, Kegelmeyer WP. SMOTE: Synthetic Minority Over-sampling Technique. *jair*. 2002;16:321-357. doi:10.1613/jair.953

41. Jurafsky D. *Speech and Language Processing*. ISBN-10 : 9780131873216. https://web.stanford.edu/~jurafsky/slp3/ed3book_sep212021.pdf

42. Zhu Y, Kiros R, Zemel R, et al. Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books. *arXiv:150606724 [cs]*. Published online June 22, 2015. Accessed December 29, 2021. http://arxiv.org/abs/1506.06724

43. Sun C, Qiu X, Xu Y, Huang X. How to Fine-Tune BERT for Text Classification? In: Sun M, Huang X, Ji H, Liu Z, Liu Y, eds. *Chinese Computational Linguistics*. Vol 11856. Lecture Notes in Computer Science. Springer International Publishing; 2019:194-206. doi:10.1007/978-3-030-32381-3_16

44. ICMRA. Horizon Scanning Assessment Report – Artificial Intelligence. https://www.icmra.info/drupal/sites/default/files/2021-08/horizon_scanning_report_artificial_intelligence.pdf. Published April 21, 2021.
45. EUROPEAN COMMISSION. Fostering a European approach to Artificial Intelligence. https://digital-strategy.ec.europa.eu/en/library/communication-fostering-european-approach-artificial-intelligence.
# Supplementary Files

Untitled.

URL: http://asset.jmir.pub/assets/0dd1e2c26dbadbcb0a9d8ad019b2069a.xlsx
# Multimedia Appendixes

Machine learning parameters.

URL: http://asset.jmir.pub/assets/d13fc454fc3e61ae05139a9237273cc2.docx

Search strategy.

URL: http://asset.jmir.pub/assets/d695c18db2f67f2290f6d6891a25c6e7.docx

Table summarizing the included studies and PRISMA certainty assessment.

URL: http://asset.jmir.pub/assets/db478d123cf3b65c5dd9c52837e4721b.xlsx

PRISMA check list.

URL: http://asset.jmir.pub/assets/01a1ec4b4b5c27b0181eeb779325294c.docx