JMIR MEDICAL INFORMATICS

Zunic et al

Review

Sentiment Analysis in Health and Well-Being: Systematic Review
Anastazia Zunic, MSc; Padraig Corcoran, PhD; Irena Spasic, PhD
School of Computer Science & Informatics, Cardiff University, Cardiff, United Kingdom

Corresponding Author:
Irena Spasic, PhD
School of Computer Science & Informatics
Cardiff University
The Parade
Cardiff, CF24 3AA
United Kingdom
Phone: 44 02920870320
Email: spasici@cardiff.ac.uk

Abstract
Background: Sentiment analysis (SA) is a subfield of natural language processing whose aim is to automatically classify thesentiment expressed in a free text. It has found practical applications across a wide range of societal contexts including marketing,economy, and politics. This review focuses specifically on applications related to health, which is defined as “a state of complete
physical, mental, and social well-being and not merely the absence of disease or infirmity.”
Objective: This study aimed to establish the state of the art in SA related to health and well-being by conducting a systematic
review of the recent literature. To capture the perspective of those individuals whose health and well-being are affected, we
focused specifically on spontaneously generated content and not necessarily that of health care professionals.
Methods: Our methodology is based on the guidelines for performing systematic reviews. In January 2019, we used PubMed,a multifaceted interface, to perform a literature search against MEDLINE. We identified a total of 86 relevant studies and extracted
data about the datasets analyzed, discourse topics, data creators, downstream applications, algorithms used, and their evaluation.
Results: The majority of data were collected from social networking and Web-based retailing platforms. The primary purposeof online conversations is to exchange information and provide social support online. These communities tend to form aroundhealth conditions with high severity and chronicity rates. Different treatments and services discussed include medications,vaccination, surgery, orthodontic services, individual physicians, and health care services in general. We identified 5 roles withrespect to health and well-being among the authors of the types of spontaneously generated narratives considered in this review:
a sufferer, an addict, a patient, a carer, and a suicide victim. Out of 86 studies considered, only 4 reported the demographic
characteristics. A wide range of methods were used to perform SA. Most common choices included support vector machines,naïve Bayesian learning, decision trees, logistic regression, and adaptive boosting. In contrast with general trends in SA research,
only 1 study used deep learning. The performance lags behind the state of the art achieved in other domains when measured by
F-score, which was found to be below 60% on average. In the context of SA, the domain of health and well-being was found to
be resource poor: few domain-specific corpora and lexica are shared publicly for research purposes.
Conclusions: SA results in the area of health and well-being lag behind those in other domains. It is yet unclear if this is becauseof the intrinsic differences between the domains and their respective sublanguages, the size of training datasets, the lack ofdomain-specific sentiment lexica, or the choice of algorithms.
(JMIR Med Inform 2020;8(1):e16023) doi: 10.2196/16023
KEYWORDS
sentiment analysis; natural language processing; text mining; machine learning

Introduction
Sentiment analysis (SA), also known as opinion mining, is asubfield of natural language processing (NLP) whose aim is toautomatically classify the sentiment expressed in a free text. Its
https://medinform.jmir.org/2020/1/e16023

XSL• FO
RenderX

origins can be traced to the 1990s including methods forclassifying the point of view [1], predicting the semanticorientation of adjectives [2], subjectivity classification [3], etc.
However, its rapid growth is correlated with the advent of Web
2.0 and the increasing availability of user-generated data such
JMIR Med Inform 2020 | vol. 8 | iss. 1 | e16023 | p. 1
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Zunic et alas product and service reviews as well as the proliferation ofsocial media communication channels.

we posed no restrictions on the platforms used to generate thedata.

SA has found practical applications across a wide range ofsocietal contexts including marketing, economy, and politics[4-8]. This review focuses specifically on applications related
to health, which is defined as “a state of complete physical,
mental, and social well-being and not merely the absence ofdisease or infirmity” [9]. The well-being itself is considered to
be a perceived or subjective state, that is, it can varyconsiderably across individuals with similar circumstances [10].
This makes well-being an ideal case study for SA. However,when it comes to matters of health, modern society tends to be
preoccupied with the negative phenomena such as diseases,
injuries, and disabilities [11], which makes SA in this domainchallenging. For instance, for a patient with a chronic condition,
having a good quality of life will not necessarily depend on theabsence of associated symptoms, but rather on the extent towhich they are managed and controlled. However, the negative
connotation of health symptoms tends to skew the SA results
toward the negative spectrum.

The remainder of the paper is organized as follows. The Methods
explains the methodology of this systematic review in detail.
Results presents the findings of the review, followed by adiscussion. The final section summarizes the main findings ofthe review.

To establish the state of the art in SA related to health andwell-being, we conducted a systematic review of the recent
literature. To capture the perspective of those individuals whose
health and well-being are affected, we focused specifically on
spontaneously generated content and not necessarily that ofhealth care professionals. This differentiates this review fromothers conducted on related topics. For example, Denecke andDeng [12] reviewed SA in medical settings, but focused on theword usage and sentiment distribution of clinical data, such as
nurse letters, radiology reports, and discharge summaries, while
public data shared by the likes of patients and caregivers wererestricted to 2 websites. On the contrary, Gohil et al [13] dealt
with user-generated data, but only considered Twitter, whereas

Methods
Guidelines
Our methodology is based on the guidelines for performing
systematic reviews described by Kitchenham [14]. It is
structured around the following steps:
1.
2.

3.
4.
5.
6.

Research questions define the scope, depth, and the overall
aim of the review.
Search strategy is an organized process designed to identify
all studies that are relevant to the research questions in an
efficient and reproducible manner.
Inclusion and exclusion criteria define the scope of asystematic review.
Quality assessment refers to a critical appraisal of included
studies to ensure that the findings of the review are valid.
Data extraction is the process of identifying the relevant
information from the included studies.
Data synthesis involves critical appraisal and synthesis ofevidence to support the findings of the review.

Research Questions
The overarching topic of this review is the SA of spontaneously
generated narratives in relation to health and well-being. The
main aim of this review was to answer the research questions
given in Table 1.

Table 1. Research questions.

aID

Question

RQ1

What are the major sources of data?

RQ2

What is the originally intended purpose of spontaneously generated narratives?

RQ3

What are the roles of their authors within health and care?

RQ4

What are their demographic characteristics?

RQ5

What areas of health and well-being are discussed?

RQ6

What are the practical applications of SAa?

RQ7

What methods have been used to perform SA?

RQ8What is the state-of-the-art performance of SA?

RQ9What resources are available to support SA related to health and well-being?

SA: sentiment analysis.

Search Strategy
To systematically identify articles relevant to SA related tohealth and well-being, we first considered relevant data sources:
the Cochrane Library [15], MEDLINE [16], EMBASE [17],and CINAHL [18]. MEDLINE was chosen as the most diverse
data source with respect to the topics covered and publication
https://medinform.jmir.org/2020/1/e16023

XSL• FO
RenderX

types. MEDLINE is a premier bibliographic database that
contains more than 29 million references to articles in life
sciences and biomedicine. Its coverage dates back to 1946, andits content is updated daily. It covers publications of various
types, for example, journal articles, case reports, conference
papers, letters, comments, guidelines, and clinical trials. Its
content is systematically indexed by Medical Subject Headings
JMIR Med Inform 2020 | vol. 8 | iss. 1 | e16023 | p. 2
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS(MeSH), a hierarchically organized terminology for cataloging
biomedical information, to facilitate identification of relevant
articles. For example, it defines the term natural language
processing as “computer processing of a language with rulesthat reflect and describe current usage rather than prescribedusage.” Therefore, this term can be used to identify articles on
this topic even when they use alternative terminology, forexample, “sentiment analysis,” “information retrieval,” and“text mining.” We used PubMed, a multifaceted interface, tosearch MEDLINE.
Having chosen MEDLINE as the primary source of information,
the next step in developing our search strategy was to define asearch query that adequately describes the chosen topic—SA
related to health and well-being. Given the MEDLINE’s focus
on biomedicine, inclusion of terms related to health andwell-being was considered redundant. Specifically, they could
improve the precision of the search (ie, reduce the number ofirrelevant articles retrieved), but could only decrease the recall(the number of relevant articles retrieved). Given the relative
recency of research into SA and its applications in biomedicine,
we expected a query focusing solely on SA to retrieve amanageable number of articles, which could then be reviewed
manually. The search query was defined as follows:((sentiment[Title] OR sentiments[Title] OR
opinion[Title] OR opinions[Title] OR emotion[Title]
OR emotions[Title] OR emotive[Title] ORaffect[Title] OR affects[Title] OR affective[Title])

Zunic et al
AND (“sentiment classification” OR “opinion
mining” OR “natural language processing” OR NLP
OR “text analytics” OR “text mining” OR“F-measure” OR “emotion classification”)) OR“sentiment analysis”
The search performed on January 24, 2019, retrieved a total of
299 articles. Notably, no articles published before 2011 wereretrieved, which confirmed our hypothesis about the relative
recency of research into SA and its applications in biomedicine.

Selection Criteria
To further refine the scope of this systematic review, we defineda set of inclusion and exclusion criteria (see Tables 2 and 3) toselect the most appropriate articles from those matching thesearch query.
Two annotators independently screened the retrieved articles
against inclusion and exclusion criteria and achieved theinterannotator agreement of 0.51 calculated using Cohen kappa
coefficient [19]. Disagreements were resolved by the third
independent annotator. A total of 95 articles were retained forfurther processing.
To ensure the rigorousness and credibility of selected studies,
they were additionally evaluated against the quality assessment
criteria defined in Table 4. A total of 9 studies were found not
to match the given criteria. This further reduced the number of
selected articles to 86. Figure 1 summarizes the outcomes of
the 4 major stages in the systematic literature review.

Table 2. Inclusion criteria.
ID

Criterion

IN1

The input text represents spontaneously generated narrative.

IN2

The input text discusses topics related to health and well-being.

IN3

The input text captures the perspective of an individual personally affected by issues related to health and well-being (eg, patient or carer)rather than that of a health care professional.

IN4

Sentiment is analyzed automatically using natural language processing.

Table 3. Exclusion criteria.
ID

Criterion

EX1

Sentiment analysis is performed in a language other than English.

EX2

The article is written in a language other than English.

EX3

The article is not peer reviewed.

EX4

The article does not describe an original study.

EX5

The article is published before January 1, 2000.

EX6

The full text of the article is not freely available to academic community.

https://medinform.jmir.org/2020/1/e16023

XSL• FO
RenderX

JMIR Med Inform 2020 | vol. 8 | iss. 1 | e16023 | p. 3
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Zunic et al

Table 4. Quality assessment criteria.
ID

Criterion

QA1

Are the aims of the research clearly defined?

QA2

Is the study methodologically sound?

QA3

Is the method explained in sufficient detail to reproduce the results?

QA4

Were the results evaluated systematically?

Figure 1. Flow diagram of the literature review process.

Data Extraction and Synthesis
Data extraction cards were designed to aid the collection of
information relevant to the research questions. They included

items described in Table 5. The selected articles were read infull to populate the data extraction cards, which were then used
to facilitate narrative synthesis of the main findings.

Table 5. Data extraction framework.

a

Item

Description

Data

Provenance, purpose, selection criteria, size, and use.

Topic

General topic discussed in the given dataset including medical conditions and treatments.

AuthorAuthor (data creator) demographics and their role in health care.

Application

Downstream application of SAa results.

MethodType of SA method used, feature selection/extraction, and any resources used to support implementation of the method.

Evaluation

Measures used to evaluate the results, specific results reported, baseline method used, and improvements over thebaseline (if any).

SA: Sentiment analysis.

https://medinform.jmir.org/2020/1/e16023

XSL• FO
RenderX

JMIR Med Inform 2020 | vol. 8 | iss. 1 | e16023 | p. 4
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Zunic et al

Results

collected from the section of joint and muscle pain relief
treatments [52].

Data Provenance

Mainstream social media provide a generic platform to engage
patients. One of their advantages in this context is that many
patients are already active users of these platforms, thus
effectively lowering barrier to entry to engaging patients online.
However, the use of social media in the context of disclosing
protected health information may raise ethical issues such as
those related to confidence and privacy. The need to engage
patients online while fully complying with data protection
regulations has led to the proliferation of websites and networks
developed specifically to provide a safe space for sharing
health-related information online. This systematic review
identified 10 platforms of this kind that have been utilized in
21 studies (see Table 6 for details).

This section discusses the main properties of data used as input
for SA in relation to research questions RQ1 and RQ2. The
majority of data were collected from the mainstream social
multimedia and Web-based retailing platforms, which provide
the most pervasive user base together with application
programming interfaces (APIs) that can support large-scale datacollection. Not surprisingly, 26 studies [20-45] used data sourced
from Twitter, a social networking service on which users post
messages restricted to 280 characters (previously 140). Twittercan be accessed via its API from a range of popular
programming languages using libraries such as TwitterR [22],Twitter4J in Java [29,41], and Tweepy in Python [45].
Facebook, another social networking service, was used to collectuser posts regarding Chron's disease [46] and depression andanxiety [47]. Comments posted on Instagram, a photo andvideo-sharing social networking service, were used to predict
depression [48]. A total of 2 studies used data from YouTube,a video-sharing website, which allows users to share videos andcomment on them. These studies collected comments on videos
related to proanorexia [49] and Invisalign experience [50].
Reddit, a social news aggregation, Web content rating, anddiscussion website, was used to learn to differentiate between
suicidal and nonsuicidal comments [51]. Amazon, a Web-based
retailer, allows users to submit reviews of products. Customers
may comment or vote on the reviews, much in the spirit of social
networking websites. Amazon is the largest single source ofconsumer reviews on the internet. Amazon reviews were

Due to ethical concerns, the data used in these studies are usually
not released publicly to support further research and evaluation.
Only one such dataset has been published. The eDiseases dataset
used in 2 studies [53,54] contains patient data from the MedHelp
website (see Table 6). The dataset contains 10 conversations
from 3 patient communities, allergies, Crohn disease, and breast
cancer, which according to a medical expert, exhibit high degreeof heterogeneity with respect to health literacy anddemographics. The conversations were selected randomly outof those that contained at least 10 user posts. Individual
sentences were annotated with respect to their factuality(opinion, fact, or experience) and polarity (positive, negative,
or neutral). Annotation was performed by 3 frequent users ofhealth forums. With approximately 3000 annotated sentenceswith high degree of heterogeneity, this dataset represents asuitable testbed for evaluating SA in the health domain.

Table 6. Health-related websites and networks.
WebsiteDescription

Used in

RateMDs [55]Allows users to post reviews about health care staff and services.

[56-58]WebMD [59]Publishes content about health and care topics, including fora that allow users to create or partic- [23,60,61]
ipate in support groups and discussions.

Ask a Patient [62]Allows users to share their personal experience about drug treatments.

[61,63]

DrugLib.com [64]Allows users to rate and review prescription drugs.

[23,61,63,65]

Breastcancer.org [66]A breast cancer community of 218,615 members in 81 fora discussing 154,832 topics.

[67,68]

MedHelp [69]Allows users to share their personal experiences and evidence-based information across 298
topics related to health and well-being.

[21,53,54,70,71]

DailyStrength [72]A social networking service that allows users to create support groups across 34 categories related [23,27]
to health and well-being.

Cancer Survivors Network [73]A social networking service that connects users whose lives have been affected by cancer andallows them to share personal experience and expressions of caring.

NHS website [77] (formerly NHS Choices)

The primary public facing website of the United Kingdom’s National Health Service (NHS) with [78]more than 43 million visits per month. It provides health-related information and allows patientsto provide feedback on services.

DiabetesDaily [79]A social networking service that connects people affected by diabetes where they can trade advice [80]and learn more about the condition.

As illustrated by the studies discussed thus far, spontaneously
generated narrative used in SA typically coincides with theuser-generated content, that is, content created by a user of an
https://medinform.jmir.org/2020/1/e16023

XSL• FO
RenderX[74-76]

online platform and made publicly available to other users. The
fifth i2b2/VA/Cincinnati challenge in NLP for clinical data [81]represents an important milestone in SA research related to
JMIR Med Inform 2020 | vol. 8 | iss. 1 | e16023 | p. 5
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS
health and well-being. The challenge focused on the task ofclassifying emotions from suicide notes. The corpus used forthis shared task contained 1319 written notes left behind bypeople who died by suicide. Individual sentences were annotatedwith the following labels: abuse, anger, blame, fear, guilt,
hopelessness, sorrow, forgiveness, happiness, peacefulness,
hopefulness, love, pride, thankfulness, instructions, andinformation. A total of 24 teams used these data to develop theirclassification systems and evaluate their performance, out of
which 19 teams published their results [82-100].
As discussed above, the vast majority of data used in studies
encompassed by this review represent user-generated content
originating from online platforms. We can differentiate between
2 main types of user-generated content: customer reviews anduser comments. A customer review is a review of a product orservice made by someone who purchased, used, or had
experience with the product or service. The main class of
products reviewed in the datasets considered here are medicinal
products. Product reviews were collected from Amazon, butalso from specialized websites such as Ask a Patient and
DrugLib.com. These reviews provide users with additional
information about a product’s efficacy and possible side effects
typically described in layman’s terms, thus lowering a barrier
to participation in health care linked to health literacy andpotentially providing better support for shared decision making.
Other websites such as RateMDs and the National HealthService (NHS) website allow users to review health care services
they received including health care professionals who provide

Zunic et alsuch services. Service reviews can be used by health careproviders to identify opportunities to improve the quality ofcare.
Web 2.0 gave rise to the publishing of one’s own content andcommenting on other user’s content on online platforms thatprovide social networking services. On mainstream social media
such as Twitter, Facebook, Instagram, YouTube, and Reddit,
patients can organize their fora around groups, hashtags, or
influencer users. The primary purpose of these conversations
is to exchange information and provide social support online.
More specialized websites such as those described in Table 6
serve the same purpose. Spontaneous narratives published onthese media represent a valuable source for identifying patients’needs, especially the unmet ones.

Data Authors
This section discusses the characteristics of those who authoredthe types of narratives discussed in the previous section. We
first discuss their roles within health and care in relation toresearch questions RQ3 followed by their demographic
characteristics in relation to question RQ4.
We have identified 5 roles with respect to health and well-being
among the authors of the types of spontaneously generated
narratives considered in this review: sufferer, addict, patient,carer, and suicide victim (see Table 7). Some of these roles may
overlap, for example, a sufferer or an addict can also be a patient
if they are receiving a medical treatment for their medical
condition.

Table 7. The roles of authors with respect to health and well-being.
Role

Description

Studies

Sufferer

A person who is affected by a medical condition.

[21,23,27,46,53,54,60,61,63,65,67,68,70,71,74-76,101,102]

AddictA person who is addicted to a particular substance.

[26,103-106]

Patient

A person receiving or registered to receive medical treatment. [21,23,27,46,50,53,54,56-58,60,61,63,65,67,68,70,71,74-76,78,80,102,107,108]Carer

A family member or friend who regularly looks after a sickor disabled person.

[23,56-58,60,61,74-76]

Suicide victim

A person who has committed suicide.

[51,82-100]

Demographic factors refer to socioeconomic characteristics
such as age, gender, education level, income level, marital status,
occupation, and religion. Most studies involving clinical datasummarize the demographics of study participants statisticallyto illustrate the extent to which its findings can be generalized.
Our focus on spontaneously generated narratives implies thatthe corresponding studies could not mandate the collection ofdemographic factors. Instead, they can only rely on information
provided by users in good faith. Different Web platforms may
record different demographic factors, which may or may not be
accessible to third parties. Nonmandatory user information will
typically give rise to missing values. Moreover, demographic
information is difficult to verify online, which raises thehttps://medinform.jmir.org/2020/1/e16023

XSL• FO
RenderX

concerns over the validity of such information even when it ispublicly available.
Table 8 states which demographic factors, if any, are recorded
when a user registers an account on the given online services
and which ones are accessible online. Only age and gender are
routinely collected, but not necessarily shared publicly.
Therefore, it should be noted when SA is used to analyze suchdata to address a clinical question, then the findings should beinterpreted with caution as it may not be possible to generalize
them across the relevant patient population. Out of 86 studies
considered in this review, only 4 reported the demographics
factors, [49,67,101,103]. Age was discussed in 3 studies[67,101,103], whereas gender was analyzed in 2 studies[49,103].

JMIR Med Inform 2020 | vol. 8 | iss. 1 | e16023 | p. 6
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Zunic et al

Table 8. Recording and accessing demographic factors.

aPlatform

Age

Gender

Education level

Income level

Marital status

Occupation

Religion

Used in

Twitter

?a/Ub

?/Nc

Xd/N

X/N

X/N

X/N

X/N

[20-45]

Facebook

Me/U

M/U

?/U

X/N

?/U

?/U

?/U

[46,47]Instagram

M/U

M/U

X/N

X/N

X/N

X/N

X/N

[48]

YouTube

M/U

?/U

X/N

X/N

X/N

X/N

X/N

[49,50]

Reddit

X/N

X/N

X/N

X/N

X/N

X/N

X/N

[51]

Amazon

X/N

X/N

X/N

X/N

X/N

X/N

X/N

[52]

RateMDs

X/N

X/N

X/N

X/N

X/N

X/N

X/N

[56-58]

WebMD

M/U

?/U

X/N

X/N

X/N

X/N

X/N

[23,60]Ask a Patient

M/Yf

M/Y

X/N

X/N

X/N

X/N

X/N

[61,63]

DrugLib.com

M/Y

M/Y

X/N

X/N

X/N

X/N

X/N

[23,61,63,65]

Breastcancer.org

M/U

?/U

X/N

X/N

X/N

?/U

X/N

[67,68]

MedHelp

?/U

M/U

X/N

X/N

X/N

X/N

X/N

[21,53,54,70,71]

DailyStrength

M/U

M/U

X/N

X/N

X/N

X/N

X/N

[23,27]

Cancer Survivors Network ?/U

?/U

X/N

X/N

X/N

X/N

X/N

[74-76]

NHSg website

?/U

?/U

?/U

X/N

X/N

X/N

X/N

[78]

DiabetesDaily

?/U

?/U

X/N

X/N

X/N

?/U

X/N

[80]

? indicates optional recording.

b

U: user-specific access.

c

N: not accessible online.

d

X: recording not available.

e

M: recording mandatory.

f

Y: accessible online.

g

NHS: National Health Service.

Areas and Applications
This section focuses on the areas of health and well-being
encompassed by the given datasets in relation to researchquestion RQ5. These areas provide context for the practicalapplications of SA, which are discussed in relation to question
RQ6.
Support groups provide patients and carers with practicalinformation and emotional support to cope with health-related
problems. An ability to record these conversations online offers
an opportunity to study and measure unmet needs of different
health communities. These communities tend to form aroundhealth conditions with high severity and chronicity rates. Not
surprisingly, SA has been used to study communities formed
around cancer, mental health problems, chronic conditions fromasthma to multiple sclerosis, pain associated with these
conditions, eating disorders, and addiction (see Table 9[109-112]). Studying the opinion expressed in spontaneous
narratives offers an opportunity to improve health care services
by taking into account unforeseen factors. For example, thecontent of social media can be used to continually monitor theeffects of medications after they have been licensed to identify
previously unreported adverse reactions [27]. Similarly, SA canbe used to differentiate between suicidal and nonsuicidal posts,
after which a real-time online counseling can be offered [51].
https://medinform.jmir.org/2020/1/e16023

XSL• FO
RenderX

The provision of health care services itself has been the subjectof SA. Table 10 outlines different treatments and services
discussed by patients whose opinions have been studied bymeans of SA. Patient reviews of specific medications can
support their decision making but can also be explored tosupport shared decision making, ultimately influencing healthoutcomes and health care utilization. Patient reviews of healthcare services can reveal how the services are experienced inpractice[20,56-58,78,107,108,113],
help
improvecommunication between patients and health care providers, andidentify opportunities for service improvement, again
influencing health outcomes and health care utilization. In termsof disease prevention, it is important to understand potential
obstacles to population-based intervention approaches such as
vaccination [25,32,33,110]. Patients’ opinions can help healthpractitioners gain insight into the reasons why some patients
may opt for traditional and complementary medicine [109].
Alternatively, understanding patients’ experience with different
treatments can support creation of personalized therapy plans[45]. SA can be used to continually monitor online conversations
to automatically create alerts for community moderators when
additional support is needed [60,74]. Practical support can beprovided by making online health information more accessible[53,54]. In particular, such information can help carers provide
better care to patients [70].
JMIR Med Inform 2020 | vol. 8 | iss. 1 | e16023 | p. 7
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Zunic et al

Table 9. Health-related problems studied by sentiment analysis.
Problem

Studied inCancer[44,45,75,109], oral [110], lung [71], breast [53,54,67,68,70,71,74,76], cervical [110], prostate [21], colorectal [30,74,76],and cancer screening [38]Mental health[34], depression [47,48,111], suicide [51,82-100], and dementia [40]Chronic condition

diabetes [41,43,44,60,71,80], Chron's disease [46,53,54], multiple sclerosis [22], and asthma [101]Eating disorderobesity [36] and anorexia [49]Addictionsmoking [103-106] and cannabis [26]Pain

[24,52], fibromyalgia [35]Infectious diseases

Ebola [28] and latent infectious disease [37]Quality of life

[29,42,112]

Table 10. Health care treatments studied by sentiment analysis
Treatment

Studied in

Medication[23,27,46,61,63,65,102]

Vaccine[25,32,110]

Surgery[114]

Orthodontic[39,50]Physician[56-58]

Health care

[20,31,78,107,108,113]

Methods Used for Sentiment Analysis
This section studies a range of methods and theirimplementations that have been used to perform SA in relation
to research question RQ7. We also describe their classification
performance to establish the state of the art in relation toquestion RQ8. SA requires an algorithm to classify sentimentassociated with narrative text. Typically, sentiment is considered
to be positive, negative, or neutral. Therefore, the problem of
SA can be defined as that of multinomial classification. Whenan order can be imposed on the considered classes, then SA canbe viewed as an ordinal regression problem.
Traditionally, lexicon-based SA methods classify the sentiment
as a function of the predefined word polarities [28,31,37,43,50].
Lexicon-based methods are the simplest kind of rule-based
methods. In general, rather than focusing on individual words,
rule-based methods focus on more complex patterns, typically

https://medinform.jmir.org/2020/1/e16023

XSL• FO
RenderX

implemented
using
regularexpressions[85,87,88,90,93-95,100,112]. Most often, these rules are used
to extract features pertinent to SA, whereas the actualclassification is based on machine learning algorithms. Table
11 provides information about specific machine learningalgorithms used. Specific implementations of these algorithms
that were used to support experimental evaluation are listed inTable 12.
To establish the state of the art, we summarized the performanceof different classification algorithms in Tables 13 and 14. The
results are provided in chronological order. Classification
performance measures reported include accuracy (A), precision(P), recall (R), and F-measure, which are calculated using true
positives (TP), true negatives (TN), false positives (FP), andfalse negatives (FN) in the following manner:
A=(TP+TN)/(TP+FP+TN+FN),
P=TP/(TP+FP), R=TP/(TP+FN), F=2PR/(P+R)

JMIR Med Inform 2020 | vol. 8 | iss. 1 | e16023 | p. 8
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Zunic et al

Table 11. Machine learning algorithms used in sentiment analysis related to health and well-being.
Algorithm

Description

Used in

Support vector machine

Builds a classification model as a hyperplane [25,26,32,33,47,53,67,76,78,82-89,91,92,95,97,98,106,107,110,114]
that maximizes the margin between the training
instances of 2 classes.

Naïve Bayes classifier

A probabilistic classifier based on Bayes theo- [26,28,32,38,53,60,61,63,78,93,94,97,98,106,107,114]
rem and an assumption that features are mutually independent.

Maximum entropy

A probabilistic classifier based on the principle [61,63,67,96,98]
of maximum entropy.

Conditional random fields

A method for labeling and segmenting struc- [85,98]
tured data based on a conditional probability
distribution over label sequences given an observation sequence.

Decision tree learning

A method that uses inductive inference to approximate a discrete-valued target function,
which is represented by a decision tree.

Random forest

An ensemble learning method that fits multiple [32,53]
decision trees on various data samples andcombines them to improve accuracy and control overfitting.

AdaBoost

AdaBoost combines multiple weak classifiers [67,74-76]
into a strong one by retraining and weighing
the classifiers iteratively based on the accuracy
achieved.

k-nearest neighbors

A nonparametric, instance-based learning algo- [47,87]
rithm based on the labels of the k nearest
training instances.

Logistic regression

A method for modeling the log odds of the di- [26,76,99,111]
chotomous outcome as a linear combinationof the predictor variables.

Convolutional neural network

A feed-forward neural network that learns to [30]extract salient features that are useful for the
given prediction task. Convolutions are used
to filter features by using nonlinear functions.
Pooling can then be used to reduce the dimensionality.

[47,78,87,97,107,111]

Table 12. Implementations of machine learning algorithms.
Library
SVM

light

Description[115]

PySVMLight [116]aUsed ina

An implementation of SVMs in C.

[88,91,98]

A Python binding to the SVMlight (see above).

[83]

LIBLINEAR (LIBSVM) Integrated software for support vector classification, regression, and[117]
distribution estimation. It supports multiclass classification.

[32,76,82,84-86,89,95,118]

Weka [119]A Java library that implements a collection of machine learning algorithms.

[20,23,32,53,54,56,60,76,78,93,94,118]

scikit-learn [120]A Python library that implements a collection of machine learning al- [51,104,109]
gorithms.

Keras [121]A high-level neural networks APIb written in Python.

[45]

TextBlob [122]A Python library that supports NLPc and implements a collection ofmachine learning algorithms.

[45,51]

SVM: support vector machine.

b

API: application programming interface.

c

NLP: natural language processing.

https://medinform.jmir.org/2020/1/e16023

XSL• FO
RenderX

JMIR Med Inform 2020 | vol. 8 | iss. 1 | e16023 | p. 9
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Zunic et al

Table 13. Classification performance.
Study

Algorithma

Accuracy (%)

Precision (%)

Recall (%)

F-measure
(%)

[110]

SVMb

70

—c

—

—

[82]

SVM

—

55.72

54.72

55.22

[83]

SVM

—

—

—

53.31

[84]

SVM

—

49

46

47

d

[85]

SVM + CRF + rules

—

60.1

36.8

45.6

[86]

SVM

—

51.9

48.59

50.18

[87]

KNNe, DTf+ SVM + rules

—

49.92

50.55

50.23

[88]

SVM + rules

—

41.79

55.03

47.5

[89]

SVM, rules

—

53.8

53.9

53.8

[90]

rules

—

45.98

44.57

45.27

[91]

SVM

—

46

54

49.41

[92]

SVM

—

55.09

48.51

51.59

[93]

NBg, rules, NB + rules

—

57.09

55.74

56.4

[94]

NB + rules

—

54.96

51.81

53.34

[95]

SVM, SVM + rules

—

—

—

50.38

[96]

ME

—

57.89

49.61

53.43

[97]

SVM + rules, NB, DT

—

56

62

59

[98]

SVM + NB + MEh + CRF + lexicon

—

58.21

64.93

61.39

[99]

LRi

—

51.14

47.64

49.33

[78]

SVM, NB, DT, bagging

88.6

—

—

89

[60]

NB

—

—

—

54

[74]

AdaBoost

79.2

—

—

—

[67]

SVM, AdaBoost, ME

79.4

—

—

—

[75]

AdaBoost

79.2

—

—

—

[61]

NB, ME, rules

—

85.25

65

73.76

[63]

NB, ME

—

84.52

66.67

74.54

[25]

SVM

88.6

—

—

—

[76]

SVM, LR, AdaBoost

79.2

—

—

—

[26]

SVM, NB, LR

—

71.47

66.91

67.23

[107]

SVM, NB, DT

—

—

—

84

[114]

SVM, NB

—

63

82

73

[28]

NB, lexicon-based

—

75.8

74.3

73

[30]

CNN

j

76.6

73.7

76.6

73.6

[106]

SVM + NB

82.04

—

—

—

[32]

SVM, NB, RFk

—

68.73

51.42

58.83

[33]

SVM

—

78.6

78.6

78.6

[111]

LR, DT

75

76.1

—

—

[38]

NB

80

—

—

—

[41]

N-gram

—

81.93

81.13

81.24

[53]

SVM, NB, RF

—

—

—

82.4

https://medinform.jmir.org/2020/1/e16023

XSL• FO
RenderX

JMIR Med Inform 2020 | vol. 8 | iss. 1 | e16023 | p. 10(page number not for citation purposes)

JMIR MEDICAL INFORMATICSa

Zunic et alStudyAlgorithma

Accuracy (%)

Precision (%)

Recall (%)

F-measure
(%)

[47]

SVM, KNN, DT

—

58

99

73

Where multiple algorithms were compared, the performance of the best performing algorithm is indicated by italic typeset.

b

SVM: support vector machine.

c

Not applicable.

d

CRF: conditional random fields.

ek-nearest neighborsf

DT: decision treeg

NB: naïve Bayes classifier.

h

ME: maximum entropyiLR: logistic regression.

j

CNN: convolutional neural network.

k

RF: random forest.

Table 14. Overall classification performance.
Aggregated value

Accuracy (%)

Precision (%)

Recall (%)

F-measure (%)

Minimum

70.00

41.79

36.8

45.27

Maximum

88.6

85.25

99

89

Median

79.20

57.89

54.87

54.81

Mean

79.80

61.54

60.23

61.52

Standard deviation

5.39

12.63

14.55

13.15

Although a wide range of methods was used, their performance
was rarely systematically tested. According to the no free lunch
theorem [123], there is no universally best learning algorithm.
In other words, the performance of machine learning algorithms
depends not only on a specific computational task at hand, butalso on the properties of data that characterize the problem.
SVMs proved to be the most popular choice (see Table 11),which outperformed naïve Bayes classifier (NB)
[26,32,53,97,114,124] and random forest [32,51,53]. On
occasion, it was outperformed by other methods, for example,
NB [78,107], maximum entropy [67], and decision tree [47].
As it can be seen from Table 13, accuracy is not routinely
reported, which makes it difficult to generalize the findings andcompare them with SA performance in other domains.
Nonetheless, we can observe that accuracy does not fall below
70%. On average, accuracy is around 80%. This is well below
accuracy achieved in SA of movie reviews, which is typicallywell over 90% [125-128]. However, it is not straightforward to
attribute these results to the intrinsic differences between thedomains and their respective sublanguages because of thedifferent choices in methods used. The methods tested on movie
reviews are based on deep learning, whereas the methods tested
on health narratives still feature traditional machine learningwith only 2 studies using neural networks [30,45]. This may be
due to the availability of data. Movie reviews are not only
publicly available, but also come ready with annotations in theform of star rating. On the other side, health narratives may
contain sensitive information and, therefore, cannot be routinely
collected en masse. The fact that deep learning does requirelarge amount of data for training may partly explain thepreferences toward different types of methods.
https://medinform.jmir.org/2020/1/e16023

XSL• FO
RenderX

Similarly, deep learning is commonly used to support SA ofservice and product reviews. However, in these domains, theresults are closer to those in health and well-being with justover 80% for service reviews and just below 80% for productreviews [129-132]. The performance still lags behind the stateof the art achieved in these 2 domains when measured by
F-score, which was found to be below 60% on average and cango as low as 45%. F-measure achieved on service and productreviews was found to be in 70s and 80s, respectively[129,133-135]. In summary, the performance of SA of healthnarratives is much poorer than that in other domains, but it is
yet unclear if this is because of nature of the domain, the sizeof training datasets, or the choice of methods. In addition to the
choice of methods, their performance largely depends on the
choice of features used to represent text. To support basic
linguistic preprocessing, most studies used Stanford CoreNLP[136] (eg, [23,61,63,88,89,95,96,98,99,113]) and Natural
Language Toolkit [137] (eg, [51,67,91,96,107,109]). Bothlibraries represent general purpose NLP tools, which may not
be suitable for processing certain sublanguages [138]. It is worth
noticing that only 4 studies explicitly stated the use of wordembeddings [30,45,53,54].

Resources
In relation to research question RQ9, this section provides an
overview of practical resources that can be used to support
development of SA approaches in the context of health andwell-being. Table 15 provides an overview of lexica that were
utilized in studies covered by this systematic review. Apart from
OpinionKB [61], none of the remaining lexica were developed
specifically for applications to health or well-being. To
determine how much of their content is specific to health and
JMIR Med Inform 2020 | vol. 8 | iss. 1 | e16023 | p. 11(page number not for citation purposes)

JMIR MEDICAL INFORMATICS
well-being, we cross-referenced against the Unified Medical
Language System (UMLS) [139] using MetaMap Lite [140].
This analysis was limited to publicly available lexica thatprovide categorical labels of sentiment polarity. The results areshown in Figure 2. On average, 18.55% (with standard deviationof 0.0603) of each lexicon accounts for sentimentally polarized
UMLS terms. In relative terms, this accounts for a significant
portion of each lexicon given their general purpose. In absolute

Zunic et alterms, the number of these terms ranges from as little as 330 in
WordNet-Affect to as much as 11,687 in SentiWordNet.
Knowing that the UMLS currently contains over 11 million
distinct terms, we can observe that at most 1% of its content iscovered by an individual lexicon referenced in Figure 2. Thismeans that lexicon-based SA approaches will, by and large,
ignore the terminology related to health and well-being.

Table 15. Lexical resources for sentiment analysis.
Resource

Description

Used inAffective Norms for English Words A set of normative emotional ratings for a large number of[141,142]
words in terms of pleasure, arousal, and dominance.

[48,52,89]

AFINN [143,144][24,52,70]

A list of 2477 words and phrases manually rated for valencewith an integer between –5 (negative) and 5 (positive).

Harvard General Inquirer [145,146] A lexicon attaching syntactic, semantic, and pragmatic infor- [53,54]
mation to words. It includes 1915 positive and 2291 negative
words.
LabMT 1.0 [147,148]

A list 10,222 words, their average happiness evaluations according to users on Mechanical Turk.

[31,48]Multi-Perspective Question Answer- A subjectivity lexicon that provides polarity scores for approx- [27,88,95,105]
ing [149,150]
imately 8000 words.
Emotion Lexicon (also called
EmoLex) [151,152]

A list of words and their associations with 8 basic emotions [27](anger, fear, anticipation, trust, surprise, sadness, joy, anddisgust) and 2 sentiments (negative and positive). The annotations were done manually by crowdsourcing.

OpinionKB [61,153]

A knowledge base of indirect opinions about drugs represented [61]by quadruples (e, a, r, p), where e refers to the effective entity,a refers to the affected entity, r is the effect of e on a, and pis the opinion polarity.

Opinion Lexicon [5,154]

A list of around 6800 positive and negative opinion words.

[22,27,68,94,112]

SentiSense [155,156]

A lexicon attaching emotional category to 2190 WordNet
synsets, which cover a total of 5496 words.

[53,54]

SentiWordNet [157,158]

An extension of WordNet that associates each synset 3 senti- [23,41,61,63,65,71,83,94,102,113]
ment scores: positivity, negativity, and objectivity.

WordNet-Affect [159,160]

An extension of WordNet that correlates a subset of synsets
suitable to represent affective concepts with affective words.
Its hierarchical structure was modelled on the WordNet hyponymy relation.

https://medinform.jmir.org/2020/1/e16023

XSL• FO
RenderX[85,88,92,94]

JMIR Med Inform 2020 | vol. 8 | iss. 1 | e16023 | p. 12(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Zunic et al

Figure 2. The representation of the UMLS in sentiment lexica.

Extending the UMLS by including sentiment polarity would
address this gap, but this problem is nontrivial as lexicon
acquisition has been known to be a major bottleneck for SA.
Lessons can be learnt from existing research that focuses on
automatic acquisition of sentiment lexicons. These approaches
can be divided into 2 basic categories: corpus- andthesaurus-based approaches. Corpus-based approaches operate
on a hypothesis that words with the same polarity cooccur in a
discourse. Therefore, their polarity may be determined fromtheir cooccurrence with the seed words of known polarity[2,161-163]. In this context, MEDLINE [16] would be an
obvious source for assembling a large corpus. Similarly,
thesaurus-based approaches exploit the structure of a thesaurus(eg, WordNet [164]) to infer polarity of unknown words fromtheir relationships to the seed words of known polarity[165-169]. They rely on a hypothesis that synonyms (eg, trauma
and injury) have the same polarity, whereas antonyms (eg, illand healthy) have the opposite polarity. Starting with the seedwords, the network of lexical relationships is crawled topropagate the known polarity in a rule-based approach. The
structure of the UMLS could be exploited in a similar manner
to infer the sentiment of its terms.

Discussion
Principal Findings
The overarching topic of this review is the SA of spontaneously
generated narratives in relation to health and well-being.
Specifically, this systematic review was conducted with the aimof answering research questions specified in Table 1. It
identified a total of 86 relevant studies, which were used tosupport the findings, which are summarized here.
https://medinform.jmir.org/2020/1/e16023

XSL• FO
RenderX

What Are the Major Sources of Data?
The majority of data were collected from the mainstream social
multimedia and Web-based retailing platforms. Mainstreamsocial media provide a generic platform to engage patients.
However, their use of social media in the context of disclosing
protected health information may raise ethical issues. The needto engage patients online while fully complying with dataprotection regulations has led to the proliferation of websites
and networks developed specifically to provide a safe space forsharing health-related information online. This systematic review
identified 10 such platforms (see Table 6 for details). In addition
to user-generated content, the fifth i2b2/VA/Cincinnati challenge
in NLP for clinical data [81] represents an important milestone
in SA research related to health and well-being. The corpus
used for this shared task contained 1319 written notes left behindby people who died by suicide. This is one of the few datasetsthat have been made available to research community. Owing
to ethical concerns, the data used in the studies included in thissystematic review are usually not released publicly to support
further research and evaluation. This makes it difficult tobenchmark the performance of SA in health and well-being,
and test the portability of methods developed. In addition, thelack of sufficiently large datasets prevents the use ofstate-of-the-art methods such as deep learning (see Tables 12
and 13).

What Is the Originally Intended Purpose of
Spontaneously Generated Narratives?
Web 2.0 gave rise to the self-publishing and commenting onother user’s content on online platforms. On mainstream social
media such as Twitter, Facebook, Instagram, YouTube, andReddit, patients can self-organize around groups, hashtags, andJMIR Med Inform 2020 | vol. 8 | iss. 1 | e16023 | p. 13(page number not for citation purposes)

JMIR MEDICAL INFORMATICS
influencer users. The primary purpose of these conversations
is to exchange information and provide social support online.
More specialized websites such as those described in Table 6
serve the same purpose.

What Are the Roles of Their Authors Within Healthand Care?
We identified 5 roles with respect to health and well-being
among the authors of the types of spontaneously generated
narratives considered in this review: a sufferer (a person who
is affected by a medical condition), an addict (a person who is
addicted to a particular substance), a patient (a person receiving
or registered to receive medical treatment), a carer (a family
member or friend who regularly looks after a sick or disabled
person), and a suicide victim (a person who has committed
suicide). Some of these roles may overlap, for example, asufferer or an addict can also be a patient if they are receiving
a medical treatment for their medical condition.

What Are Their Demographic Characteristics?
Our focus on spontaneously generated narratives implies thatthe corresponding studies could not mandate the collection ofdemographic factors. Different Web platforms may record
different demographic factors, which may not be accessible tothird parties. Demographic information is also difficult to verify
online, which raises the concerns over the validity of suchinformation even when it is publicly available. Table 8 states
which demographic factors, if any, are recorded when a user
registers an account on the given online services and which ones
are accessible online. Only age and gender are routinely
collected, but not necessarily shared publicly. Therefore, any
findings resulting from these data should be interpreted withcaution as it may not be possible to generalize them across therelevant patient population. Out of 86 studies considered in thisreview, only 4 reported the demographic characteristics.

What Areas of Health and Well-Being Are Discussed?
Online communities tend to form around health conditions withhigh severity and chronicity rates. Not surprisingly, SA has
been used to study communities formed around cancer, mental
health problems, chronic conditions from asthma to multiple
sclerosis, pain associated with these conditions, eating disorders,and addiction (see Table 9). The provision of health care services
itself has been the subject of SA. Different treatments andservices discussed by patients whose opinions have been studied
by means of SA include medications, vaccination, surgery,orthodontic services, individual physicians, and health careservices in general.

What Are the Practical Applications of Sentiment
Analysis?
Analyzing the sentiment expressed in spontaneous narratives
offers an opportunity to improve health care services by taking
into account unforeseen factors. For example, social media canbe used to continually monitor the effects of medications to
identify previously unknown adverse reactions. Similarly, SA
can be used to differentiate between suicidal and nonsuicidal
posts, after which a real-time online counseling can be offered.
Patient reviews of specific medications can support theirhttps://medinform.jmir.org/2020/1/e16023

XSL• FO
RenderX

Zunic et aldecision making but can also be explored to support shareddecision making, ultimately influencing health outcomes andhealth care utilization. Patient reviews of health care services
can help identify opportunities for service improvement, thus
influencing health outcomes and health care utilization. In termsof disease prevention, patients’ opinions can help healthpractitioners understand potential obstacles to population-based
intervention approaches such as vaccination. Understanding
patients’ experience with different treatments can support
creation of personalized therapy plans.

What Methods Have Been Used to Perform Sentiment
Analysis?
A wide range of methods have been used to perform SA. Most
common choices include SVMs, naïve Bayesian learning,decision trees, logistic regression, and adaptive boosting. Other
approaches include maximum entropy, conditional random
fields, random forests, and k-nearest neighbors. The findings
show strong bias toward traditional machine learning. A single
study used deep learning. This is in stark contrast with generaltrends in SA research.

What Is the State-of-the-Art Performance of Sentiment
Analysis?
On average, accuracy is around 80%, and it does not fall below
70%. This is well below accuracy achieved in SA of movie
reviews, which is typically well over 90%. In SA of service andproduct reviews, the results are closer to those in health andwell-being with just more than 80% for service reviews and justbelow 80% for product reviews. However, the performance stilllags behind the state of the art achieved in these 2 domains whenmeasured by F-score, which was found to be below 60% on
average. F-measure achieved on service and product reviews isfound to be above 70% and 80%, respectively. In summary, theperformance of SA of health narratives is much poorer than thatin other domains.

What Resources Are Available to Support Sentiment
Analysis Related to Health and Well-Being?
A wide range of lexica were utilized in studies covered by thissystematic review (see Table 15. Notably, out of 11 lexica, only
1 was developed specifically for a domain related to health orwell-being. The lack of domain-specific lexicons may partly
explain the poorer performance recorded in this domain.

Conclusions
In summary, this review has uncovered multiple opportunities
to advance research in SA related to health and well-being.
Keeping in mind the no free lunch theorem, researchers in thisarea need to put more effort in systematically exploring a wide
range of methods and testing their performance. Community
efforts to create and share a large, anonymized dataset would
enable not only rigorous benchmarking of existing methods but
also exploration of new approaches including deep learning.
This should help the field catch up with the most recent
developments in SA. The creation of domain-specific sentimentlexica stands to further improve the performance of SA related
to health and well-being. Although many studies have dealt
with automatic construction of domain-specific sentiment lexica
JMIR Med Inform 2020 | vol. 8 | iss. 1 | e16023 | p. 14(page number not for citation purposes)

JMIR MEDICAL INFORMATICS
using methods such as random walks, no such studies have been
identified in this systematic review. Finally, health-related
applications of SA require systematic collection of demographic

Zunic et aldata to illustrate the extent to which the findings can begeneralized.

Acknowledgments
This work is part of a PhD project funded by Cardiff University via Vice-Chancellor’s International Scholarships for Research
Excellence. The scholarship has been awarded to AŽ, and her project is supervised by IS and PC.

Authors' Contributions
IS designed the study. AŽ conducted the search and data extraction. All authors were responsible for critical evaluation, analysis,
and presentation of the results. AŽ and IS drafted the manuscript. PC critically evaluated the article. All authors approved thefinal version before submission.

Conflicts of Interest
None declared.

References
1.
2.

3.

4.

5.

6.
7.

8.

9.
10.
11.
12.
13.
14.
15.
16.
17.
18.

Wiebe J, Bruce R. Probabilistic classifiers for tracking point of view. Progress in communication sciences 1995:125-142[FREE Full text]
Hatzivassiloglou V, McKeown KR. Predicting the Semantic Orientation of Adjectives. In: Proceedings of the 35th Annual
Meeting of the Association for Computational Linguistics and Eighth Conference of the European Chapter of the Association
for Computational Linguistics. 1997 Presented at: ACL'98/EACL'98; July 7-12, 1997; Madrid, Spain p. 174-181 URL:
https://www.aclweb.org/anthology/P97-1023/ [doi: 10.3115/979617.979640]
Wiebe JM, Bruce RF, O'Hara TP. Development and Use of a Gold-standard Data Set for Subjectivity Classifications. In:
Proceedings of the 37th annual meeting of the Association for Computational Linguistics on Computational Linguistics.
1999 Presented at: ACL'99; June 20-26, 1999; College Park, Maryland, USA p. 246-253 URL: https://www.aclweb.org/
anthology/P99-1032/ [doi: 10.3115/1034678.1034721]
Hu M, Liu B. Mining Opinion Features in Customer Reviews. In: Proceedings of the 19th national conference on Artificalintelligence. 2004 Presented at: AAAI'04; July 25 - 29, 2004; San Jose, California, USA p. 755-760 URL: https://dl.acm.org/
citation.cfm?id=1597269
Hu M, Liu B. Mining and Summarizing Customer Reviews. In: Proceedings of the tenth ACM SIGKDD international
conference on Knowledge discovery and data mining. 2004 Presented at: KDD'04; August 22 - 25, 2004; Seattle, Washington,
USA p. 168-177 URL: https://dl.acm.org/citation.cfm?id=1014073 [doi: 10.1145/1014052.1014073]
Bollen J, Mao H, Zeng X. Twitter mood predicts the stock market. J Comput Sci 2011;2(1):1-8. [doi:10.1016/j.jocs.2010.12.007]
Efron M. Cultural orientation: Classifying subjective documents by cociation analysis. In: Proceedings of the AAAI Fall
Symposium on Style and Meaning in Language, Art, and Music. 2004 Presented at: AAAI'04; July 25-29, 2004; San Jose,
California p. 41-48.
Ramteke J, Shah S, Godhia D, Shaikh A. Election Result Prediction Using Twitter Sentiment Analysis. In: Proceedings ofthe 2016 International Conference on Inventive Computation Technologies. 2016 Presented at: ICICT'16; August 26-27,
2016; Coimbatore, India p. 1-5. [doi: 10.1109/inventive.2016.7823280]
World Health Organisation. Geneva, Switzerland: World Health Organisation; 2006. Constitution of the World HealthOrganisation URL: https://www.who.int/governance/eb/who_constitution_en.pdf [accessed 2019-11-12]
Huber M, Knottnerus JA, Green L, van der Horst H, Jadad AR, Kromhout D, et al. How should we define health? Br Med
J 2011 Jul 26;343:d4163. [doi: 10.1136/bmj.d4163] [Medline: 21791490]
Berg O. Health and quality of life. Acta Sociologica 1975;18(1):3-22. [doi: 10.1177/000169937501800102]
Denecke K, Deng Y. Sentiment analysis in medical settings: new opportunities and challenges. Artif Intell Med 2015
May;64(1):17-27. [doi: 10.1016/j.artmed.2015.03.006] [Medline: 25982909]
Gohil S, Vuik S, Darzi A. Sentiment analysis of health care tweets: review of the methods used. JMIR Public Health Surveill
2018 Apr 23;4(2):e43 [FREE Full text] [doi: 10.2196/publichealth.5789] [Medline: 29685871]
Kitchenham B. Procedures for performing systematic reviews. Keele University, Keele 2004;33(2004):1-26 [FREE Fulltext]
Cochrane Library: Cochrane Reviews. URL: https://www.cochranelibrary.com/ [accessed 2019-11-12]
National Library of Medicine. MEDLINE: Description of the Database URL: https://www.nlm.nih.gov/bsd/medline.html[accessed 2019-11-12]
Embase. URL: https://www.embase.com [accessed 2019-11-12]
EBSCO Health. CINAHL Database URL: https://health.ebsco.com/products/the-cinahl-database [accessed 2019-11-12]

https://medinform.jmir.org/2020/1/e16023

XSL• FO
RenderX

JMIR Med Inform 2020 | vol. 8 | iss. 1 | e16023 | p. 15(page number not for citation purposes)

JMIR MEDICAL INFORMATICS
19.
20.
21.

22.
23.
24.

25.

26.

27.

28.

29.

30.

31.
32.

33.

34.
35.

36.

37.

38.

39.
40.

Cohen J. A Coefficient of Agreement for Nominal Scales. Educ Psychol Meas 1960;20(1):37-46. [doi:10.1177/001316446002000104]
Yoon S, Bakken S. Methods of knowledge discovery in tweets. NI 2012 (2012) 2012;2012:463 [FREE Full text] [Medline:
24199142]
Mishra MV, Bennett M, Vincent A, Lee OT, Lallas CD, Trabulsi EJ, et al. Identifying barriers to patient acceptance ofactive surveillance: content analysis of online patient communications. PLoS One 2013;8(9):e68563 [FREE Full text] [doi:10.1371/journal.pone.0068563] [Medline: 24039699]
Ramagopalan S, Wasiak R, Cox AP. Using Twitter to investigate opinions about multiple sclerosis treatments: a descriptive,
exploratory study. F1000Res 2014;3:216 [FREE Full text] [doi: 10.12688/f1000research.5263.1] [Medline: 25520780]
Wiley MT, Jin C, Hristidis V, Esterling KM. Pharmaceutical drugs chatter on Online Social Networks. J Biomed Inform
2014 Jun;49:245-254 [FREE Full text] [doi: 10.1016/j.jbi.2014.03.006] [Medline: 24637141]
Tighe PJ, Goldsmith RC, Gravenstein M, Bernard HR, Fillingim RB. The painful tweet: text, sentiment, and community
structure analyses of tweets pertaining to pain. J Med Internet Res 2015 Apr 2;17(4):e84 [FREE Full text] [doi:10.2196/jmir.3769] [Medline: 25843553]
Zhou X, Coiera E, Tsafnat G, Arachi D, Ong M, Dunn AG. Using social connection information to improve opinion mining:
identifying negative sentiment about HPV vaccines on Twitter. Stud Health Technol Inform 2015;216:761-765. [doi:10.3233/978-1-61499-564-7-761] [Medline: 26262154]
Daniulaityte R, Chen L, Lamy FR, Carlson RG, Thirunarayan K, Sheth A. 'When 'Bad' is 'Good'': identifying personalcommunication and sentiment in drug-related tweets. JMIR Public Health Surveill 2016 Oct 24;2(2):e162 [FREE Full text][doi: 10.2196/publichealth.6327] [Medline: 27777215]
Korkontzelos I, Nikfarjam A, Shardlow M, Sarker A, Ananiadou S, Gonzalez GH. Analysis of the effect of sentiment
analysis on extracting adverse drug reactions from tweets and forum posts. J Biomed Inform 2016 Aug;62:148-158 [FREEFull text] [doi: 10.1016/j.jbi.2016.06.007] [Medline: 27363901]
Ofoghi B, Mann M, Verspoor K. Towards early discovery of salient health threats: a social media emotion classification
technique. Pac Symp Biocomput 2016;21:504-515 [FREE Full text] [doi: 10.1142/9789814749411_0046] [Medline:
26776213]
Palomino M, Taylor T, Göker A, Isaacs J, Warber S. The online dissemination of nature-health concepts: lessons fromsentiment analysis of social media relating to 'Nature-deficit disorder'. Int J Environ Res Public Health 2016 Jan 19;13(1):pii:
E142 [FREE Full text] [doi: 10.3390/ijerph13010142] [Medline: 26797628]
Bian J, Zhao Y, Salloum RG, Guo Y, Wang M, Prosperi M, et al. Using social media data to understand the impact ofpromotional information on laypeople's discussions: a case study of lynch syndrome. J Med Internet Res 2017 Dec
13;19(12):e414 [FREE Full text] [doi: 10.2196/jmir.9266] [Medline: 29237586]
Davis MA, Zheng K, Liu Y, Levy H. Public response to Obamacare on Twitter. J Med Internet Res 2017 May 26;19(5):e167[FREE Full text] [doi: 10.2196/jmir.6946] [Medline: 28550002]
Du J, Xu J, Song H, Liu X, Tao C. Optimization on machine learning based approaches for sentiment analysis on HPV
vaccines related tweets. J Biomed Semantics 2017 Mar 3;8(1):9 [FREE Full text] [doi: 10.1186/s13326-017-0120-6][Medline: 28253919]
Du J, Xu J, Song H, Tao C. Leveraging machine learning-based approaches to assess human papillomavirus vaccination
sentiment trends with Twitter data. BMC Med Inform Decis Mak 2017 Jul 5;17(Suppl 2):69 [FREE Full text] [doi:10.1186/s12911-017-0469-6] [Medline: 28699569]
Gruebner O, Lowe SR, Sykora M, Shankardass K, Subramanian SV, Galea S. A novel surveillance approach for disastermental health. PLoS One 2017;12(7):e0181233 [FREE Full text] [doi: 10.1371/journal.pone.0181233] [Medline: 28723959]
Haghighi PD, Kang Y, Buchbinder R, Burstein F, Whittle S. Investigating subjective experience and the influence of
weather among individuals with fibromyalgia: a content analysis of Twitter. JMIR Public Health Surveill 2017 Jan 19;3(1):e4[FREE Full text] [doi: 10.2196/publichealth.6344] [Medline: 28104577]
Kang Y, Wang Y, Zhang D, Zhou L. The public's opinions on a new school meals policy for childhood obesity prevention
in the US: a social media analytics approach. Int J Med Inform 2017 Jul;103:83-88. [doi: 10.1016/j.ijmedinf.2017.04.013][Medline: 28551006]
Lim S, Tucker CS, Kumara S. An unsupervised machine learning model for discovering latent infectious diseases usingsocial media data. J Biomed Inform 2017 Feb;66:82-94 [FREE Full text] [doi: 10.1016/j.jbi.2016.12.007] [Medline:
28034788]
Metwally O, Blumberg S, Ladabaum U, Sinha SR. Using social media to characterize public sentiment toward medical
interventions commonly used for cancer screening: an observational study. J Med Internet Res 2017 Jun 7;19(6):e200[FREE Full text] [doi: 10.2196/jmir.7485] [Medline: 28592395]
Noll D, Mahon B, Shroff B, Carrico C, Lindauer SJ. Twitter analysis of the orthodontic patient experience with braces vsInvisalign. Angle Orthod 2017 May;87(3):377-383. [doi: 10.2319/062816-508.1] [Medline: 28059576]
Oscar N, Fox PA, Croucher R, Wernick R, Keune J, Hooker K. Machine learning, sentiment analysis, and tweets: an
examination of Alzheimer's disease stigma on Twitter. J Gerontol B Psychol Sci Soc Sci 2017 Sep 1;72(5):742-751. [doi:10.1093/geronb/gbx014] [Medline: 28329835]

https://medinform.jmir.org/2020/1/e16023

XSL• FO
RenderX

Zunic et al

JMIR Med Inform 2020 | vol. 8 | iss. 1 | e16023 | p. 16(page number not for citation purposes)

JMIR MEDICAL INFORMATICS
41.

42.

43.
44.
45.
46.

47.
48.

49.

50.

51.
52.

53.
54.

55.
56.

57.

58.
59.
60.
61.
62.
63.
64.
65.
66.
67.

Salas-Zárate MD, Medina-Moreira J, Lagos-Ortiz K, Luna-Aveiga H, Rodríguez-García MA, Valencia-García R. Sentiment
analysis on tweets about diabetes: an aspect-level approach. Comput Math Methods Med 2017;2017:5140631 [FREE Fulltext] [doi: 10.1155/2017/5140631] [Medline: 28316638]
Cao X, MacNaughton P, Deng Z, Yin J, Zhang X, Allen JG. Using Twitter to better understand the spatiotemporal patternsof public sentiment: a case study in Massachusetts, USA. Int J Environ Res Public Health 2018 Feb 2;15(2):pii: E250
[FREE Full text] [doi: 10.3390/ijerph15020250] [Medline: 29393869]
Gabarron E, Dorronzoro E, Rivera-Romero O, Wynn R. Diabetes on Twitter: a sentiment analysis. J Diabetes Sci Technol
2019 May;13(3):439-444. [doi: 10.1177/1932296818811679] [Medline: 30453762]
Pai RR, Alathur S. Assessing mobile health applications with Twitter analytics. Int J Med Inform 2018 May;113:72-84.
[doi: 10.1016/j.ijmedinf.2018.02.016] [Medline: 29602436]
Zhang L, Hall M, Bastola D. Utilizing Twitter data for analysis of chemotherapy. Int J Med Inform 2018 Dec;120:92-100.
[doi: 10.1016/j.ijmedinf.2018.10.002] [Medline: 30409350]
Roccetti M, Marfia G, Salomoni P, Prandi C, Zagari RM, Kengni FL, et al. Attitudes of Crohn's disease patients:
infodemiology case study and sentiment analysis of Facebook and Twitter posts. JMIR Public Health Surveill 2017 Aug
9;3(3):e51 [FREE Full text] [doi: 10.2196/publichealth.7004] [Medline: 28793981]
Islam MR, Kabir MA, Ahmed A, Kamal AR, Wang H, Ulhaq A. Depression detection from social network data using
machine learning techniques. Health Inf Sci Syst 2018 Dec;6(1):8. [doi: 10.1007/s13755-018-0046-0] [Medline: 30186594]
Ricard BJ, Marsch LA, Crosier B, Hassanpour S. Exploring the utility of community-generated social media content for
detecting depression: an analytical study on Instagram. J Med Internet Res 2018 Dec 6;20(12):e11817 [FREE Full text][doi: 10.2196/11817] [Medline: 30522991]
Oksanen A, Garcia D, Sirola A, Näsi M, Kaakinen M, Keipi T, et al. Pro-anorexia and anti-pro-anorexia videos on YouTube:sentiment analysis of user responses. J Med Internet Res 2015 Nov 12;17(11):e256 [FREE Full text] [doi: 10.2196/jmir.5007][Medline: 26563678]
Livas C, Delli K, Pandis N. 'My Invisalign experience': content, metrics and comment sentiment analysis of the most popularpatient testimonials on YouTube. Prog Orthod 2018 Jan 22;19(1):3 [FREE Full text] [doi: 10.1186/s40510-017-0201-1][Medline: 29354889]
Aladağ AE, Muderrisoglu S, Akbas NB, Zahmacioglu O, Bingol HO. Detecting suicidal ideation on forums: proof-of-concept
study. J Med Internet Res 2018 Jun 21;20(6):e215 [FREE Full text] [doi: 10.2196/jmir.9840] [Medline: 29929945]
Adams DZ, Gruss R, Abrahams AS. Automated discovery of safety and efficacy concerns for joint & muscle pain relief
treatments from online reviews. Int J Med Inform 2017 Apr;100:108-120. [doi: 10.1016/j.ijmedinf.2017.01.005] [Medline:
28241932]
Carrillo-de-Albornoz J, Vidal JR, Plaza L. Feature engineering for sentiment analysis in e-health forums. PLoS One
2018;13(11):e0207996 [FREE Full text] [doi: 10.1371/journal.pone.0207996] [Medline: 30496232]
Carrillo-de-Albornoz J, Aker A, Kurtic E, Plaza L. Beyond opinion classification: extracting facts, opinions and experiencesfrom health forums. PLoS One 2019;14(1):e0209961 [FREE Full text] [doi: 10.1371/journal.pone.0209961] [Medline:
30625206]
RateMDs. URL: https://www.ratemds.com/ [accessed 2019-11-12]
Alemi F, Torii M, Clementz L, Aron DC. Feasibility of real-time satisfaction surveys through automated analysis of patients'
unstructured comments and sentiments. Qual Manag Health Care 2012;21(1):9-19. [doi: 10.1097/QMH.0b013e3182417fc4][Medline: 22207014]
Wallace BC, Paul MJ, Sarkar U, Trikalinos TA, Dredze M. A large-scale quantitative analysis of latent factors and sentiment
in online doctor reviews. J Am Med Inform Assoc 2014;21(6):1098-1103 [FREE Full text] [doi:10.1136/amiajnl-2014-002711] [Medline: 24918109]
Hopper AM, Uriyo M. Using sentiment analysis to review patient satisfaction data located on the internet. J Health Organ
Manag 2015;29(2):221-233. [doi: 10.1108/JHOM-12-2011-0129] [Medline: 25800334]
WebMD - Better information. Better health. URL: https://www.webmd.com/ [accessed 2019-11-12]
Huh J, Yetisgen-Yildiz M, Pratt W. Text classification for assisting moderators in online health communities. J Biomed
Inform 2013 Dec;46(6):998-1005 [FREE Full text] [doi: 10.1016/j.jbi.2013.08.011] [Medline: 24025513]
Noferesti S, Shamsfard M. Resource construction and evaluation for indirect opinion mining of drug reviews. PLoS One
2015;10(5):e0124993 [FREE Full text] [doi: 10.1371/journal.pone.0124993] [Medline: 25962135]
Ask a Patient. URL: https://www.askapatient.com/ [accessed 2019-11-12]
Noferesti S, Shamsfard M. Using Linked Data for polarity classification of patients' experiences. J Biomed Inform 2015
Oct;57:6-19 [FREE Full text] [doi: 10.1016/j.jbi.2015.06.017] [Medline: 26210363]
DrugLib. URL: http://www.druglib.com/ [accessed 2019-11-12]
Asghar MZ, Ahmad S, Qasim M, Zahra SR, Kundi FM. SentiHealth: creating health-related sentiment lexicon using hybrid
approach. Springerplus 2016;5(1):1139 [FREE Full text] [doi: 10.1186/s40064-016-2809-x] [Medline: 27504237]
Breast Cancer Information and Support. URL: https://www.breastcancer.org/ [accessed 2019-11-12]
Zhang S, Bantum E, Owen J, Elhadad N. Does sustained participation in an online health community affect sentiment?
AMIA Annu Symp Proc 2014;2014:1970-1979 [FREE Full text] [Medline: 25954470]

https://medinform.jmir.org/2020/1/e16023

XSL• FO
RenderX

Zunic et al

JMIR Med Inform 2020 | vol. 8 | iss. 1 | e16023 | p. 17(page number not for citation purposes)

JMIR MEDICAL INFORMATICS
68.

69.
70.
71.

72.
73.
74.

75.

76.
77.
78.

79.
80.

81.
82.
83.
84.
85.

86.

87.

88.
89.
90.
91.
92.
93.
94.

Cabling ML, Turner JW, Hurtado-de-Mendoza A, Zhang Y, Jiang X, Drago F, et al. Sentiment analysis of an online breast
cancer support group: communicating about tamoxifen. Health Commun 2018 Sep;33(9):1158-1165 [FREE Full text] [doi:10.1080/10410236.2017.1339370] [Medline: 28678549]
MedHelp. URL: https://medhelp.org/ [accessed 2019-11-12]
Yang FC, Lee AJ, Kuo SC. Mining health social media with sentiment analysis. J Med Syst 2016 Nov;40(11):236. [doi:10.1007/s10916-016-0604-4] [Medline: 27663246]
Lu Y, Wu Y, Liu J, Li J, Zhang P. Understanding health care social media use from different stakeholder perspectives: acontent analysis of an online health community. J Med Internet Res 2017 Apr 7;19(4):e109 [FREE Full text] [doi:10.2196/jmir.7087] [Medline: 28389418]
DailyStrength. URL: https://www.dailystrength.org/
Cancer Survivor Network. URL: http://csn.cancer.org/ [accessed 2019-11-12]
Portier K, Greer GE, Rokach L, Ofek N, Wang Y, Biyani P, et al. Understanding topics and sentiment in an online cancer
survivor community. J Natl Cancer Inst Monogr 2013 Dec;2013(47):195-198. [doi: 10.1093/jncimonographs/lgt025][Medline: 24395991]
Zhao K, Yen J, Greer G, Qiu B, Mitra P, Portier K. Finding influential users of online health communities: a new metricbased on sentiment influence. J Am Med Inform Assoc 2014 Oct;21(e2):e212-e218 [FREE Full text] [doi:10.1136/amiajnl-2013-002282] [Medline: 24449805]
Bui N, Yen J, Honavar V. Temporal causality analysis of sentiment change in a cancer survivor network. IEEE Trans
Comput Soc Syst 2016 Jun;3(2):75-87 [FREE Full text] [doi: 10.1109/TCSS.2016.2591880] [Medline: 29399599]
National Health Service. URL: https://www.nhs.uk/ [accessed 2019-11-12]
Greaves F, Ramirez-Cano D, Millett C, Darzi A, Donaldson L. Use of sentiment analysis for capturing patient experiencefrom free-text comments posted online. J Med Internet Res 2013 Nov 1;15(11):e239 [FREE Full text] [doi: 10.2196/jmir.2721][Medline: 24184993]
DiabetesDaily. URL: https://www.diabetesdaily.com [accessed 2019-11-12]
Akay A, Dragomir A, Erlandsson B. A novel data-mining approach leveraging social media to monitor consumer opinionof sitagliptin. IEEE J Biomed Health Inform 2015 Jan;19(1):389-396. [doi: 10.1109/JBHI.2013.2295834] [Medline:
25561458]
Pestian JP, Matykiewicz P, Linn-Gust M, South B, Uzuner O, Wiebe J, et al. Sentiment analysis of suicide notes: a sharedtask. Biomed Inform Insights 2012 Jan 30;5(Suppl 1):3-16 [FREE Full text] [doi: 10.4137/bii.s9042] [Medline: 22419877]
Cherry C, Mohammad SM, de Bruijn B. Binary classifiers and latent sequence models for emotion detection in suicide
notes. Biomed Inform Insights 2012;5(Suppl. 1):147-154 [FREE Full text] [doi: 10.4137/BII.S8933] [Medline: 22879771]
Desmet B, Hoste V. Combining lexico-semantic features for emotion classification in suicide notes. Biomed Inform Insights
2012;5(Suppl 1):125-128 [FREE Full text] [doi: 10.4137/BII.S8960] [Medline: 22879768]
Dzogang F, Lesot M, Rifqi M, Bouchon-Meunier B. Early fusion of low level features for emotion mining. Biomed Inform
Insights 2012;5(Suppl 1):129-136 [FREE Full text] [doi: 10.4137/BII.S8973] [Medline: 22879769]
Liakata M, Kim J, Saha S, Hastings J, Rebholz-Schuhmann D. Three hybrid classifiers for the detection of emotions insuicide notes. Biomed Inform Insights 2012;5(Suppl 1):175-184 [FREE Full text] [doi: 10.4137/BII.S8967] [Medline:
22879774]
Luyckx K, Vaassen F, Peersman C, Daelemans W. Fine-grained emotion detection in suicide notes: a thresholding approach
to multi-label classification. Biomed Inform Insights 2012;5(Suppl 1):61-69 [FREE Full text] [doi: 10.4137/BII.S8966][Medline: 22879761]
McCart JA, Finch DK, Jarman J, Hickling E, Lind JD, Richardson MR, et al. Using ensemble models to classify thesentiment expressed in suicide notes. Biomed Inform Insights 2012;5(Suppl 1):77-85 [FREE Full text] [doi:10.4137/BII.S8931] [Medline: 22879763]
Nikfarjam A, Emadzadeh E, Gonzalez G. A hybrid system for emotion extraction from suicide notes. Biomed Inform
Insights 2012;5(Suppl 1):165-174 [FREE Full text] [doi: 10.4137/BII.S8981] [Medline: 22879773]
Pak A, Bernhard D, Paroubek P, Grouin C. A combined approach to emotion detection in suicide notes. Biomed Inform
Insights 2012;5(Suppl 1):105-114 [FREE Full text] [doi: 10.4137/BII.S8969] [Medline: 22879766]
Pedersen T. Rule-based and lightly supervised methods to predict emotions in suicide notes. Biomed Inform Insights
2012;5(Suppl 1):185-193 [FREE Full text] [doi: 10.4137/BII.S8953] [Medline: 22879775]
Read J, Velldal E, Ovrelid L. Labeling emotions in suicide notes: cost-sensitive learning with heterogeneous features.
Biomed Inform Insights 2012;5(Suppl 1):99-103 [FREE Full text] [doi: 10.4137/BII.S8930] [Medline: 22879765]
Roberts K, Harabagiu SM. Statistical and similarity methods for classifying emotion in suicide notes. Biomed Inform
Insights 2012;5(Suppl 1):195-204 [FREE Full text] [doi: 10.4137/BII.S8958] [Medline: 22879776]
Sohn S, Torii M, Li D, Wagholikar K, Wu S, Liu H. A hybrid approach to sentiment sentence classification in suicide notes.
Biomed Inform Insights 2012;5(Suppl 1):43-50 [FREE Full text] [doi: 10.4137/BII.S8961] [Medline: 22879759]
Spasić I, Burnap P, Greenwood M, Arribas-Ayllon M. A naïve Bayes approach to classifying topics in suicide notes. Biomed
Inform Insights 2012;5(Suppl 1):87-97 [FREE Full text] [doi: 10.4137/BII.S8945] [Medline: 22879764]

https://medinform.jmir.org/2020/1/e16023

XSL• FO
RenderX

Zunic et al

JMIR Med Inform 2020 | vol. 8 | iss. 1 | e16023 | p. 18(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Zunic et al

95.

Wang W, Chen L, Tan M, Wang S, Sheth AP. Discovering fine-grained sentiment in suicide notes. Biomed Inform Insights
2012;5(Suppl 1):137-145 [FREE Full text] [doi: 10.4137/BII.S8963] [Medline: 22879770]
96. Wicentowski R, Sydes MR. Emotion detection in suicide notes using maximum entropy classification. Biomed Inform
Insights 2012;5(Suppl 1):51-60 [FREE Full text] [doi: 10.4137/BII.S8972] [Medline: 22879760]
97. Xu Y, Wang Y, Liu J, Tu Z, Sun J, Tsujii J, et al. Suicide note sentiment classification: a supervised approach augmented
by web data. Biomed Inform Insights 2012;5(Suppl 1):31-41 [FREE Full text] [doi: 10.4137/BII.S8956] [Medline: 22879758]
98. Yang H, Willis A, de Roeck A, Nuseibeh B. A hybrid model for automatic emotion recognition in suicide notes. Biomed
Inform Insights 2012;5(Suppl 1):17-30 [FREE Full text] [doi: 10.4137/BII.S8948] [Medline: 22879757]
99. Yeh E, Jarrold W, Jordan J. Leveraging psycholinguistic resources and emotional sequence models for suicide note emotion
annotation. Biomed Inform Insights 2012;5(Suppl 1):155-163 [FREE Full text] [doi: 10.4137/BII.S8979] [Medline:
22879772]
100. Yu N, Kübler S, Herring J, Hsu Y, Israel R, Smiley C. LASSA: emotion detection via information fusion. Biomed Inform
Insights 2012;5(Suppl. 1):71-76 [FREE Full text] [doi: 10.4137/BII.S8949] [Medline: 22879762]
101. Mammen JR, Java JJ, Rhee H, Butz AM, Halterman JS, Arcoleo K. Mixed-methods content and sentiment analysis ofadolescents' voice diaries describing daily experiences with asthma and self-management decision-making. Clin Exp Allergy
2019 Mar;49(3):299-307. [doi: 10.1111/cea.13250] [Medline: 30113733]
102. Asghar MZ, Khan A, Ahmad S, Qasim M, Khan IA. Lexicon-enhanced sentiment analysis framework using rule-basedclassification scheme. PLoS One 2017;12(2):e0171649 [FREE Full text] [doi: 10.1371/journal.pone.0171649] [Medline:
28231286]
103. Cobb NK, Mays D, Graham AL. Sentiment analysis to determine the impact of online messages on smokers' choices touse varenicline. J Natl Cancer Inst Monogr 2013 Dec;2013(47):224-230. [doi: 10.1093/jncimonographs/lgt020] [Medline:
24395996]
104. Cohn AM, Amato MS, Zhao K, Wang X, Cha S, Pearson JL, et al. Discussions of alcohol use in an online social network
for smoking cessation: analysis of topics, sentiment, and social network centrality. Alcohol Clin Exp Res 2019
Jan;43(1):108-114. [doi: 10.1111/acer.13906] [Medline: 30326140]
105. Chu KH, Valente TW. How different countries addressed the sudden growth of e-cigarettes in an online tobacco control
community. BMJ Open 2015 May 21;5(5):e007654 [FREE Full text] [doi: 10.1136/bmjopen-2015-007654] [Medline:
25998038]
106. Chen Z, Zeng DD. Mining online e-liquid reviews for opinion polarities about e-liquid features. BMC Public Health 2017
Jul 7;17(1):633 [FREE Full text] [doi: 10.1186/s12889-017-4533-z] [Medline: 28683797]
107. Doing-Harris K, Mowery D, Daniels C, Chapman W, Conway M. Understanding patient satisfaction with received healthcare
services: a natural language processing approach. AMIA Annu Symp Proc 2016;2016:524-533 [FREE Full text] [Medline:
28269848]
108. Alemi F, Jasper H. An alternative to satisfaction surveys: let the patients talk. Qual Manag Health Care 2014;23(1):10-19.
[doi: 10.1097/QMH.0000000000000014] [Medline: 24368718]
109. Diorio C, Afanasiev M, Salena K, Marjerrison S. 'A world of competing sorrows': a mixed methods analysis of media
reports of children with cancer abandoning conventional treatment. PLoS One 2018;13(12):e0209738 [FREE Full text][doi: 10.1371/journal.pone.0209738] [Medline: 30576389]
110. Corley C, Mihalcea R, Mikler A, Sanfilippo A. Predicting individual affect of health interventions to reduce HPV prevalence.
In: Arabnia H, Tran QN, editors. Software Tools and Algorithms for Biological Systems. New York, New York, USA:
Springer; 2011:181-190.
111. Jung H, Park H, Song T. Ontology-based approach to social data sentiment analysis: detection of adolescent depression
signals. J Med Internet Res 2017 Jul 24;19(7):e259 [FREE Full text] [doi: 10.2196/jmir.7452] [Medline: 28739560]
112. Chen L, Gong T, Kosinski M, Stillwell D, Davidson RL. Building a profile of subjective well-being for social media users.
PLoS One 2017;12(11):e0187278 [FREE Full text] [doi: 10.1371/journal.pone.0187278] [Medline: 29135991]
113. Rastegar-Mojarad M, Ye Z, Wall D, Murali N, Lin S. Collecting and analyzing patient experiences of health care fromsocial media. JMIR Res Protoc 2015 Jul 2;4(3):e78 [FREE Full text] [doi: 10.2196/resprot.3433] [Medline: 26137885]
114. Liu R, Zhang X, Zhang H. Web-video-mining-supported workflow modeling for laparoscopic surgeries. Artif Intell Med
2016 Nov;74:9-20. [doi: 10.1016/j.artmed.2016.11.002] [Medline: 27964803]
115. SVM light. URL: http://svmlight.joachims.org/ [accessed 2019-11-12]
116. Cauchois B. pysvmlight. URL: https://bitbucket.org/wcauchois/pysvmlight [accessed 2019-11-12]
117. LIBSVM -- A Library for Support Vector Machines. URL: https://www.csie.ntu.edu.tw/~cjlin/libsvm/ [accessed 2019-11-12]
118. Dashtipour K, Poria S, Hussain A, Cambria E, Hawalah AY, Gelbukh A, et al. Multilingual sentiment analysis: state of theart and independent comparison of techniques. Cognit Comput 2016;8:757-771 [FREE Full text] [doi:10.1007/s12559-016-9415-7] [Medline: 27563360]
119. Department of Computer Science: University of Waikato. URL: https://www.cs.waikato.ac.nz/ml/weka/ [accessed 2019-11-12]
120. scikit-learn. URL: https://scikit-learn.org/ [accessed 2019-11-12]
121. Keras Documentation. URL: https://keras.io/ [accessed 2019-11-12]
122. TextBlob: Simplified Text Processing. URL: https://textblob.readthedocs.io/en/dev/ [accessed 2019-11-12]
https://medinform.jmir.org/2020/1/e16023

XSL• FO
RenderX

JMIR Med Inform 2020 | vol. 8 | iss. 1 | e16023 | p. 19(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Zunic et al123. Wolpert DH. The lack of a priori distinctions between learning algorithms. Neural Comput 1996;8(7):1341-1390 [FREEFull text] [doi: 10.1162/neco.1996.8.7.1341]
124. Hutto C, Gilbert E. Vader: A parsimonious rule-based model for sentiment analysis of social media text. In: Proceedingsof Eighth International AAAI Conference on Weblogs and Social Media. 2014 Presented at: ICWSM-14; June 1–4, 2014;
Ann Arbor, Michigan, USA.
125. Yang Z, Dai Z, Yang Y, Carbonell J, Salakhutdinov R, Le QV. XLNet: Generalized autoregressive pretraining for language
understanding. arXiv 2019 [FREE Full text]
126. Howard J, Ruder S. Universal Language Model Fine-tuning for Text Classification. In: Proceedings of the 56th Annual
Meeting of the Association for Computational Linguistics. 2018 Presented at: ACL'18; July 15-20, 2018; Melbourne,
Australia p. 328-339. [doi: 10.18653/v1/p18-1031]
127. Gray S, Radford A, Kingma DP. GPU kernels for block-sparse weights. arXiv 2017 [FREE Full text]
128. Johnson R, Zhang T. Supervised and Semi-supervised Text Categorization Using LSTM for Region Embeddings. In:
Proceedings of the 33rd International Conference on International Conference on Machine Learning - Volume 48. 2016
Presented at: ICML'16; June 19 - 24, 2016; New York, New York, USA p. 526-534 URL: https://arxiv.org/abs/1602.02373
129. Xu H, Liu B, Shu L, Yu PS. BERT Post-training for review reading comprehension and aspect-based sentiment analysis.
In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics:
Human Language Technologies, Volume 1. 2019 Presented at: NAACL'19; June 3-5, 2019; Minneapolis, USA p. 2324-2335
URL: https://arxiv.org/abs/1904.02232
130. Huang B, Ou Y, Carley K. Aspect Level Sentiment Classification With Attention-over-attention Neural Networks. In:
International Conference on Social Computing, Behavioral-Cultural Modeling and Prediction and Behavior Representationin Modeling and Simulation. 2018 Presented at: SBP-BRiMS'18; July 10-13, 2018; Washington, DC, USA p. 197-206
URL: https://arxiv.org/abs/1804.06536 [doi: 10.1007/978-3-319-93372-6_22]
131. Li X, Bing L, Lam W, Shi B. Transformation Networks for Target-oriented Sentiment Classification. In: Proceedings of
the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2018 Presented at:
ACL'18; July 15-20, 2018; Melbourne, Australia p. 946-956 URL: https://arxiv.org/abs/1805.01086 [doi:10.18653/v1/p18-1087]
132. Chen P, Sun Z, Bing L, Yang W. Recurrent Attention Network on Memory for Aspect Sentiment Analysis. In: Proceedingsof the 2017 Conference on Empirical Methods in Natural Language Processing. 2017 Presented at: the Conference on
Empirical Methods in Natural Language Processing; September 7–11, 2017; Copenhagen, Denmark p. 452-461 URL:
https://www.aclweb.org/anthology/D17-1047/ [doi: 10.18653/v1/d17-1047]
133. Xu H, Liu B, Shu L, Yu P. Double Embeddings and CNN-based Sequence Labeling for Aspect Extraction. In: Proceedingsof the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). 2018 Presented
at: ACL'18; July 15-20, 2018; Melbourne, Australia p. 592-598 URL: https://www.aclweb.org/anthology/P18-2094/ [doi:10.18653/v1/p18-2094]
134. Li X, Lam W. Deep Multi-task Learning for Aspect Term Extraction With Memory Interaction. In: Proceedings of the2017 Conference on Empirical Methods in Natural Language Processing. 2017 Presented at: EMNLP'17; September 7–11,
2017; Copenhagen, Denmark p. 2886-2892 URL: https://www.aclweb.org/anthology/D17-1310/ [doi: 10.18653/v1/d17-1310]
135. Wang W, Pan S, Dahlmeier D, Xiao X. Recursive Neural Conditional Random Fields for Aspect-based Sentiment Analysis.
In: Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. 2016 Presented at:
EMNLP'16; November 1–5, 2016; Austin, Texas, USA p. 616-626 URL: https://www.aclweb.org/anthology/D16-1059/[doi: 10.18653/v1/d16-1059]
136. Manning C, Surdeanu M, Bauer J, Finkel J, Bethard S, McClosky D. The Stanford CoreNLP natural language processing
toolkit. In: Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations.
2014 Presented at: ACL'14; June 22-27, 2014; Baltimore, Maryland, USA p. 55-60 URL: https://www.aclweb.org/anthology/
P14-5010/ [doi: 10.3115/v1/p14-5010]
137. Bird S. NLTK: The Natural Language Toolkit. In: Proceedings of the COLING/ACL on Interactive presentation sessions.
2006 Presented at: COLING-ACL'06; July 17 - 18, 2006; Sydney, Australia p. 69-72 URL: https://www.aclweb.org/
anthology/P06-4018/ [doi: 10.3115/1225403.1225421]
138. Friedman C, Kra P, Rzhetsky A. Two biomedical sublanguages: a description based on the theories of Zellig Harris. J
Biomed Inform 2002 Aug;35(4):222-235 [FREE Full text] [doi: 10.1016/s1532-0464(03)00012-1] [Medline: 12755517]
139. Bodenreider O. The Unified Medical Language System (UMLS): integrating biomedical terminology. Nucleic Acids Res
2004 Jan 1;32(Database issue):D267-D270 [FREE Full text] [doi: 10.1093/nar/gkh061] [Medline: 14681409]
140. Demner-Fushman D, Rogers WJ, Aronson AR. MetaMap Lite: an evaluation of a new Java implementation of MetaMap.
J Am Med Inform Assoc 2017 Jul 1;24(4):841-844 [FREE Full text] [doi: 10.1093/jamia/ocw177] [Medline: 28130331]
141. Bradley MM, Lang PJ. The University of Vermont. 1999. Affective Norms for English Words (ANEW): Instruction Manualand Affective Ratings URL: https://www.uvm.edu/pdodds/teaching/courses/2009-08UVM-300/docs/others/everything/bradley1999a.pdf [accessed 2019-11-12]
142. Center for the Study of Emotion and Attention. ANEW Message URL: https://csea.phhp.ufl.edu/media/anewmessage.html[accessed 2019-11-12]
https://medinform.jmir.org/2020/1/e16023

XSL• FO
RenderX

JMIR Med Inform 2020 | vol. 8 | iss. 1 | e16023 | p. 20(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Zunic et al143. Nielsen FA. A new ANEW: Evaluation of a word list for sentiment analysis in microblogs. In: Proceedings of the ESWC
2011 Workshop on 'Making Sense of Microposts': Big things come in small packages. 2011 Presented at: ESWC'11; 2011;
Heraklion, Greece p. 93-98 URL: https://arxiv.org/abs/1103.2903
144. AFINN. URL: http://www2.imm.dtu.dk/pubdb/views/publication_details.php?id=6010 [accessed 2019-11-12]
145. Stone PJ, Dunphy DC, Smith MS, Ogilvie DM. General Inquirer: A Computer Approach to Content Analysis. Cambridge,
Massachusetts, USA: MIT Press; 1966.
146. Harvard General Inquirer. URL: http://www.wjh.harvard.edu/~inquirer/homecat.htm [accessed 2019-11-12]
147. Dodds PS, Harris KD, Kloumann IM, Bliss CA, Danforth CM. Temporal patterns of happiness and information in a global
social network: hedonometrics and Twitter. PLoS One 2011;6(12):e26752 [FREE Full text] [doi:10.1371/journal.pone.0026752] [Medline: 22163266]148. Language Assessment by Mechanical Turk (labMT) Sentiment Words. URL: https://trinker.github.io/qdapDictionaries/
labMT.html [accessed 2019-11-12]
149. Wilson T, Wiebe J, Hoffmann P. Recognizing Contextual Polarity in Phrase-level Sentiment Analysis. In: Proceedings ofthe conference on Human Language Technology and Empirical Methods in Natural Language Processing. 2005 Presented
at: HLT'05; October 06 - 08, 2005; Vancouver, Canada p. 347-354 URL: https://dl.acm.org/citation.cfm?id=1220619 [doi:10.3115/1220575.1220619]
150. MPQA. URL: http://mpqa.cs.pitt.edu/ [accessed 2019-11-12]
151. Mohammad SM, Turney PD. Crowdsourcing a word-emotion association lexicon. Comput Intel 2013;29(3):436-465 [FREEFull text] [doi: 10.1111/j.1467-8640.2012.00460.x]
152. NRC Emotion Lexicon. URL: http://sentiment.nrc.ca/lexicons-for-research/ [accessed 2019-11-12]
153. OpinionKB. URL: https://doi.org/10.1371/journal.pone.0124993.s001
154. Opinion Lexicon. URL: https://www.cs.uic.edu/~liub/FBS/sentiment-analysis.html#lexicon [accessed 2019-11-12]
155. de Albornoz JC, Plaza L, Gervás P. SentiSense: An easily scalable concept-based affective lexicon for sentiment analysis.
In: Proceedings of the Eighth International Conference on Language Resources and Evaluation. 2012 Presented at: LREC'12;
May 21-27, 2012; Istanbul, Turkey p. 3562-3567 URL: https://www.aclweb.org/anthology/L12-1089/
156. SentiSense Affective Lexicon. URL: http://nlp.uned.es/~jcalbornoz/SentiSense.html [accessed 2019-11-12]
157. Baccianella S, Esuli A, Sebastiani F. SentiWordNet 3.0: An Enhanced Lexical Resource for Sentiment Analysis and Opinion
Mining. In: Proceedings of the Seventh International Conference on Language Resources and Evaluation. 2010 Presented
at: LREC'10; May 17-23, 2010; Valletta, Malta p. 2200-2204 URL: https://www.aclweb.org/anthology/L10-1531/
158. SentiWordNet. URL: http://sentiwordnet.isti.cnr.it/ [accessed 2019-11-12]
159. Strapparava C, Valitutti A. WordNet Affect: An Affective Extension of WordNet. In: Proceedings of the Fourth International
Conference on Language Resources and Evaluation. 2004 Presented at: LREC’04; May 26-28, 2004; Lisbon, Portugal
URL: https://www.aclweb.org/anthology/L04-1208/
160. WordNet Domains. WordNet-Affect URL: http://wndomains.fbk.eu/wnaffect.html [accessed 2019-11-12]
161. Turney PD, Littman ML. Measuring praise and criticism: Inference of semantic orientation from association. ACM TransInf Syst 2003;21(4):315-346. [doi: 10.1145/944012.944013]
162. Taboada M, Anthony C, Voll K. Methods for Creating Semantic Orientation Dictionaries. In: Proceedings of the Fifth
International Conference on Language Resources and Evaluation. 2006 Presented at: LREC’06; May 22-28, 2006; Genoa,
Italy p. 427-432 URL: https://www.aclweb.org/anthology/L06-1250/
163. Du W, Tan S, Cheng X, Yun X. Adapting Information Bottleneck Method for Automatic Construction of Domain-oriented
Sentiment Lexicon. In: Proceedings of the third ACM international conference on Web search and data mining. 2010
Presented at: WSDM'10; February 4 - 6, 2010; New York, New York, USA p. 111-120 URL: https://dl.acm.org/citation.
cfm?id=1718502 [doi: 10.1145/1718487.1718502]
164. Miller GA. WordNet: a lexical database for English. Commun ACM 1995;38(11):39-41 [FREE Full text] [doi:10.1145/219717.219748]
165. Kim S, Hovy E. Determining the Sentiment of Opinions. In: Proceedings of the 20th international conference on
Computational Linguistics. 2004 Presented at: COLING'04; August 23 - 27, 2004; Geneva, Switzerland URL: https://www.
aclweb.org/anthology/C04-1200/ [doi: 10.3115/1220355.1220555]
166. Kamps J, Marx M, Mokken R, Rijke M. Using WordNet to Measure Semantic Orientations of Adjectives. In: Proceedingsof the Fourth International Conference on Language Resources and Evaluation. 2004 Presented at: LREC’04; May 26-28,
2004; Lisbon, Portugal p. 1115-1118 URL: https://www.aclweb.org/anthology/L04-1473/
167. Hassan A, Radev D. Identifying Text Polarity Using Random Walks. In: Proceedings of the 48th Annual Meeting of the
Association for Computational Linguistics. 2010 Presented at: ACL'10; July 11 - 16, 2010; Uppsala, Sweden p. 395-403
URL: https://www.aclweb.org/anthology/P10-1041/
168. Dragut E, Yu C, Sistla P, Meng W. Construction of a Sentimental Word Dictionary. In: Proceedings of the 19th ACM
international conference on Information and knowledge management. 2010 Presented at: CIKM'10; October 26 - 30, 2010;
Toronto, Canada p. 1761-1764. [doi: 10.1145/1871437.1871723]
169. Lu Y, Castellanos M, Dayal U, Zhai C. Automatic Construction of a Context-aware Sentiment Lexicon: An Optimization
Approach. In: Proceedings of the 20th international conference on World wide web. 2011 Presented at: WWW'11; Marchhttps://medinform.jmir.org/2020/1/e16023

XSL• FO
RenderX

JMIR Med Inform 2020 | vol. 8 | iss. 1 | e16023 | p. 21(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Zunic et al

28 - April 1, 2011; Hyderabad, India p. 347-356 URL: https://dl.acm.org/citation.cfm?id=1963456 [doi:10.1145/1963405.1963456]

Abbreviations
API: application programming interface
NB: naïve Bayes classifier
NLP: natural language processing
SA: sentiment analysis
SVM: support vector machine
UMLS: Unified Medical Language System

Edited by C Lovis; submitted 27.08.19; peer-reviewed by E Cambria, B Polepalli Ramesh, F Alemi; comments to author 26.10.19;
revised version received 26.10.19; accepted 27.10.19; published 28.01.20
Please cite as:
Zunic A, Corcoran P, Spasic I
Sentiment Analysis in Health and Well-Being: Systematic Review
JMIR Med Inform 2020;8(1):e16023
URL: https://medinform.jmir.org/2020/1/e16023
doi: 10.2196/16023
PMID: 32012057©Anastazia Zunic, Padraig Corcoran, Irena Spasic. Originally published in JMIR Medical Informatics (http://medinform.jmir.org),
28.01.2020. This is an open-access article distributed under the terms of the Creative Commons Attribution License(https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium,provided the original work, first published in JMIR Medical Informatics, is properly cited. The complete bibliographic information,a link to the original publication on http://medinform.jmir.org/, as well as this copyright and license information must be included.

https://medinform.jmir.org/2020/1/e16023

XSL• FO
RenderX

JMIR Med Inform 2020 | vol. 8 | iss. 1 | e16023 | p. 22(page number not for citation purposes)