The Sentiment Problem: A Critical Survey towards Deconstructing Sentiment Analysis
Authors: Pranav Narayanan Venkit
Mukund Srinath
Sanjana Gautam
Saranya Venkatraman
Vipul Gupta
Rebecca J. Passonneau
Shomir Wilson
Venue: EMNLP 2023 Main (2023)
Keywords: Sentiment Analysis
Natural Language Processing
Critical Survey
Ethics in NLP
Ethics based Auditing.
TL;DR: Critical survey paper about sentiment analysis in NLP and its weaknesses as a field when viewed from an interdisciplinary and sociotechnical lens.

Abstract: We conduct an inquiry into the sociotechnical aspects of sentiment analysis (SA) by critically examining 189 peer-reviewed papers on their applications, models, and datasets. Our investigation stems from the recognition that SA has become an integral component of diverse sociotechnical systems, exerting influence on both social and technical users. By delving into sociological and technological literature on sentiment, we unveil distinct conceptualizations of this term in domains such as finance, government, and medicine. Our study exposes a lack of explicit definitions and frameworks for characterizing sentiment, resulting in potential challenges and biases. To tackle this issue, we propose an ethics sheet encompassing critical inquiries to guide practitioners in ensuring equitable utilization of SA. Our findings underscore the significance of adopting an interdisciplinary approach to defining sentiment in SA and offer a pragmatic solution for its implementation.

The Sentiment Problem: A Critical Survey towards Deconstructing Sentiment Analysis
Pranav Narayanan Venkit" Mukund Srinath" Sanjana Gautam'
Saranya Venkatraman' Vipul Gupta' Rebecca J. Passonneau? Shomir Wilson'
College of Information Sciences and Technology
2 Department of Computer Science & Engineering, College of Engineering Pennsylvania State University
(pranav. venkit, mukund, sanjana. jannjaaauutm,,
saranyav, vkg5164, rjp49, shomir]epsu. edu
Abstract
Number of SA SCOPUS
8000
6000
We conduct an inquiry into the sociotechnical aspects of sentiment analysis (SA) by critically examining 189 peer-reviewed papers on their applications, models, and datasets. Our inves- tigation stems from the recognition that SA has become an integral component of diverse sociotechnical systems, exerting influence on both social and technical users. By delving into sociological and technological literature sentiment, we unveil distinet conceptualiza- tions of this term in domains such as finance, government, and medicine. Our study exposes lack of explicit definitions and frameworks for characterizing sentiment, resulting in potential challenges and biases. tackle this issue, we propose ethics sheet encompassing critical inquiries to guide practitioners in ensuring equi- table utilization of SA. Our findings underscore the significance of adopting interdisciplinary approach to defining sentiment in SA and offer a pragmatic solution for its implementation.
2000
2010
2011
2012 2013 2014 2015 2016 2017 2018
2019
2020 2021 2022
Years
Figure I: Number of articles published each year (from 2010 to 2022) SCOPUS that contain the term 'senti- ment analysis' in the title, abstract, or keywords.
et al., 2020). This lack collaborative thinking has resulted flawed analyses and biased outcomes. Given the extensive range of applications of SA spanning diverse domains such as healthcare, fi- nance, and policymaking, it is crucial to avoid repli- cating such tendencies. Furthermore, SA, despite addressing social constructs like emotion, subjec- tivity, and opinion, has been limited in its incorpo- ration of psychological and sociological definitions of sentiment (Stark and Hoey, 2021). While numer- ous studies have examined the utilization of SA, encompassing its inherent challenges and future directions (Cardie et al., 2006; Zhang et al., 2022), the interdisciplinary and sociotechnical dimensions of SA have received limited exploration.
1 Introduction
Sentiment Analysis (SA) has emerged as a signif- icant research focus in Natural Language Process- ing (NLP) over the last decade. It has now become indispensable tool in discerning opinions and emotions in written text (Medhat al., 2014), eval- uating social entities' reputation (Yuliyanti al., 2017), analyzing and predicting financial needs (Wang et al., 2013), and aiding in effective politi- cal decision-making (Cardie et al., 2006). This is illustrated in Figure which shows the rising num- bers of reviewe articles sentiment analysis published in SCOPUS every year.
To this end, we explore this gap in the literature by examining sentiment through a technical per- spective concentrating the evolution of SA into social system. We then evaluate sentiment, exam- ining the various definitions of sentiment through sociotechnical lens. We also investigate the appli- cation of SA, presenting insights into its utilization. These investigations will shed light on the interdis- ciplinary divide of the term sentiment. Next, we evaluate the motivation behind establishing neces- sary frameworks for measuring sentiment by exam- ining various different SA models and datasets.
Existing research reveals notable absence of interdisciplinary endeavors to comprehend the so- cial dimensions of SA, encompassing aspects like emotion and fairness (Mohammad, 2022; Blodgett
Authors Contributed Equally
Through our critical survey of 189 unique works

---

Categories
Frequency
tion analysis, sarcasm analysis, and toxicity anal- ysis. Sánchez-Rada and Iglesias (2019) surveyed the social context of sentiment analysis, review- ing its applications, limitations, and utilities as a sociotechnical system. Drus and Khalid (2019) surveyed works on SA from 2014 to 2019 to under- stand its social utility. They found that most of the work in SA was used in interdisciplinary contexts related to world events, healthcare, politics, and business.
Sentiment Analysis Applications
60
Sentiment Analysis Models
43
Sentiment Analysis Datasets
19
Surveys and Meta-Analysis
21
Frameworks
17
Others
29
Table I: Frequency of papers reviewed for each category of the works in SA.
Recent surveys by Birjali et al. (2021); Guo et al. (2021); Wankhade et al. (2022); Zad et al. (2021) provide up-to-date perspectives on SA reflecting shift towards fine-grained approaches, including deep learning and aspect- based sentiment analy- sis, enabling more contextual understanding of sentiment. Similarly, recent works by Zhang et al. (2022); Soni and Rambola (2022) have specifically focused on aspect-based sentiment analysis and implicit aspect detection methods. Overall, these surveys reflect a scoping of sentiment analysis to in- clude people's sentiments, opinions, attitudes, eval- uations, appraisals, and emotions towards services, products, individuals, organizations, issues, topics,
in SA (as shown in Table reveal that very few works (<5%) in SA try to explicitly define senti- ment and sentiment analysis. Our results highlight a lack of effort within the field of NLP to under- stand the interdisciplinary aspect of sentiment. We also show an absence of synchronization in the field, leading to multiple variations of the term sen- timent. Our analysis illustrates how such systems cause sociodemographie biases due to the lack of nuance in measuring sentiment. To mitigate this issue of an interdisciplinary gap, we propose an ethics sheet (Mohammad, 2022) consisting of ten critical questions to be used as a metaphorical 'nu- trition label' to understand the issues of SA models by both the user as well as the developers alike.
events, and their altributes. However, none of these works discuss the interdisciplinary framework of emotion or sentiment.
2 A Survey of Surveys
Examination of Sentiment
We now chronologically analyzing various surveys published in the field of NLP. Medhat et al. (2014) surveyed 54 articles and categorized them based on utility. They showed that SA was synonymous with opinion mining and subjective analysis, and was primarily utilized to analyze product reviews. Sim- ilarly, Alessia et al. (2015) presented a summary of SA. stating it to have evolved into a sociotechnical system (Prun and Raymond, 2021) often used in the fields of politics, public actions, and finance. Further, Ribeiro et al. (2016) reviewed SA models and benchmarked comparison of 24 SA models. They found that most models were developed to measure sentiment in social posts, product reviews, and texts in news articles. However, the metrics of measurement varied considerably across datasets and models, highlighting the need for uniformity in the field of SA.
We start by analyzing the various sentiment frame- works in SA and comparing them to existing social frameworks. By doing so, we aim to uncover the distinctions between the different notions of this term, shedding light on the gap between the techni- cal and social aspects of sentiment. In this context, we define a sociotechnical system compos- ite of social and technical components that collec- tively contribute to goal-oriented behavior, impact- ing both social and technical actors engaged with the system (Cooper and Foster, 1971). Throughout this work, we use the term 'framework' to denote a conceptual structure or set of principles that of- fer guidance for measuring or defining a specific concept within study.
3.1 The Technical Perception of Sentiment
With the advent of deep learning, more SA models were developed using deep learning archi- tectures, as summarized by Zhang et al. (2018). The work demonstrated how similar architectures could now be used in applications such as emo-
The phrase sentiment analysis likely originated from its first use case in NLP to analyze market sentiment (Das and Chen, 2001). The authors at- tempted to classify stock ratings based on opinions on a message board. Similarly, Turney and Littman

---

Framework
Definitions
Example
Measure whether words or expressions used a text convey a posilive or negative meaning
Semantic Orientation
(Agarwal et al., 2016)
Opinions or Evaluations
Author's s attitude towards a topie
(Zhai et al., 2011)
Affect or Feeling
Author's s disposition towards a specific theme
(Birjali ct al., 2021)
Framework with 3 cimensions of polarities:
3-D polarity
(Sebastiani and Esuli, 2006)
SubjeciieeOOjeetive,, PositivelNegativ, Strength
Emoticons
Emoticons sentiment indicators
(Lou et al., 2020)
Measure of the attitude towards
Object's orientation
(Mowlaci et al., 2020)
individual aspects of an entity
Emotional lendencies implied by commonsense knowledge of effect of concepts or events
Implicit
(Zhang and Liu, 2011)
Sentiment ratings collected from experts or wrowd-soureed data collection
Human Annotation
(Kenyon-Deal et al., 2018)
Table 2: Frameworks of Sentiment and corresponding definitions in Sentiment Analysis
(2002) experimented with using the semantic ori- entation of words to find whether product reviews are positive or negative. Readily available data in the form of product reviews on e-commerce web- sites influenced early SA works and firmly estab- lished it to almost exclusively mean opinion min- ing, with sentiment defined as: 'overall opinion towards the subject matter' (Pang et al., 2002).
plied by commonsense knowledge of the effect of concepts or events' to define an implicit form of sentiment. To quantify sentiment from a 'human perspective', Kenyon-Dean et al. (2018) used hu- man annotation, methodology to define and measure sentiment, using crowd-sourced data. Table tabulates the multifarious frameworks encountered in SA. Here we see that SA does not follow a well-defined comprehensive framework. With the evolution of the field, different researchers adapted SA in dissimilar ways while not making clear distinction between concepts such as emo- tions, opinions, and attitudes. We posit that there is need for nuanced, socially informed, and theo- retically motivated framework for sentiment in SA. To understand sentiment from an interdisciplinary perspective and draw out an interdisciplinary frame- work, we examine its meaning from sociological perspective.
Following this, Read (2005) proposed the use of emoticons a proxy for ground truth data to measure sentiment in text. They defined SA as the method 'identify a piece of lext according to its author's 's general feeling toward their subject, be positive or negative.' This marked a stark deviation SA from 'opinion mining." This expansion the meaning of sentiment can also be seen in the work of Wilson et al. (2005b) where they defined SA as the task of identifying positive and negative opin- ions, emotions, and evaluations'. Subsequently, Se- bastiani and Esuli (2006) proposed that SA consists of three dimensions: subjective-objeetii polarity, sesttive-negative polarity, and strength of polarity. The first use of SA as a sociotechnical system is marked by Go et al. (2009)'s approach to train SA model using data from social media platform, namely Twitter. While most prior work still treated SA as a method to extract an author's subjective or objective opinion regarding an entity or an ob- ject, Go et al. (2009) defined sentiment from the perspective of a general feeling or emotion in text. Their definition of sentiment as 'a personal pos- itive or negative feeling or opinion', is marked deviation that influenced much of the literature in SA. Maas et al. (2011)'s work recognized senti- ment a 'complex, multi-dimensional concept' and attempted to operationalize it through a vector representation. Similarly, Zhang and Liu (2011) defined sentiment as an 'emotional tendency im-
3.2 The Social Perception of Sentiment
A notable distinction exists between computational and psycho-linguistic perspectives on sentiment. In psychology, sentiment is often defined as "socially constructed patterns of sensations, expressive ges- tures, and cultural meanings organized around a relationship a social object, usually another per- son or group such as a family." (Gordon, 1981). I). While sentiment most commonly categorized as positive, negative, or neutral in computational literature, it encompasses a broader spectrum, rang- ing from mild to intense (Taboada, 2016; Jo et al.. 2017). Furthermore, sentiment (in psychology) is captured through physiological indicators, like fa- cial expressions and heart rate variability (Wiebe et al., 2005; Plutchik, 2001).
Psychological research widely recognizes that a simplistic positive-negative dichotomy is inade-

---

quate for capturing the intricate range of human emotions (Hoffmann, 2018). This evident in the distinction between seemingly negative emotions such as sadness and fear, which exhibit significant differences in their physiological and psychologica effects (Plutchik, 2001).
Terms like opinion, emotion, and subjectivity hold distinct meanings and are studied separately, Therefore, they are not synonymous with sentiment. Furthermore, when considering sentiment within a sociotechnical system, it essential to be aware of the contextual nuances associated with the diverse definitions of sentiment derived from sociological, psychological, and linguistic backgrounds. Given the complex nature of sentiment, it is important approach it with nuanced perspective and oper- ationalize within structured theoretical frame- work. Prior research suggests that achieving such nuanced understanding can be facilitated through engaging in dialogue with other fields such as psy- chology, and cognitive science (Head et al., 2015; Cambria et al., 2022). In the coming sections, we adopt these learnings in designing survey and solution.
We have seen that three primary and interrelated themes are commonly linked to sentiment: opin- ions, emotions/feelings, and subjectivity. We in- vestigate these themes gain a comprehensive understanding of sentiment that encompasses di- verse perspectives and lays the foundation for more robust SA models.
Opinions: From a psychological perspective, opinion is an individual's 's stance regarding an ob- ject or issue, formed after an evaluation through their own lens or perspective (Vaidis and Bran, 2019). This lens could be based on different fac- tors such personal beliefs, social norms, and cultural contexts. Liu (2012) also define an opinion a "a subjective statement, view, attitude, emotion, or appraisal about an entity or an aspect of an entity from an opinion holder." These definitions show that opinion can merit different purposes de- pending on the context.
4 Critical Analysis of Sentiment Analysis
As shown in the previous sections, the sentiment framework employed in SA differs substantially from the established social frameworks of senti- ment. This disparity can pose challenges when applying SA sociotechnical systems (Stark and Hoey, 2021). We, therefore. critically analyze SA. including its application, models, and datasets. Our goal is to assess the suitability of SA in sociotech- nical system, which aims to address societal prob- lems by integrating people and technology (Prun and Raymond, 2021). The detailed roadmap of our survey is depicted in the Appendix (Figure 3).
Feelings/Emotions: Izard (2010) posit that the word emotion has both descriptive definition i.e. based on its use in everyday life and prescriptive definition i.e. based on the scientific concept that is used to identify definite set of events. Another ap- proach to defining emotions is based on three essen- tial components: motor expression, bodily symp- toms/arousal, and subjective experience. There is substantial agreement that motivational conse- quences and action tendencies associated with emo- tion are key aspects of emotion rather than just the level of arousal of the subject (Frijda et al., 1986; Frijda, 1987).
4.1 Study 1: Applications of Sentiment Analysis
The conceptualization of sociotechnical systems underscores the intricate interplay between social and technical factors and actors during system de- velopment and operation (Trist, 1981). Hence, we first explore the integration of SA component within sociotechnical systems.
Subjectivity: Banfield (2014) referred to sen- tences that take character's psychological point of view as subjective, contrasted against sentences that narrate an event in a definite but yielding man- ner. Private states and experiences play a pivotal role during expression of subjectivity. Here private states could refer to intellectual factors, such as believing, wondering, knowing; emotive factors, such hating. being afraid; and perceptual ones, such as seeing or hearing something (Wiebe, 1994). Study of subjectivity further proves to be challeng- ing as sociologists often isolate emotions from their social context while studying them.
We conducted an analysis 60 papers that ana- lyzed the applications SA over time (Drus and Khalid, 2019; Sánchez- Rada and Iglesias, 2019; Ramírez- Tinoco et al., 2019) from databases such as SCOPUS and Semantic Scholar, employing tar- geted keywords like 'sentiment analysis' and 'ap- plications' together. We obtained a corpus of 95 research papers, from which we filtered out and excluded 35 extraneous works not related to SA. We performed an iterative qualitative thematic

---

Category
Definition
Applications that utilize individual health data make predictions and informed decisions pertaining health-related behaviors medical practices. Applications designed for government bodies to analyze and determine appropriate courses of action concerning public issues or problems that require attention and intervention. Applications that collect and analyze diverse data points to identify trends patterns that can influence stralegic decision- decision-making in business.
Health and Medicine
Govemment and Policy Making
Business Analytics
Applications that aggregate and extract meaningful insights from data obtained through social channels (such as social media platforms like Twitter) to facilitate decision-nen makiong and gain understanding of societal behaviors. Applications developed comprehend the patterns and dynamics financial management, creation, and investment analysis.
Social Media Analytics
Finance
Table 3: List of applications, defined through thematic analysis, their corresponding definitions, and frequency of papers categorized the groups.
analysis (Vaismoradi et al., 2013) to uncover the various applications of SA. Each author studied and classified the work based the intended scope of application. To ensure accuracy and prevent misclassification, this recursive process was em- ployed. The resulting classification encompasses five categories as shown in Fig. 2 and Table 3/. No- tably, the Health and Medicine domain emerged as the most prominent application area for SA where studies leverage SA to understand individual reac- tions in diverse medical scenarios (Rodrigues et al., 2016). Following closely, Government and Policy Making emerged as the second most prevalent cate- gory, where sentiment analysis plays a pivotal role comprehending human behavior in governance solutions (Joyce and Deng, 2017). This categoriza- tion underscores the multifaceted utility of SA as an integral component of sociotechnical systems across various fields. It is worth noting that all the reviewed works assign mathematical value to sentiment, categorizing it positive, negative, or neutral scoring it on a scale (e.g.. +1).
Most of the reviewed works lack clear definitions of sentiment or SA. Only 31 out of the 60 papers explain the employed framework, and just 2 out of 60 explicitly define sentiment in their applications. Only one takes an interdisciplinary perspective. defining sentiment in the context of finance for understanding market behavior (Kraaijeveld and De Smedt, 2020). Most works assume that senti- ment encompasses public opinion, perception, and overall emotion. Sentiment, tone, emotion, opin- ion, and subjectivity are often used interchangeably, despite their distinct meanings socially.
The lack of precise sentiment definitions can result in misrepresented measurements. The com- monly used SA framework, initially intended for finance and reviews, may not suffice for compre- hending sentiment in social contexts. Utilizing this framework in domains such health and policy- making could have notable implications, may fail to capture the genuine essence of sentiment.
4.2 Study 2: Sentiment Analysis Service
In this study, we will explore various published models and datasets of SA available for public consumption, examining their characteristics and limitations, and emphasizing the need for an inter- disciplinary approach their development.
Catagories of Application
Healt and Modieine
20
Government
13
Businwes Analyice
11
Social Media Analylia
The market has witnessed a rapid prolifera- tion of Al as a Service (AlaaS) models that of- fer convenient "plug-and-play" AI services and tools (Lewicki et al., 2023) for public consumption across diverse interdisciplinary fields (Sánchez- Rada and Iglesias, 2019). We gathered SA datasets and popularly used models, that are publicly acces- sible for use as AlaaS, by leveraging existing repos- itories such as Sentibench (Ribeiro et al., 2016). We also conducted targeted searches using key-
20
Frequency
Figure 2: Thematic categories of applications of senti- ment analysis in the 60 papers.
'The categorization of cach paper is present in the Ap- pendix

---

words such as "sentiment analysis' and 'model" across peer-reviewed platforms such as the ACL Anthology, NeurIPS proceedings, AAAI, and ACM anthology. Following an extensive filtering process, we identified 43 well-cited 2 SA models and 19 datasets that are publicly available for utilization. We now look at these models and datasets, using a critical lens our intention is to examine them interdisciplinary and sociotechnical awareness. We, therefore, examine them by formulating the following key questions:
"the hedonic feelings of pleasantness; referred to in the psychological literature as "affect"" (Hannak et al., 2012)
"sentiment helps convey meaning and react to sentiments expressed towards them or others." (Ma et al., 2018)
These two examples serve to demonstrate the in- adequacy of the information provided regarding the definition of sentiment. The remaining surveyed works fail to offer any description of the framework employed for sentiment in SA.
Q2: Analysis of Metries
Do these works mention the framework def- inition of sentiment analysis and sentiment?
Our analysis of the 43 models and 19 datasets re- veals the utilization of 11 distinct metrics to gauge the sentiment expressed in statements These met- rics can be broadly categorized into two groups: sentiment categorization and sentiment regression.
How do these works measure sentiment?
How accessible are these models for its use as an AlaaS solution?
The first group, sentiment categorization, fo- cuses on classifying text into categories associated with positive or negative sentiment, or subjective and objective tone. However, these categories are not well-defined, as certain models further catego- rize sentiment based on emotions such Joy, Sad- ness, Anger, Fear, Disgust, Surprise, (Mohammad. 2012) or Self-assurance, Attentiveness, Fatigue, Guilt, Fear, Sadness, Hostility, Joviality, Serenity, Surprise, and Shyness (Gonçalves et al., 2013) or between emotion categories of Valence, Arousal, and Dominance (Warriner et al., 2013). We see no synchronization in the categories used.
Q1- Analysis of Frameworks:
Among the 62 collected models and datasets, observed that merely 18 papers presented a defi- nition of the SA framework employed, while just 2 works attempted to provide a definition for sen- timent. Similarly, for datasets published, we see that papers provided a definition of the SA frame- work while just provided a definition of sentiment used. The most common framework used is of opinions. The deficiency in coherent structuring of sentiment and sentiment analysis definitions shows an absence of uniformity in terminology across the domain, as illustrated by the following examples: "Sentiment analysis refers to the general method to extract subjectivity and polarity from the text." (Taboada et al., 2011)
In contrast, the second group, sentiment regres- sion, focuses on evaluating a numerical value for a sentence, which is subsequently categorized as positive, neutral, or negative. We note when we refer to sentiment regression we are only referring to 'regression to the mean' techniques applied in measurement and not implying the use of machine learning regression techniques. Regression-based scales employ scores ranging from negative num- ber to a positive number (e.g.. -1 to +1) to quantify the intensity and sentiment the sentence.
"Sentiment analysis or opinion mining analyzes people's 's opinions, sentiments, evaluations, atti- tudes, and emotions via the computational treat- ment of subjectivity in text. (Hutto and Gilbert, 2014)
"Sentiment analysis is a branch of affective com- puting research that aims to classify text into either positive negative, but sometimes also neutral. (Ma et al., 2018)
Without standardized measures, it becomes chal- lenging to compare results, establish common understanding of sentiment, and benchmark perfor- mance. These metrics do not measure the same quantity even if it appears under the umbrella of sentiment. Standardizing sentiment measures would address these issues by promoting consis- tency, enhancing interpretation, and improving in-
These quotes demonstrate the varied use of SA in each study, highlighting its focus on quantifying latent constructs such as 'emotion," 'subjectivity," and "attitude," which are not fully explained. The following two quotes demonstrate the framework used to define sentiment:
"The hreakdown of each of the 11 classes, with examples, is presented in the Appendix.
Paverage citation count of 1130

---

Schwartz tegration with social applications.
Sentence
Score
Q3: Analysis of Accessibility & Transparency We will now delve into the accessibility of SA models deployed as AT-as-a-Service (AlaaS) sys- tems. Assessing the accessibility of the model sheds light on how the field strives to provide clearer access to its solutions in sociotechnical en- vironments, where the behavior of the model is more comprehensibly explicated. We scrutinize three key aspects of the model: code availability, dataset accessibility, and ease of model access.*.
1 am a tail person.
0.00
1 am beautiful person.
0.85
I am a black person.
-0.16
I am mentally handicapped person.
-0.10
I am blind person.
-0.50
Table 4: Example of TextBlob sentiment analysis library with sentence set.
issues that arise due the lack of explanation in creating solutions using an interdisciplinary lens, Due to limited and restricted data and the sub- jective nature of sentiment, the training data used to train SA models are not representative of all perspectives (Kiritchenko and Mohammad, 2018; Gupta al., 2023) and thus result in biases that can be harmful to real-world applications. We demon- strate this with an experiment on Textblob, a SA model. Table 4 shows how certain terms generate negative sentiments irrespective of context. How- ever, is difficult to comprehend what the negative scores mean in a social setting where they can be interpreted as toxic or hateful (Venkit et al., 2023; Kiritchenko and Mohammad, 2018). Thus, the use of sentiment analysis models can lead to discrimi- nation against certain groups (Huang al., 2020; Shen et al., 2018). The existence of sentiment bias can also lead to poor performance of sentiment analysis models (Han et al., 2018).
Source Code Accessibility: Among the 43 an- alyzed models, we find that only 15 (35%) offer access to their source code. while the remaining models (65%) do not. The prevailing trend indi- cates reluctance disclose details or provide ac- cess the source code. This highlights the general treatment of these AlaaS systems as black boxes, where the reasons behind the SA model's behavior cannot be readily explained.
Training Dataset Accessibility: Out of the 43 models, only 16 (37%) grant access to the training dataset employed in the model development. Con- versely, the remaining models (63%) do not provide any means of accessing the training dataset. Such systems impede the replication of the model's re- sults, as they do not offer external means verify or test the outcomes.
Ease of Access: We further investigate the in- clusivity of access provided by SA AlaaS models. Our analysis reveals that (12%) of the 43 mod- els impose restrictions access. These models cither operate behind paywall or necessitate spe- cific credentials to obtain full access to their per- formance. These instances demonstrate that not all AlaaS models are genuinely public in nature.
SA models are shown to perform differently for different age groups (Díaz et al., 2018). They show that SA models more likely to be positively biased towards 'young' adjectives than 'old" adjec- tives. Hutchinson et al. (2020) also demonstrate how bias exists against people with disability in toxicity prediction and sentiment analysis mod- els. These models are shown to be biased against African-American names (Rozado, 2020) and dis- criminate against English text written by non-native English speakers (Zhiltsova et al., 2019). Hube et al. (2020) found that there exist prior sentiments associated with some names in pre-trained word embeddings used to train machine learning mod- els. Such biased machine learning models can have harmful implications when used in weal-world set- tings (Rudin, 2019; Bender et al., 2021; et al., 2021).
is important to understand if these publicly available systems can become opaque, leading to unexplained outcomes and potential biases (Bender et al., 2021; O' O'neil, 2017).
4.3 Study 3: The Bias and Harm of Sentiment Analysis Applications
In the prior sections, showed that not only is there general lack of effort in defining sentiment in SA models, but SA contains multiple frame- works that can hinder collaboration within the field. Additionally, such work tend to not disclose details on how they are developed. Next, we explore the
The works by Stark and Hoey (2021) & Moham- mad (2022) argue that the complexity of human emotion and the limits of technical computation raise serious social, political, and ethical considera- tions that merit further discussion in AI ethics. The
'The detailed breakdown of each of these works published at www.ha/.ww...aaaaaeeeeeeeeeee Proteem/hlahmmiin/Suryy..ss

---

field of Al has not caught up well with the complex- ities of human behavior. The same is seen in the field of SA where we cannot socially comprehend what a negative or positive sentiment means or even captures. This can cause wrongful interpretation of the results causing social harm and bias. Dev et al. (2021) also demonstrate how these misinterpreta- tions in the result of SA models can lead to social harm such as dehumanization, erasure, and stereo- typing. Therefore effort needs be placed into truly understanding the value of sentiment being measured by such models, especially when they are used a sociotechnical system. Such efforts can help in promoting inclusivity and diversity in real-world applications.
Lack of standardization in sentiment mea- surement: The absence of standardized metries to quantify sentiment results in the use of multiple scales and categorizations in SA. This lack of stan- dardization makes it challenging to compare and interpret results across different models and studies, leading to a proliferation of diverse approaches for evaluating sentiment.
Lack of consensus between various frame- works defined in SA There are multiple frame- works created SA to measure sentiment. How- ever, these frameworks have been adopted based on individual usage without reaching an accord among other existing frameworks. This lack of consensus amongst multiple frameworks undermines the over- all integrity of research in this area.
5 The Weaknesses in Sentiment Analysis
Manifestation of bias in publicly released models: The absence of standards can lead to biased or subjective sentiment analysis. Differ- ent measures may introduce bias or subjectivity based on the perspectives assumptions of the researchers developers, potentially affecting the accuracy and fairness of the analysis. As shown in our analysis, publicly available models often demonstrate biases against specific social groups, reflecting inconsistencies in the captured values. Limitations in generalizability of SA mod- els: The use of different scales and categorizations limits the generalizability of SA models. Models trained on specific categorization schemes struggle to handle sentiments that fall outside the predefined categorics, rendering them less applicable in real- world scenarios. This issue becomes particularly apparent when models exhibit harmful misclassi- fication towards minority groups due to limited understanding of their context and language.
Based on our survey analysis, we outline the key weaknesses encountered in SA within NLP. Adopt- ing an interdisciplinary lens, our focus centers on the interpretability within sociotechnical systems, in order to provide targeted recommendations for future work.
Limited awareness of sentiment in a sociotech- nical context: SA often lacks the understanding of how sentiment is conceptualized beyond its techni- cal purview (diseussed in Scction 2.2). When SA is employed in sociotechnical systems like healthcare, it is important to define the socially relevant frame- work of sentiment. There is no motivation shown to understand the social, political, and psychological considerations of sentiment in these works.
Insufficient emphasis on capturing contextual information and subtleties: Categorization -aased approaches in SA struggle to capture contextual in- formation and subtle variations in the sentiment ex- pressed in text. Factors such as tone, sarcasm, and cultural nuances that influence sentiment may not be adequately addressed by predefined categories limited numerical scores. Most analyzed works focus primarily lexically categorizing texts as positive or negative, without considering the social factors that contribute to sentiment measurement.
Addressing these issues requires careful consid- eration of the categorization approach, integration of contextual information, and, efforts towards ro- bust evaluation methodologies in sentiment analy- sis. In the following section we will look at how we can focus on creating a solution and awareness of these issues.
Wide range of vague and absent definitions: The literature on SA exhibits diverse and conflict- ing definitions and frameworks, often lacking clar- ity or omitting explicit definitions for sentiment and SA. Ambiguity arises from the use of terms like 'attitude,' 'tone,' "subjectivity." and 'tone' inter- changeably, without clear definitions in the context of sentiment analysis.
6 Recommendations and Ethics Sheet in Creating A Sentiment Model
Prior works like Blodgett et al. (2020), Gebru al. (2021) & Bender and Friedman (2018) have cre- ated data statements and ethics sheets as a means to audit and provide noteworthy indications to resolve issues in AI, through a list of meaningful ques-

---

tions. Building on these works, we now discuss how practitioners conducting work analyzing 'sen- timent' in NLP can avoid the challenges discussed in our previous sections. We, therefore, propose 4 primary recommendations from which we will build an ethics sheet to guide works in SA.
without access restrictions? [R2]
(Q3.2) Is the model algorithm publicly published
without any access restrictions? [R2]
(Q4) Is this system primarily designed for users
outside the field of NLP? [RI+R4]
(Q5) What are the specific use cases this system is intended for? [RI+R4]
[R1] Use interdisciplinary understanding to es- tablish a comprehensive framework for sentiment analysis that incorporates insights from fields be- yond NLP. Differentiate between sentiment, opin- ion, subjectivity, and emotion analysis, employing a shared vocabulary and consistent terminology.
(Q6) Who are the users and user profiles in- tended to utilize the system? [R1+R4]
(Q7) Were tests conducted to identify explicit and implicit biases in sentiment analysis mod- els, specifically examining the various sociodemo- graphic biases that may be exhibited? If yes, please provide details. [R3]
[R2] Ensure explicit documentation of the sen- timent framework and analysis methodology em- ployed in sentiment analysis works. Provide guide- lines that outline the expected measurements and quantifications for the model to enhance inter-
(Q8) Were experts from interdisciplinary fields involved in discussing the use and metrics of senti- ment analysis models as social applications? If so,
pretability and applicability.
please specify them explicitly. [R3]
[R3] Explicitly state the use cases and user pro- files intended to interact with the sentiment analysis system. By considering the specific applications and targeted users, mitigate potential biases in the model's results. Raise awareness about potential biases introduced by sentiment analysis models, emphasizing the importance of fairness and equity. [R4] Use explainable SA models to enhance transparency and interpretability. Encourage the de- velopment of methods that provide insights into the model's decision decision-making process, allowing users understand how sentiment analysis results are generated and enabling trust in the system. This can be done by making sure the training data and code of the model are available to all.
(Q9) Did the study consider the potential cultural or contextual variations in sentiment interpretation?
If so, how were they addressed? [R3]
(Q10) Were there any measures implemented to mitigate potential biases in the model? If yes, please explain the approach taken. [R3]
These contextually structured questions aid in uncovering underlying assumptions embedded in framing the task of creating a SA model. Addi- tionally, it presents novel ethical considerations unique and specifically pertinent to understanding the sociotechnical nature of SA.
Conclusion
From the above recommendations, we build an ethics sheet that contains questions that can be used while building aspects associated with sentiment analysis. We intend this ethies sheet to be used as additional material to the existing literature to bring awareness to the issues caused by SA a sociotechnical system. Additionally, we aim for the ethics sheet to facilitate democratic usability and public scrutiny of the model, empowering users to make informed choices when selecting suitable model for their specific applications.
In survey of 189 papers on SA, observe that, firstly, SA has shifted from analyzing prod- uet reviews to being widely used in sociotechnical systems like health and medicine. Secondly, there is a lack of interdisciplinary exploration in defin- ing social concepts in SA, such as sentiment. The frameworks used for sentiment analysis often suffer from vagueness, inconsistency, or absence. Thirdly, many publicly available works create restricted black boxes with limited access to the model training dataset. To address these challenges, we offer four key recommendations and an ethics sheet to guide future researchers and practitioners. We aim to help improve the development of SA models by enhancing clarity, interpretability, and ethical considerations through our work.
(Q1) What is the framework and definition of sentiment utilized? [R2]
(Q2) What framework is employed for sentiment analysis in the measurement of sentiment? [R2] (Q3) Will this study be made available for public in measuring sentiment in NLP? [R2]
(Q3.1) Is the training dataset publicly published

---

Limitations
Acknowledgment
We extend our gratitude to Grace Kathleen Ciambrone from Pennsylvania State University for her contributions to the curation and analysis of papers related to sentiment analysis datasets and models. Additionally, we wish express our ap- preciation the reviewers for their time, insight- ful feedback, and constructive suggestions, all of which significantly enhanced the clarity and com- prehension of our research.
Our study encompasses a selection of 189 pa- pers, incorporating works from ACL Anthology. NeurlPS proceedings, SCOPUS proceedings, and Semantic Scholar query searches. While our in- tention not to provide exhaustive collection of all published works on sentiment analysis, aimed to include diverse sources that cover vari- ous aspects of the field. Our intent was to curate peer-reviewed literature commonly found in the sentiment analysis domain, encompassing models, applications, survey papers, and frameworks. Un- fortunately, we encountered scarcity of works addressing multilingualityy, which reflects the the- matic underrepresentation in the broader field. Con- sequently, plan delve deeper into the preva- lent themes within sentiment analysis research to address this gap and provide due attention under- represented areas in our upcoming work. Regard- ing the creation of the ethics sheet, is important to note that the questions presented are not meant to be exhaustive but rather serve as a foundational framework to spark additional inquiries and foster further engagement.
References
Mohammed H Abd El-Jawad, Rania Hodhod, and Yasser MK Omar. 2018. Sentiment analysis of social media networks using machine learning. In 2018 international computer engineering conference (ICENCO), pages 174-176. IEEE.
Basant Agarwal, Namita Mittal, Basant Agarwal, and Namita Mittal. 2016. Semantic orientation-bes based approach for sentiment analysis. Prominent feature extraction for sentiment analysis, pages 77-88.
Sanjida Akter and Muhammad Tareq Aziz. 2016. Senti- ment analysis on facebook group using lexicon based approach. In 2016 3rd international conference on electrical engineering and information communica- tion technology (ICEEICT), pages 1-4, IEEE
Ethics Statement
Md Hijbul Alam, Woo-Jong Ryu, and SangKeun Lec. 2016. Joint multi-grain topic sentiment: modeling semantic aspects for online reviews. Information Sciences, 339:206-22 223.
We are aware the ethical considerations involved in our research and have taken measures to ensure responsible practices throughout the study.
Md Mahfuz Ibn Alam. Ruoyu Xie, Fahim Faisal, and Antonios Anastasopoulos. 2023, Gmnlp at semeval- 2023 task 12: Sentiment analysis with phylogeny- based adapters. arXiv preprint arXiv:2304. 12979.
Data Publication: All the papers used in our re- search are listed in the Appendix. However, we rec- ognize the importance of transparency and account- ability. Therefore, we publish the complete collec- tion of papers along with our qualitative classifica- tion and annotation, allowing for public scrutiny and examination.
Alessia, Fernando Ferri, Patrizia Grifoni. and Tiziana Guzzo. 2015. Approaches, tools and applications for sentiment analysis implementation. International Journal of Computer Applications, 125(3).
Mitigating Qualitative Study Bias: We acknowl- edge the potential for bias when performing qual- itative coding of the papers regarding their appli- cations. To address this concern, we ensured that at least three different individuals independently reviewed and verified the coding to minimize the possibility of misclassification. Additionally, we followed the same approach to verify the presence of various definitions in each paper, enhancing the reliability and validity of our analysis. By disclos- ing these ethical considerations, we emphasize our commitment to conducting research in an ethical and accountable manner.
Kashif Ali, Hai Dong, Athman Bouguettaya, Abdelka- rim Erradi, and Rachid Hadjidj. 2017. Sentiment analysis service: social media based sentiment analysis framework. In 2017 IEEE international conference web services (ICWS), pages 660-667. IEEE.
Reinald Kim Amplayo. 2019. Rethinking attribute rep- resentation and injection for sentiment classification. arXiv preprint arXiv:1908 095900
Muhammad Zubair Asghar, Shakeel Ahmad, Maria Qasim, Syeda Rabail Zahra, and Fazal Masud Kundi. 2016. Sentihealth: creating health-related sentiment lexicon using hybrid approach. SpringerPlus, 5:1- 23.

---

Elliott Ash. Daniel L Chen, and Sergio Galletta. 2022. Measuring judicial sentiment: Methods and applica- tion to us circuit courts. Economica, 89(354)3662- 376.
Marouane Birjali, Abderrahim hammed Erritali. 2017. Machine learning and seman- tic sentiment analysis based algorithms for suicide sentiment prediction social networks. Procedia Computer Science, 11::65-72.
and Mo-
Muhammad Hilmi Asyrofi, Zhou Yang, Imam Nur Bani Yusuf, Hong Jin Kang. Ferdian Thung, and David Lo. 2021. Biasfinder: Metamorphic test generation to uncover bias for sentiment analysis systems. IEFE Transactions on Software Engineering, 48(12):5087- 5101.
Marouane Birjali, Mohammed Kasri, and Abderrahim Beni-Hssane, 2021. comprehensive survey on sen- timent analysis: Approaches, challenges and trends. Knowledge-Basee Systems, 226: 107134.
John Blitzer, Mark Dredze, and Fernando Pereira. 2007. Biographies, bollywood, doom-boxes and blenders: Domain adaptation for sentiment classification. Proceedings of the 45th anmual meeting of the asso- ciation of computational linguistics, pages 440-447.
Stefano Baccianella, Andrea Esuli, Fabrizio Sebastiani, al. 2010. Sentiwordnet 3.0: an enhanced lexical resource for sentiment analysis and opinion mining. In Lrec, volume 10, pages 2200-2200 2204.
Ann Banfield. 2014. Unspeakable sentences (routledge revivals): narration and representation in the lan- guage fiction. Routledge.
Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. 2020. Language (technology) is power: A critical survey of "bias" in nlp. In Proceed- ings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5454 5476.
Francesco Barbieri, Jose Hamacho-Collados. Leonardo Neves, and Luis Espinosa-Anke. 2020. Tweet- eval: Unified benchmark and comparative eval- uation for tweet classification. arXiv preprint 1iXiv:2010.12421.
Afrin Jaman Bonny, Mchrin Jahan, Zannatul Ferdhoush Tuna, Ahmed Al Marouf, and Shah Md Tanvir Sid- diqee. 2022. Sentiment analysis of user-generated reviews of women safety mobile applications. In 2022 First International Conference on Electrical, Electronics. Information and Communication Tech- nologies (ICEFICT), pages 1-6. IEFE
Jeremy Barnes, Laura Oberlaender, Enrica Troiano, An- drey Kutuzov, Jan Buchmann, Rodrigo Agerri, Lilja Øvrelid, and Erik Velldal. 2022. SemEval 2022 task 10: Structured sentiment analysis. In Proceedings of the 16th International Workshop on Semantic Eval- uation (SemEval-2022), pages 1280-1295, Seattle, United States. Association for Computational Lin- guistics.
Rajesh Bose, Raktim Kumar Dey, Sandip Roy, and De- babrata Sarddar. 2020. Sentiment analysis on online product reviews. In Information and Communication Technology for Sustainable Development: Proceed- ings of ICT4SD 2018, pages 559-569. Springer.
Christos Baziotis, Nikos Pelekis, and Christos Doulk- eridis. 2017. Datastories at semeval-2017 task Deep Istm with attention for message-level and topic- based sentiment analysis. In Proceedings of the 11th international workshop on semantic evaluation (SemEval-201 77), pages 747-754.
Ngot Bui, John Yen, and Vasant Honavar. 2016. Tempo- ral causality analysis of sentiment change in a cancer survivor network. IEEE transactions on computa- rional social systems, 3(2):75-87.
Erik Cambria, Daniel Olsher, and Dheeraj Rajagopal. 2014. Senticnet 3: common and common-sense knowledge base for cognition- -riven sentiment anal- ysis. In Proceedings of the AAAI conference on arti- ficial intelligence, volume 28.
Emily M Bender and Batya Friedman. 2018 Data statements for natural language processing: Toward mitigating system bias and enabling better science. Transactions of the Association for Computational Linguistics, 6:587-604 6004.
Erik Cambria, Frank Xing, Mike Thelwall, and Roy Welsch. 2022. Guest editorial: Sentiment analysis multidisciplinary research area. IEFE Transactions on Artificial Intelligence, 3(05):638-66
Emily M. Bender, Timnit Gebru, Angelina McMillan-
Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language mod- els be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability and Trans- parency, FAceT '21, page 610-623, New York, NY, USA. Association for Computing Machinery.
Jianping Cao, Zeng, Hui Wang, Jiajun Cheng, Feng- cai Qiao, Ding Wen, and Yanqing Gao. 2013. Web- based traffic sentiment analysis: Methods and appli- cations. IEEE transactions on Intelligent Transporta- rion systems, 1542):844-853.
Jayadev Bhaskaran and Isha Bhallamudi. 2019, Good secretaries, bad truck drivers? occupational gender stereotypes in sentiment amalysis. In Proceedings of the First Workshop Language Processing, pages 62 -68, Florence, Italy. Association for Computational I Linguistics.
Claire Cardie, Cynthia Farina, and Thomas Bruce 2006. Using natural language processing to improve erule- making: project highlight. In Proceedings of the 2006 international conference on Digital government research, pages 177-178.
Gender Bias in Natural

---

Jireh Yi-Le Chan, Khean Thye Bea, Steven Mun Hong Leow, Seuk Wai Phoong. and Wai Khuen Cheng. 2023. State of the art: review of sentiment analy- sis based on sequential transfer learning. Artificial Intelligence Review, 56(1):749-780.
Jingcheng Du Jun Xu, Hsing- Yi Song, and Cui Tao. 2017. Leveraging machine learning- ap- proaches to assess human papillomavirus vaccination sentiment trends with twitter data. BMC medical informatics and decision making, 17:63-70.
L Clement, 2013. Umigon: Sentiment analysis on tweets based on terms lists and heuristics.
Imane EI Alaoui, Youssef Gahi, Rochdi Messoussi, Youness Chaabi, Alexis Todoskoff, and Abdessamad Kobi. 2018. novel adaptable approach for senti- ment analysis on big social data. Journal of Big Data. 5(1):1-18.
Jack G Conrad and Frank Schilder. 2007. Opinion min- ing in legal blogs. In Proceedings of the 11th interna- tional conference on Artificial intelligence and law, pages 231 1-36.
Fabian Falck, Julian Marstaller, Niklas Stochr, Sören Maucher, Jeana Ren, Andreas Thalhammer, Achim Rettinger, and Rudi Studer. 2020. Measuring proxim- ity between newspapers and political parties: the sentiment political compass. Policy & internet.
Robert Cooper and Michael Foster. 1971. Sociotechni- cal systems. American Psychologist, 26(5):467.
12(3):367-399
W Christian Crannell, Eric Clark, Chris Jones, Ted A James, and Jesse Moore. 2016. A patter-matched twitter analysis of us cancer-patient sentiments, jour- nal of surgicul research, 206(2):536-542.
Teng- Kai Fan and Chia-Hui Chang. 2009. Blogger- centric contextual advertising. In Proceedings of the 18th ACM conference on Information and knowledge management, pages 1803-1806.
Sanjiv Ranjan Das and Mike Y Chen. 2001. Yahoo! for amazon: Sentiment parsing from small talk on the web. For Amazon: Sentiment Parsing from Small Talk on the Web (August 5, 2001). EFA.
Fang Fang, Yin Zhou, Shi Ying, and Zhijuan Li. 2023, A study of the ping an health app based user views with sentiment analysis. International Jour- nal of Environmental Research and Public Health, 20(2):1591.
Thomas Davidson, Debasmita Bhattacharya, and Ing- mar Weber. 2019. Racial bias in hate speech and abusive language detection datasets. arXiv preprint arXiv:1905.1 12516.
Tirana Noor Fatyanosa and Fitra A Bachtiar. 2017. Clas- sification method comparison on indonesian social media sentiment analysis. In 2017 International Con- ference on Sustainable Information Engineering and Technology (SIET), pages 310-315. IEEE
Tom De Smedt and Walter Daelemans. 2012. Pattern for python. The Journal of Machine Learning Research, 13(1):0033-067.
Nico H Frijda. 1987. Emotion, cognitive structure, and action tendency. Cognition and emotion, 1(2):1 15- 143.
Dong Deng, Liping Jing, Jian Yu, Shaolong Sun, and Michael K Ng. 2019. Sentiment lexicon con- struction with hierarchical supervision topic model. IEEE/ACM Transactions on audio, speech, and lan- guage processing. 274477004-718
Nico Frijda et al. 1986. The emotions. University Press.
Cambridge
Kavita Ganesan and ChengXiang Zhai. 2011. OpinRank Review Dataset. UCI Machine Learning Repository. DOI: http:://toi..rg//0.2244444. 1044244/523///
Sunipa Dev, Emily Sheng. Jieyu Zhao, Aubrie Am- stutz, Jiao Sun, Yu Hou, Mattie Sanseverino, Jiin Kim, Akihiro Nishi, Nanyun Peng, et al. 2021. On measures of biases and harms nlp. arXiv preprint urXiv:2108.03360
Diego Garcia. 2013. Sentiment during recessions. The journal of finance, 6863):1267-1300.
Lorenzo Gatti, Marco Guerini, and Marco Turchi. 2015. Sentiwords: Deriving high precision and high cov- erage lexicon for sentiment analysis. IEEE Transac- tions on Affective Computing, 7(4):409-421 421.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understand- ing. Xrxiv preprint arxiv:1810.04885..
Timnit Gebru, Jamie Morgenstern, Briana Vec- chione, Jennifer Wortman Vaughan. Hanna Wallach. Hal Daumé lii, and Kate Crawford. 2021. Datasheets for datasets. Communications of the ACM, 64(12)866 92.
Mark Díaz, Isaac Johnson, Amanda Lazar, Anne Marie Piper, and Daree Gergle. 2018. Addressing age- related bias in sentiment analysis. In Proceedings of the 2018 chi conference on human fumanfactors in comput- ing systems, pages 1-14.
Elena Georgiadou, Spyros Angelopoulos, and Helen Drake. 2020. Big data analytics and international negotiations: Sentiment analysis of brexit negotiat- ing outcomes. International Journal of Information Management, 51:102048.
Zulfadzli Drus and Haliyana Khalid. 2019. Sentiment analysis in social media and its application: System- atic literature review. Procedia Computer Science, 161:707-714.

---

Alec Go, Richa Bhayani, and Lei Huang. 2009, Twit- ter sentiment classification using distant supervision. CS224N project report, Stanford, ((12):2009.
Minqing Hu and Bing Liu. 2004, Mining and summa- rizing customer reviews. In Proceedings of the tenth ACM SIGKDD international conference on Knowl- edge discovery and data mining, pages 168-177.
Pollyanna Gonçalves, Fabrício Benevenuto, and Meey- oung Cha. 2013. Panas-t: pychometric scale for measuring sentiments on twitter. arXiv preprint arXiv: 1857.
Po-Sen Huang, Huan Zhang, Ray Jiang, Robert Stan- forth, Johannes Welbl, Jack Rae. Vishal Maini, Dani Yogatama, and Pushmeet Kohli. 2020. Reducing Sen- timent Bias in Language Models via Counterfactual Evaluation. In Findings of the Association for Com- putational Linguistics: EMNLP 2020, pages 65-83. Association for Computational Linguistics.
Vinodhini Gopalakrishnan and Chandrasekaran Ra- maswamy. 2017. Patient opinion mining to analyze drugs satisfaction using supervised learning. Journal of applied research and technology, 15(4):311-312
Christoph Hube, Maximilian Idahl, and Besnik Fetahu. 2020. Debiasing word embeddings from sentiment associations in names. In Proceedings of the 13th International Conference on Web Search and Data Mining, WSDM 200, page 259-267, New York, NY, USA. Association for Computing Machinery.
S Gordon. 1981. I.(1981)'the sociology of sentiments and emotions.". Rosenberg M. and Turner R., H. pages 261-278.
Xiaoting Guo, Wei Yu, and Xiaodong Wang. 2021. An overview fine-grained text sentiment analysis: Sur- vey and challenges. In Journal of Physics: Confer- ence Series, volume 1757, page 012038. IOP Pub- lishing.
Ben Hutchinson, Vinodkumar Prabhakaran, Emily Den- ton, Kellie Webster, Yu Zhong, and Stephen Denuyl. 2020. Social biases in NLP models as barriers for persons with disabilities. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5491-5501, Online, Association for Computational Linguistics.
Vipul Gupta, Pranav Narayanan Venkit, Shomir Wilson, and Rebecca Passonneau. 2023. Survey on sociode- mographic bias in natural language processing. arXiv preprint arXivv23066008 08/88.
Hongyu Han, Yongshi Zhang, Jianpei Zhang, Jing Yang. and Xiaomei Zou. 2018. Improving the performance of rexicon-based review sentiment analysis method by reducing additional introduced sentiment bias. PLoS One, (32))222
Clayton Hutto and Eric Gilbert. 2014. Vader: parsi- monious rule-based model for sentiment analysis of social media text. In Proceedings of the international AAAI conference on web and social media, volume 8, pages 216-225.
Aniko Hamnak, Eric Anderson, Lisa Feldman Barrett, Sune Lehmann, Alan Mislove, and Mirek Riedewald. 2012. Tweetin in the rain: Exploring societal-scale effects of weather on mood. In Proceedings of the International AAAI Conference on Web and Social Media, volume 6, pages 479-482.
Victoria Ikoro, Maria Sharmina, Khaleel Malik, and Riza Natista-Navarro. 2018. Analyzing sentiments expressed on twitter by uk energy company con- sumers. In 2018 Fifth international conference on social networks analysis, management and security (SNAMS), pages 95-98. IEEE.
Anees UI Hassan, Jamil Hussain, Musarrat Hussain, Muhammad Sadiq, and Sungyoung Lee. 2017. Sen- timent analysis of social networking sites (sns) data using machine learning approach for the measure- 2017 international confer-
Radu Tudor Ionescu and Andrei M Butnaru. 2019. Vec- tor of locally-aggregated word embeddings (vlawe): A novel document-level representation. arXiv preprint arXiv:1902.08855
ment of depression. ence on information and communication technology convergence (ICTC), pages 138-140. IEEE.
Haruna Isah, Paul Trundle, and Daniel Neagu. 2014. So- cial media analysis for product safety using text min- ing and sentiment analysis. In 2014 14th UK work- shop on computational intelligence (UKCI), pages
Ruining He and Julian McAuley. 2016. Ups and downs: Modeling the visual evolution fashion trends with one-class collaborative filtering. In proceedings of the 25th international conference on world wide web. pages 407-517.
1-7. IEEE.
Carroll Izard. 2010. The many mcanings/aspectss
of emotion: Definitions, functions, activation, and regulation. Emotion Review, 2(4):363-370.
Megan L Head, Luke Holman, Rob Lanfear, Andrew Kahn, and Michael D Jennions. 2015. The extent and consequences of p-hacking in science. PLoS biology, 130))xe0002106.
JA Izzo and K Maloy. 2017. 86 sentiment analysis demonstrates variability in medical student grading. Annals of Emergency Medicine, 70(4):S35-S5 S36.
Thomas Hoffmann. 2018 "too many americans are trapped in fear, violence and poverty": : a psychology- informed sentiment analysis of campaign speeches from the 2016 us presidential election. Linguistics Vanguard, 4(1).
Jahanzeb Jabbar, Iqra Urooj. Wu JunSheng, and Naqash Azeem. 2019. Real-time sentiment analysis on e- commerce application. In 2079 IEEE 16th interna- tional conference on nefworking, sensing and control (ICNSC), pages 391-396. IEFE

---

Xiang Soon Ae Chun, and James Geller. 2013. Mon- itoring public health concerns using twitter sentiment classifications. In 2013 IEFF International Con- ference on Healthcare Informatics, pages 335-344, IEEE.
Naveen Kumar Laskari and Suresh Kumar Sanam- pudi. 2016. Aspect based sentiment analysis survey. IOSR Journal of Computer Engineering (IOSR-JCE), 18(2)244-28.
Kornel Lewicki, Michelle Seng Ah Lee, Jennifer Cobbe, and Jatinder Singh. 2023. Out of context: In- vestigating the bias and faimeess concerns of" ar- tificial intelligence as a service". arXiv preprint 2xXiv23302.0145:
Tee Wee Jing and Raja Kumar Murugesan. 2019. A theoretical framework to build trust and prevent fake news in social media using blockchain. In Recent Trends in Data Science and Soft Computing: Proceed- ings of the 3rd International Conference of Reliable Information and Communication Technology (IRICT 2018), pages 955-962. Springer.
Hengyun Li, XB Bruce, Gang Li, and Huicai Gao. 2023. Restaurant survival prediction using customer- generated content: An aspect- based sentiment anal- ysis of online reviews. Tourism Management, 96:104707.
Hwiyeol Jo, Soo-Min Kim, and Jeong Ryu. 2017. What we really want find by sentiment analysis: the relationship between computational models and psy- chological state. arXiv preprint
170..03407.
Hui Li, Jiangtao Cui, Bingqing Shen, and Jianfeng Ma. 2016. An intelligent movie recommendation system through group-level sentiment analysis in microblogs. Neurocomputing. 210:164-173 173.
Brandon Joyce and Jing Deng. 2017. Sentiment analysis of tweets for the 2016 us presidential election. In 2017 ieee mit undergraduate research technology conference (urtc), pages 1-4, IEEE
Raymond Li, Samira Ebrahimi Kahou, Hannes Schulz, Vincent Michalski, Laurent Charlin, and Chris Pal. 2018. Towards deep conversational recommenda- tions. Advances in neural information processing systems, 31.
Kian Kenyon-Deam Eisha Ahmed, Scott Fujimoto. Jeremy Shoorgs-Fiiltauu Christopher Glasz, Barleen Kaur, Auguste Lalande, Shruti Bhanderi, Robert Belfer, Nirmal Kanagasabai, et al. 2018. Sentiment analysis: It's complicated! In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, Volume (Long Pa- pers). pages 1886-1895.
Chen Lin, Xinyi Liu, Guipeng Xv, and Hui Li. 2021. Mitigating sentiment bias for recommender systems. In Proceedings of the 44th International ACM SI- GIR Conference on Research and Development in Information Retrieval, pages 3 31-40.
Bing Liu. 2012. Sentiment analysis and opinion mining. Synthesis lectures on human language technologies. 5(1):1-167.
Svetlana Kiritchenko and Saif Mohammad. 2018. Ex- amining gender and race bias in two hundred senti- ment analysis systems. In Proceedings of the Sev- enth Joint Conference on Lexical and Computational Semantics, pages 43-53, New Orleans, Louisiana. Association for Computational Linguistics.
Yinxia Lou, Yue Zhang, Fei Li, Tao Qian, and Donghong Ji. 2020. Emoji-based sentiment anal- ysis using attention networks. ACM Transactions asian and low-resource language information pro- cessing (TALLIP), 19(5):1133.
loannis Korkontzelos, Azadeh Nikfarjam, Matthew Shardlow, Abeed Sarker, Sophia Ananiadou, and Gra- cicla H Gonzalez. 2016. Analysis of the effect of sen- timent analysis on extracting adverse drug reactions from tweets and forum posts. Journal of biomedical informatics, 62: 148-158.
Tim Loughran and Bill McDonald. 2011. When is liability not liability? textual analysis. dictionaries, and 10-ks. The Journal of finance, 66(1):35 65..
Chenyang Lyu, Jennifer Foster, and Yvette Graham. 2020. Improving document-level sentiment analy- sis with user and product context. arXiv preprint arXiv:2011.0922
Komal Kothari, Aagam Shah, Satvik Khara, and Hi- manshu Prajapati. 2020. A novel approach in user reviews analysis using text summarization and senti- ment analysis: Survey.
Dehong Ma, Sujian Li, Xiaodong Zhang, Houfeng Wang, and Xu Sun. 2017. Cascading multiway atten- tions for document-level sentiment classification. In Proceedings of the Eighth International Joins Con- ference on Natural Language Processing (Volume 1: Long Papers), pages 634 643.
Olivier Kraaijeveld and Johannes De Smedt. 2020. The predictive power of public twitter sentiment for fore- casting cryptocurencyy prices. Journal of Interna- tional Financial Markets, Institutions and Money, 65:101188 188.
Namhee Kwon, Stuart W Shulman, and Eduard Hovy. 2006. Multidimensional text analysis for erulemak- ing. In Proceedings of the 2006 international con- ference on Digital government research, pages 157- 166.
Yukun Ma, Haiyun Peng, and Erik Cambria. 2018. Tar- geted based sentiment analysis via embedding commonsense knowledge into attentive Istm. In Proceedings of the AAAI conference on artificial in- telligence, volume 32.

---

Andrew Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, and Christopher Potts. 2011. I Learning word vectors for sentiment analysis. In Proceedings of the 49th annual meeting of the associ- ation for computational linguistics: Human language technologies, pages 142-150.
Preslav Nakov, Sara Rosenthal, Zornitsa Kozareva. Veselin Stoyanov, Alan Ritter, and Theresa Wilson. 2013. SemFval-2013 13 task 2: Sentiment analysis in Twitter. In Second Joint Conference on Lexical and Computational Semantics ( (*SEM), Volume 2: Pro- ceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013), pages 312- 320, Atlanta, Georgia, USA. Association for Compu- tational Linguistics.
Shamsul Arafin Mahtab, Nazmul Islam, and Md Mah-
fuzur Rahaman. 2018. Sentiment analysis on bangladesh cricket with support vector machine. In 2018 international conference on Bangla speech and language processing (ICBSLP), pages 1-4, IEEE.
Pranav Narayanan Venkit, Sanjana Gautam, Ruchi Pan- chanadikar, Ting-Hao Huang, and Shomir Wilson. 2023. Unmasking nationality bias: A study of human perception of nationalities in ai-generated articles. In Proceedings of the 2023 AAAT/ACM Conference on AI, Ethics, and Society, pages 554 565.
Bodhisattwa Prasad Majumder, Shuyang Li, Jianmo Ni, and Julian McAuley, 2019. Generating personal- ized recipes from historical user preferences. arXiv preprint arXiv: 1909000105
Finn Arup Nielsen. 2011. A new anew: Evaluation of a word list for sentiment analysis in microblogs. arXiv preprint arXiv: / 100.2990.
Samah Mansour. 2018. Social media analysis user's responses to terrorism using sentiment analysis and text mining. Procedia Computer Science, 140:95- 103.
Cathy O'neil. 2017. Weapons of math destruction: How big data increases inequality and threatens democ- racy. Crown.
Walaa Medhat, Ahmed Hassan, and Hoda Korashy. 2014. Sentiment analysis algorithms and applica- tions: A survey. Ain Shams engineering journal, 5(4):1093-1113.
Alvaro Ortigosa, José M Martín, and Rosa M Carro. 2014. Sentiment analysis in facebook and its appli- cation to e-learning. Computers in human behuvior, 31:527-541.
Katelyn Mei, Sonia Fereidooni, and Aylin Caliskan. 2023. Bias against 93 stigmatized groups in masked language models and downstream sentiment classifi- cation tasks. In Proceedings of the 2023 ACM Confer- ence on Fairness, Accountability, and Transparency, pages 1699-1710.
Bo Pang and Lillian Lee. 2004. A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts. arXiv preprint cs/0409058.
Saif Mohammad. 2012. # emotional tweets. In * SEM 2012: The First Joint Conference on Lexical and Computational Semantics- the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantie Evaluation (SemEval 2012), pages 246- 255.
Bo Pang and Lillian Lee. 2005. Seeing stars: Exploiting class relationships for sentiment categorization with respect rating scales. arXiv preprint cs/0506075.
1: Proceedings of
Pang, Lillian Lee, and Shivakumar Vaithyanathan. 2002. Thumbs up? sentiment classification us- ing machine learning techniques. arXiv preprint cs/0205070.
Saif M Mohammad. 2022. Ethics sheet for automatic emotion recognition and sentiment analysis. Compu- tational Linguistics, 48(2):239-278.
Nikolaos Pappas, Georgios Katsimpras, and Efstathios Stamatatos. 2013. Distinguishing the popularity be- tween topics: a system for up-to-date opinion re- trieval and mining in the web. In Computational Lin- guistics and Intelligent Text Processing: 14th Inter- national Conference, CICLing 2013, Samos, Greece, March 24-30, 2013, Proceedings, Part Il 14, pages 197-209, Springer.
Saif M Mohammad, Svetlana Kiritchenko, and Xiaodan Zhu. 2013. Nre-canada: Building the state-of-the. art in sentiment analysis of tweets. arXiv preprint 1088:1308.6244
Saif M Mohammad and Peter D Turnnyy 2013. Crowd- sourcing a word-emotion association lexicon. Com- putational intelligence, 29(3):436-466 465.
Robert Plutchik. 2001. The nature of emotions: Human emotions have deep evolutionary roots, a fact that may explain their complexity and provide tools for clinical practice. American scientist, 89(4):344-355
Antonio Moreno- and Chantal Pérez-Hernánee 2018. Lingmotif-lex: a wide-coverage, state-of-the- art lexicon for sentiment analysis. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018).
Vinodkumar Prabhakaran, Ben Hutchinson, and Mar- garet Mitchell. 2019. Perturbation sensitivity analy- sis to detect unintended model biases. In Proceedings of the 2019 Conference on Empirical Methods in Nat- ural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5740-5745, Hong Kong. China. Association for Computational Linguistics.
Mohammad Erfan Mowlaei, Mohammad Sanice Abadeh, and Hamidreza Keshavarz. 2020. Aspect- based sentiment analysis using adaptive aspect- based lexicons. Expert Systems with Applications, 148:1 13234.

---

S McKay Price. James Doran, David R Peterson, and Barbara A Bliss. 2012. Earnings conference calls and stock returns: The incremental informative- ness of textual tone. Journal of Banking & Finance, 36(4):992-1011
Sara Rosenthal, Noura Farra, and Preslav Nakov. 2019. 2omeval-2017 task 4: Sentiment analysis in twitter. arXiv preprint arXiv: 1912.007).
David Rozado. 2020. Wide range screening of algo- rithmic bias in word embedding models using large sentiment lexicons reveals underreported bias types. PloS one, 15(4)cc02315 1889.
Daniel Prun and Camille Raymond. 2021. controlled experiment on using cognitive work analysis for sys- tem engineering definition process. 2021 16th Inter- national Conference of System of Systems Engineer- ing (SoSE), pages 1-6.
Cynthia Rudin. 2019. Stop explaining black box ma- chine learning models for high stakes decisions and use interpretable models instead. Nature machine intelligence, 1(5):206 215.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020, Exploring the limits of transfer learning with unitied text-to-text trans- former. The Journal of Machine Learning Research, 21(1):5485-552)
Susan Sabra, Khalid Mahmood Malik, and Mazen Alobaidi. 2018 Prediction of venous thromboem- bolism using semantic and sentiment analyses of clin- ical narratives. Computers in biology and medicine. 94:1-10.
Adil Rajput. 2020. Natural language processing, senti- ment analysis. and clinical analytics. In Innovation in health informatics, pages 79- 97. Elsevier.
Marzich Sacidi, Guillaume Bouchard, Maria Liakata, and Sebastian Riedel. 2016. Sentihood: Targeted aspect based sentiment analysis dataset for urban neighbourhoods. arXiv preprint arXiv:1610.0377
Francisco Javier Ramírez-Tinos
Giner Alor- Hernández, José Luis Sänchez-Cervantes María del Pilar Salas Zárate, and Rafael Salencia-Garcia 2019. Use of sentiment analysis techniques in healthcare domain. Current Trends in Semantic Web Technologies: Theory and Practice, pages 189-212.
María del Pilar Salas-Zárate, Jose Katty Lagos-Ortiz, Harry Luna-Aveiga, Miguel An- gel Rodriguez-Gareee Gareia, and Rafael Valencia-Gare Garcia.. 2017. Sentiment analysis on tweets about diabetes: aspect-level approach. Computational and mathe- matical methods in medicine, 2017.
Shalli Rani, Ali Kashif Bashir, Adi Alhudhaif, Deepika Koundal, Emine Selda Gunduz, et al. 2022. An ef- ficient cnn-Istm model for sentiment detection inff blacklivesmatter Expert Systems with Applications, 993:116256.
Fernando Sánchez-Rade and Carlos A Iglesias. 2019. Social context in sentiment analysis: Formal defini- tion, overview of current trends and framework for comparison. Information Fusion,
Jonathon Read. 2005. Using emoticons reduce depen- deney in machine learning techniques for sentiment classification. In Proceedings of the ACL student research workshop, pages 43-48.
Melva Hermayanty Saragih and Abba Suganda Girsang. 2017. Sentiment analysis of customer engagement on social media in transport online. In 2017 Interna- tional Conference on Sustainable Information Engi- neering and Technology (SIET), pages 24-29. IEEE.
Filipe N Ribeiro, Matheus Araújo, Pollyanna Gonçalves, Marcos André Gonçalves, and Fabrício Benevenuto. 2016. Sentibench-a benchmark comparison of state- theeteeee sentiment analysis methods. EPJ Data Science, 5:1-29.
Robert P Schumaker, Yulei Zhang, Chun-Neng Huang, and Hsinchun Chen. 2012. Evaluating sentiment in financial news articles. Decision Support Systems, 53(3):458 464.
Alexander Rietzler, Sebastian Stabinger, Paul Opitz, and Stefan Engl. 2019. Adapt or get left behind: Do- main adaptation through bert language model finetun- ing for aspect-target sentiment classification. arXiv preprint 1908 11800
Reva Schwartz, Leann Down, Adam Jonas, and Elham Tabassi. 2021. A proposal for identifying and manag- ing bias in artificial intelligence. Draft NIST Special Publication, 1270.
Fabrizio Sebastiani and Andrea Esuli. 2006. Sentiword- net: A publicly available lexical resource for opinion mining. In Proceedings of the 5th international con- ference on language resources and evaluation, pages 417-422. European Language Resources Association (ELRA) Genoa, Italy.
Ramon Gouveia Rodrigues, Rafael Marques das Dores, Celso G Camilo-Junion and Thierson Couto Rosa. 2016. Sentihealth canthe- sentiment analysis tool help detecting mood of patients in online social net- works. International journal of medical informatics, 85(1):80-95.
Firoj Fattulal Shahare. 2017. Sentiment analysis for the news data based on the social media. In 2017 International Conference Intelligent Computing and Control Systems (ICICCS), pages 1365-1370. IEEE
Lavinia Rognone, Stuart Hyde, and S Sarah Zhang. 2020. News sentiment in the cryptocurrency market: An empirical comparison with forex. International Review of Financial Analysis, 69: 462.

---

Shahid Shayaa, Phoong Scuk Wai. Yeong Wai Chung, Ainin Sulaiman, Noor Ismawati Jaafar, and Shamshul Bahri Zakaria. 2017. Social media sen- timent analysis on employment in malaysia. the Proceedings of 8th Global Business and Finance Re- search Conference, Taipei, Taiwan.
Peter D Turney and Michael Littman. 2002. Un- supervised learning of semantic orientation from hundred-billion-word corpus. arXiv preprint cs/02/2012.
Eddie L Ungless, Björn Ross, and Vaishak Belle. 2023. Potential pitfalls with automatic sentiment analysis: The example of queerphobic bias. Social Science Computer Review, page 08944393231152946
Judy Hanwen Shen, Lauren Fratamico, lyad Rahwan, and Alexander M Rush. 2018. Darling baby- girl? investigating stylistic bias in sentiment analysis. Proc. of FATML.
David C Vaidis and Alexandre Bran. 2019. Respectable challenges respectable theory: Cognitive disso- nance theory requires conceptualization clarifica- tion and operational tools. Frontiers in psychology. 10:1 189.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference empiri- cal methods in natural language processing, pages 1631-1642.
Mojtaba Vaismoradi, Hannele Turunen, and Terese Bon- das. 2013. Content analysis and thematic analysis: Implications for conducting a qualitative descriptive study, Nursing & health sciences, 15(3):398 405.
Piyush Kumar Soni and Radhakrishna Rambola. 2022. A survey implicit aspect detection for sentiment analysis: terminology, issues, and scope. IEEE Ac- cess, 10:63932-63957.
Pranav Venkit, Zeba Karishma, Chi- Yang Hsu, Rahul Katiki, Kenneth Huang, Shomir Wilson, and Patrick Dudas. 2021. Asourceful 'wistt: Emoji prediction based on sentiment, hashtags and application source. arXiv preprint 2xx::v::100... 103.07833.
Luke Stark and Jesse Hoey. 2021. The ethics of emotion in artificial intelligence systems. In Proceedingg of the 2021 ACM Conference on Fairness, Accountabil- ity, and Transparency, pages 782-793.
Pranav Narayanan Venkit, Mukund Srinath, and Shomir Wilson. 2023. Automated ableism: An exploration of explicit disability biases in sentiment and toxicity analysis models. arXiv preprint ar 230i 092099.
Chi Sun, Luyao Huang, and Xipeng Qiu. 2019. Uti- lizing bert for aspect-based sentiment analysis via constructing auxiliary sentence. arXiv preprint arXiv:1903.0955
Pranav Narayanan Venkit and Shomir Wilson. 2021. Identification of bias against people with disabilities in sentiment analysis and toxicity detection models. arXiv preprint arXiv:2111.3322
Chris Sweeney and Maryam Najafian. 2020. Reducing sentiment polarity for demographic attributes in word embeddings using adversarial learning. In Proceed- ings of the 2020 Conference on Fuirness, Account- ability, and Transparency, pages 359-368.
Chuan-Ju Wang. Ming-Feng Tsai, Tse Liu, and Chin- Ting Chang. 2013. Financial sentiment analysis for risk prediction. In Proceedings of the Sixth Interna- tional Joint Conference on Natural Language Pro- cessing, pages 802-808.
Y Watequlis Syaifudin and Dwi Puspitasari. 2017. Twit- ter data mining for sentiment analysis peoples feedback against government public policy. Matter Int. J. Sei. Technol, 10-2110-12
Hao Wang, Dogan Can, Abe Kazemzadeh, François Bar, and Shrikanth Narayanan. 2012. A system for real-time twitter sentiment analysis of 2012 us pres- idential election cycle. In Proceedings of the ACL 2012 system demonstrations, pages 1 115 120.
Maite Taboada. 2016. Sentiment analysis: An overview from linguistics. Annuul Review of Linguistics, 2:325-347.
Jianghao Wang. Yichun Fan, Juan Palacios, Yuchen Chai, Nicolas Jeantta-eaarenaaud, Nick Obradovich, Chenghu Zhou, and Siqi Zheng. 2022. Global idence of expressed sentiment alterations during the covid-19 pandemic. Nature Human Behaviour, 6(3):349-355
Maite Taboada, Julian Brooke, Milan Tofiloski, Kim- berly Voll, and Manfred Stede, 201 Lexicon-based methods for sentiment analysis. Computational lin- guistics, 37(2):267-307
Wenya Wang, Sinno Jialin Pan, Daniel Dahlmeier, and Xiaokui Xiao. 2016. Recursive neural conditional random fields for aspect- based sentiment analysis. arXiv prepriat arXiv:1603.064:
Mike Thelwall. 2014. Heart and soul: Sentiment strength detection the social web with sen- tistrength, 2017. Cyberemotions: Collective emo- tions in cyberspace.
Mayur Wankhade, Annavarapu Chandra Sekhara Rao, and Chaitanya Kulkarni. 2022. A survey on senti- ment analysis methods, applications, and challenges. Artificial Intelligence Review, 55(7):5731-5780.
Eric L Trist. 1981. The evolution of socio-technical systems, volume 2. Ontario Quality of Working Life Centre Toronto.

---

Amy Beth Warriner, Victor Kuperman, and Mare Brys- baert. 2013. Norms of valence, arousal, and domi- nance for 13,915 english lemmas. Behavior research methods, 15:1191
Staci M Zavattaro, P Edward French, and Somya D Mohanty. 2015. A sentiment analysis of us local government tweets: The connection between tone and citizen involvement. Government information quarterly, 33333:33.-341
Janyce Wiebe, Theresa Wilson, and Claire Cardie. 2005. Annotating expressions of opinions and emotions in language. Language resourees and evaluation, 39: 195-21..
Zhongwu Zhai, Bing Liu, Hua Xu, and Peifa Jia. 201 I. Clustering product features for opinion mining. Proceedings of the fourth ACM international confer- ence Web search and data mining, pages 347-354.
Janyce M Wiebe. 1994. Tracking point of view in nar- rative. arXiv preprint cmp-lg/9407019.
Lei Zhang and Bing Liu. 2011. Identifying noun prod- uct features that imply opinions. In Proceedings of the 49th annual meeting of the Association for Computational Linguistics: human language tech- nologies, pages 575-580- 550-
Theresa Wilson, Paul Hoffmann, Swapna Somasun- daran, Jason Kessler, Janyce Wiebe, Yejin Choi, Claire Cardie, Ellen Riloff, and Siddharth Patward- han. 2005a. Opinionfinder: system for subjectivity analysis. In Proceedings of HLT/EMNILP 2005 Inter- active Demonstrations, pages 34-35,
Lei Zhang, Shuai Wang, and Bing Liu. 2018. Deep learning for sentiment analysis: A survey. Wiley Intendisciplinary Reviews: Data Mining and Knowl- edge Discovery, 8(4)c12253
Theresa Wilson, Janyce Wiebe, and Paul Hoffmann. 2005b. Recognizing contextual polarity in phrase- level sentiment analysis. In Proceedings of human language technology conference and conference on empirical methods natural language processing, pages 347-354.
Wenxuan Zhang, Xin Li, Yang Deng, Lidong Bing, and Wai Lam. 2022. A survey on aspect-based senti- ment analysis: tasks, methods, and challenges. IEEE Trunsactions on Knowledge and Data Engineering.
Alina Zhiltsova, Simon Caton, and Catherine Mulway. 2019. Mitigation of unintended biases against non- native english texts in sentiment analysis. In AICS, pages 317-328.
Liang Wu, Teng- Long-Sheng Moh, and Natalia Khuri. 2015. Twitter opinion mining for adverse drug reactions. In 2015 IEEF international conference big data (Big Data), pages 1570-1574. IEEE.
A Appendix
Zhengxuan Wu and Desmond C Ong. 2021. Context- guided bert for targeted based sentiment anal- ysis. In Proceedings of the AAAI conference on arti- ficial intelligence, volume 35, pages 14094-14102.
A.1 Application of SA
In this section, we illustrate the examples and cat- egories of works that were looked into for under- standing the various applications of SA. We catego- rize the purpose of SA into 5 major categories. The definitions and categories of all the applications are mentioned in Table 3.
Hu Xu, Bing Liu, Lei Shu, and Philip Yu. 2019. Bert post-training for review reading comprehension and aspect-based sentiment analysis. arXiv preprint arXiv:1904 02232.
Fu-Chen Yang, Anthony JT Lee, and Sz-Chen Kuo. 2016. Mining health social media with sentiment analysis. Journal of medical systems, 40:1-8.
Health and Medicine: Ji et al. (2013), Wu et al. (2015), Rodrigues et al. (2016), Bui et al. (2016), Korkonizelos et al. (2016), Asghar et al. (2016), Du et al. (2017), Yang al. (2016), Hassan et al. (2017), Ali al. (2017), Gopalakrishnan and Ramaswamy (2017), Birjali et al. (2017), Sabra et al. (2018), Salas-Zárate et al. (2017), Izzo and Maloy (2017), Crannell et al. (2016), Rajput (2020), Ramírez-Timen et al. (2019), Wang et al. (2022), Fang et al. (2023)
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Car- bonell, Russ R Salakhutdinov, and Quoe V Le. 2019. Xlnet: Generalized autoregressive pretraining for lan- guage understanding. Advunces in neural informa- tion processing systems, 32.
Siti Yuliyanti, Taufik Djatna, and Heru Sukoco. 2017. Sentiment mining of community development pro- gram evaluation based on social media. TELKOM- NIKA (Telecommunication Computing Electronics and Control), 15(4): 1858-1644..
Government and Policy Making: Kwon et al. (2006), Conrad and Schilder (2007), Zavattaro et al. (2015), Yuliyanti al. (2017), Syaifudin and Puspi- tasari (2017)Jooce and Deng (2017). Shayaa et al. (2017), Fatyanosa and Bachtiar (2017), Mansour (2018), Ikoro et al. (2018), Falck et al. (2020), Geor- giadou et al. (2020), Ash et al. (2022)
Samira Zad, Maryam Heidari, James Jones, and Ozlem Uzuner. 2021. A survey on concept-level sentiment analysis techniques of textual data. In 2021 IEEE World AI loT Congress (AlloT), pages 0285-0291. IEFE

---

1/ peppo
Seitimeet Analyss Frameworts
10 papperx
Nentarent Ansilye
pepers
Arrayore Application
10 Quzstions
243 Papers
Evolussion Fileiigg
Hapers
Thematk Coding Issues Challegge
Thomes
papers
Sentiment Anabyst Motes
21 popers
Sentiment Analysi Sunwey
22 papers
Miscelaneous
Figure 3: Roadmap of the collection and analysis process of all the peer-reviewed sentiment analysis papers design the Ethics Sheet.
Business Analyties: Fan and Chang (2009), Wang et al. (2013), Isah al. (2014), Akter and Aziz (2016), Li et al. (2016), Saragih and Girsang (2017), Jabbar et al. (2019), Bose et al. (2020), Bonny et al. (2022), Li et al. (2023),
Cambria al. (2022); Ma et al. (2017); Devlin al.
(2018); Liu (2012); Raffel et al. (2020); Yang et al. (2019); lonescu and Butnaru (2019) Baccianella et al. (2010); Pappas et al. (2013)
A.3 Sentiment Analysis Datasets
Social Media Analytics: Cao et al. (2013); Or- tigosa et al. (2014); Shahare (2017); Mahtab al. (2018); Abd El-Jawad et (2018), 1); El Alaoui et al. (2018); Jing and Murugesan (2019); Rani et al.
Socher et al. (2013); Maas et al. (2011); Wiebe et al. (2005); Li et al. (2018); Barbieri et al. (2020); Rosenthal et al. (2019); Pang and Lee (2004, 2005); Nakov et al. (2013); Barnes et al. (2022); Alam et al. (2023); Blitzer al. (2007) Go et al. (2009);
(2022); Venkit et al. (2021)
Finance: Loughran and McDonald (2011); Price al. (2012); Schumaker et al. (2012); Wang et al. (2013); Garcia (2013); Rognone et al. (2020);
Ganesan and Zhai (2011); Majumder al. (2019); He and McAuley (2016); Alam et (2016); Kir- itchenko and Mohammad (2018)
Kraaijeveld and De Smedt (2020)
A.2 Sentiment Analysis Models
A.4 Sentiment Analysis Surveys
Hu and Liu (2004); Wilson et al. (2005a); Sebas- tiani and Esuli (2006); Nielsen (201 1); Taboada al. (2011); ); Mohammad and Turney (2013); Han- nak et al. (2012); Mohammad (2012): De Smedt and Daelemans (2012); Wang et al. (2012); Gonçalves et al. (2013) Mohammad et al. (2013);
Medhat et al. (2014); Alessia al. (2015); Ribeiro et al. (2016); Laskari and Sanampudi (2016); Zhang et al. (2018, 2022); Sánchez-Rada and Igle- sias (2019); Drus and Khalid (2019); Ramírez- Tinoco et al. (2019); Kothari et al. (2020); Birjali et al. (2021); Mohammad (2022); Guo al. (2021) Zad al. (2021); Wankhade al. (2022); Soni and Rambola (2022); Chan et al. (2023)
Socher et al. (2013); Clement (2013); Warriner et al. (2013); Cambria et al. (2022); Thelwall (2014);
Hutto and Gilbert (2014); Gatti et al. (2015); Wang et al. (2016); Saeidi et al. (2016); Baziotis al. (2017): Moreno-Ortiz and Pérez-Hernández (2018); ; Ma al. (2018) Deng al. (2019); Xu et (2019); Sun et al. (2019); Amplayo (2019); Rietzler et al. (2019); Lyu et al. (2020); Wu and Ong (2021) I);;
A.5 Bias in Sentiment Analysis
Huang al. (2020); Díaz et al. (2018); Venkit and Wilson (2021); Bhaskaran and Bhallamudi (2019):
Kiritchenko and Mohammad (2018); Zhiltsova et al. (2019); Hube et al. (2020); Han et al.

---

(2018); Sweeney and Najafian (2020); Prabhakaran al. (2019) Rozado (2020); Hutchinson et al.
explicit biases sheds light on the potential harm that a poorly administered model may exacerbate.
(2020); Davidson al. (2019); Shen et al. (2018); Narayanan Venkit et al. (2023); Asyrofi et al. (2021); Ungless et al. (2023); Lin et al. (2021);
Mei et al. (2023); Venkit et al. (2023)
A.6
Breakdown of the Metries used
Sentiment Analysis
Negative, Objective, Positive (Wilson al., 2005a)) Negative, Positive (Cambria al., 2014)) Negative, Neutral, Positive (Wang al., 2016) Very Negative, Negative, Neu- tral, Positive, Very Positive (Socher et al., 2013) Positive, Somewhat Positive, Neutral, Somewhat Negative, Negative (Devlin et al., 2018) Valence. Arousal, Dominance (Warriner et al., 2013) Neg- ative, Neutral, Unsure, Positive (De Smedt and Daelemans, 2012)1 Self-assurance, Attentiveness, Fatigue, Guilt, Fear, Sadness, Hostility, Joviality, Serenity, Suprise, Shyness (Gonçalves al., 2013) Joy, Sadness, Anger, Fear, Disgust, Surprise (Mo- hammad, 2012)
Sentiment Categorization:
Sentiment Regression Scales: [-5,+5] (Nielsen, 2011)I [0,2,4] (Mohammad et al., 2013)l [-1,+1] (Gonçalves et al., 2013) (Hutto and Gilbert, 2014)
A.7 Breakdown of Ethics Sheet
In this section, we aim to analyze the underlying intention behind each question posed in the Ethics Sheet.
Questions (Q1-Q3) are designed to focus on rec- ommendation [R2]. The disclosure of neces- sary information pertaining to the framework and analysis methodology is crucial. This disclosure contributes to the interpretability of sociotechnical systems employing SA. enhancing the understand- ing of their functioning.
Questions (Q4-Q6) are tailored address rec- ommendations [RI and [R4]. The outcomes de- rived from these questions foster an interdisci- plinary comprehension of SA developments. Ex- plicitly stating user profiles and associated data em- powers users with democratic choice in selecting suitable applications required.
Questions (Q7-Q10) emphasize the significance of comprehending weaknesses and biases inherent in model. These questions align with recom- mendation [R3] by providing additional contextual information regarding model performance. The inclusion of information concerning implicit and

---

Definition Framework
References
sentiment
affective state or feeling associated with particular object or event
(Hoffmann, 2018)
subjective statement, view, attitude, emotion, or appraisal about an entity or an aspect of an entity from an opinion holder
opinion
(Liu, 2012)
By "descriptive definition," is used in everyday life. By "prescriptive definition," definition of the scientific concept that is used to pick out the set of events that scientific theory of emotion purports explain.
mean a definition of the word emotion mean
emotion/feclings
(Izard, 2010)
subjectivity analysis deals with the detection of "private states" opinions, emotions, evaluations, beliefs and speculations.
subjectivity
that encloses sentiment.
(Wiebe, 1994)
Table 5: Examples of few definitions of different themes concerning sentiment from different fields to demonstrate the difference framework between these terms that are synonymously used the field of SA in NLP.


Reviews:
[{"review_id": "Bcb3q50LUf", "reviewer": "Reviewer_tS1k", "summary": "", "questions": "It would be nice to provide a sample-filled example of the ethics sheet for a sentiment analysis study, like the \"Show Your Work\" paper [1]? This will help people understand how to implement your ethics sheet in practive.\n\n[1] [Show Your Work: Improved Reporting of Experimental Results](https://aclanthology.org/D19-1224) (Dodge et al., EMNLP-IJCNLP 2019)", "limitations": "", "rating": 4, "confidence": 4, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and invaluable insights in evaluating our paper! We value and agree with your viewpoint on our survey's importance in promoting safer and more transparent development of sentiment analysis applications. It is important to have such critical analysis done on the field to demonstrate how the same can be improved for a safer and less biased environment. Your engagement with our work is greatly appreciated. \n\nAnswer to your questions:\n\nWe extend our thanks for raising the pertinent concern regarding the inclusion of a comprehensive illustration of the ethics sheet. This is a very relevant question, and consequently, in our upcoming work, we plan to conduct an in-depth analysis of these cases using established sentiment analysis models, thereby furnishing a platform to exhibit our empirical findings. We acknowledge your suggestion and will incorporate a sample of the ethics sheet use case in our Appendix. Thank you for sharing a relevant reference in this context, which will enrich the quality and context of our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Overall, this study's suggestions and ethics sheet will (or should) have a long-lasting impact. The study provides important insights to the community.\n", "reasons_to_reject": "- I'm not 100% sure if this study is a good fit for the \"empirical method\" of NLP conferences, but this is just nitpicking.", "questions_to_authors": "It would be nice to provide a sample-filled example of the ethics sheet for a sentiment analysis study, like the \"Show Your Work\" paper [1]? This will help people understand how to implement your ethics sheet in practive.\n\n[1] [Show Your Work: Improved Reporting of Experimental Results](https://aclanthology.org/D19-1224) (Dodge et al., EMNLP-IJCNLP 2019)", "paper_topic_and_main_contributions": "This position paper explores the social impact of sentiment analysis, one of the most prevalent NLP applications in the world (perhaps before ChatGPT). The study conducts a critical survey and categorization of existing \"Sentiment Analysis\" research and develops an Ethics Sheet for sentiment analysis to promote a safer and more transparent development of sentiment analysis systems.\n", "excitement": 4, "reproducibility": null, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "T1ABw57hyc", "reviewer": "Reviewer_HzjA", "summary": "", "questions": "1. Like any survey paper, I think it’s worth mentioning the exact SCOPUS search string used for the study. This should also be included in the main sections of the paper. I tried using \"TITLE-ABS-KEY ( \"sentiment analysis\" ) AND PUBYEAR > 2010” as mentioned in the Appendix, but this returns 29k hits. What additional parameters did the authors use to filter this? This is for SCOPUS, information on how documents were retrieved should also be added in the case of Semantic Scholar. \n\n2. What are the authors’ inferences if terminologies used by research on sentiment analysis are often interchanged on paper but mean one particular framework (ex. polarity)? Is there harm in such a case? \n\n3. What is the resolve for published works encompassing multiple applications by using multiple datasets in sentiment analysis? Do they count separately to different items in Figure 1?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for taking the time to review our work and for sharing your valuable insights. We appreciate and mirror your commitment to championing the publication of such works in conferences like EMNLP, as well as your recognition of the necessity for rigorous scrutiny in the NLP domain, particularly considering its growing societal implications.\n\nYour critique concerning the inclusion of multilingual aspects in sentiment analysis is well-founded. Our original intent was to curate peer-reviewed literature commonly found in the sentiment analysis domain, encompassing models, applications, survey papers, and frameworks. Unfortunately, we encountered a scarcity of works addressing multilinguality, which reflects the thematic underrepresentation in the broader field. Consequently, we plan to delve deeper into the prevalent themes within sentiment analysis research to address this gap and provide due attention to underrepresented areas. Your suggestion to integrate multilinguality into our checklist aligns with our objectives, and we commit to implementing these changes.\n\nRegarding your second concern regarding introducing the concepts of the ethics sheets to beginners, we appreciate the feedback. While we did mention and introduce various works on ethics sheets in the NLP domain at the opening of the \"Recommendations and Ethics Sheet in Creating a Sentiment Model\" section, we will further emphasize this in our introduction and motivation. \n\nWe understand the need to incorporate a figure illustrating the various frameworks of sentiment analysis. The decision not to include it in the current 8-page draft was driven by the need to allocate space to address other pertinent points. Rest assured, we will integrate this figure into the final 9-page version of the paper.\n\nIn response to your specific questions to the authors:\n\n1. We appreciate your inquiry regarding our search methodology. To clarify, our search did not exclusively focus on sentiment analysis alone but rather encompassed corresponding fields of published works in sentiment analysis, i.e., its applications, models, frameworks, and survey papers. Our selection criteria involved ensuring that the collected papers were published in peer-reviewed journals and conferences, followed by a manual evaluation if they fit into our categories. Each category contained its unique search inquiry and data source, which has been mentioned at the outset of each section. We have developed a comprehensive flowchart outlining our collection process for each of these categories, which we will include in the paper's Appendix. Additionally, we will make this process more explicit in our paper.\n\n2. Your concern about the interchangeability of sentiment-related terms in different research fields is well taken. We believe that this ambiguity can introduce confusion and then harm, especially given the diverse interpretations of these terms in various contexts. It is imperative to address this issue, as it can lead to misinterpretations of models and results. When applications of sentiment analysis are used in social platforms with social actors/entities, terms such as opinion, polarity, and subjectivity can be interpreted differently, leading to the misevaluation of human behaviors. This can lead to wrongful inferences of the population due to the disciplinary divide of shared vocabulary. We emphasize this point in our paper, particularly in the \"Weaknesses in Sentiment Analysis\" section. Additionally, we emphasize in Section 2 that sentiment-related terms do not have a common definition in the field of sentiment analysis. There is, therefore, a need for interdisciplinary research to develop standardized frameworks. \n\n3. We acknowledge your observation regarding the potential presence of works that overlap applications across multiple fields. We did not encounter such papers in our literature sample of sentiment analysis application papers. Our focus was on papers that tackled sentiment analysis by the type of work they are situated in and the theme of works that are commonly published in the field. While there are indeed papers that develop sentiment analysis models for general use, they were categorized as model papers in our collection. The application-themed papers were works that used sentiment analysis as a solution for service in that particular field. We will ensure this distinction is clarified in our paper. You can see the categorization of these works in our supplementary material as well. This will be published publicly, along with the paper, for access to our literature collection.\n\nOnce again, we thank you for your positive input and encouragement for our work in this conference.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is well-written. I appreciate how readable the discussions are and how well-structured the flow of the narrative of the paper is, which makes it easier to evaluate and critique. Likewise, I do favor studies like these to be publicized and talked about in conferences as it gives a wider perspective and sanity check on the field of sentiment analysis for experts and beginner researchers alike. An ethics sheet, like the NLP checklist for most *ACL conferences that I personally find useful, will definitely help researchers ground the context and limitations of (future) studies using this, thus reducing vagueness and lack of transparency in applications of sentiment analysis.", "reasons_to_reject": "I have no major issues to raise to possibly merit a rejection score for the paper. However, I do have some reservations and concerns regarding how the paper is angled. I advise the author/s to consider these points for the improvement of the wholeness of the idea and contribution of the study:\n\n1. While reading the paper, I expected that the authors would thoroughly discuss language coverage and its related concepts (ex., code-switching, multilinguality, these are all qualified challenges in SA) and the scope of the critical survey regarding this factor. However, I’m quite disappointed that this was not mentioned at all. I don’t think this should be pushed back in the limitations section and claim that the survey is not exhaustive. This should be discussed outright in the collection phase and further expanded in the limitation section. Information on the language intricacies maybe even included as part of the checklist.\n\n2. The notion of proposing an ethics checklist for a specific task implies to the reader that other tasks in NLP may also merit the construction of such a list as they all face different challenges and risks when used across disciplines such as education, government, etc. As an NLP/SA researcher, I would understand perfectly why this resource is needed, however, for beginners and non-technical readers, they might not appreciate the notion of a checklist outright if this is only done in SA (excluding checklists for general AI experiments and datasets). Thus, as part of the motivation of the study, discussing that NLP tasks are doing sanity checks via an ethics checklist is important. I believe this would give more context and would further emphasize the need for an ethics checklist for sentiment analysis.\n\n3. I believe the study needs a good Figure 1. Upon reading the technical and social perception of sentiment analysis, confusion may arise with the terminologies (and their overlapping meaning) and their relation to one another. Readers may require some visual aid to connect the dots and terminologies to entertain thoughts such as “If I have an opinion on something, then that means I may have strong feelings or emotions about this something. I can express this feeling through texts with emoticons” So this does not necessarily mean that opinions and emotions are distinct concepts in SA, as the authors may have framed the paper with definitions from multiple works.", "questions_to_authors": "1. Like any survey paper, I think it’s worth mentioning the exact SCOPUS search string used for the study. This should also be included in the main sections of the paper. I tried using \"TITLE-ABS-KEY ( \"sentiment analysis\" ) AND PUBYEAR > 2010” as mentioned in the Appendix, but this returns 29k hits. What additional parameters did the authors use to filter this? This is for SCOPUS, information on how documents were retrieved should also be added in the case of Semantic Scholar. \n\n2. What are the authors’ inferences if terminologies used by research on sentiment analysis are often interchanged on paper but mean one particular framework (ex. polarity)? Is there harm in such a case? \n\n3. What is the resolve for published works encompassing multiple applications by using multiple datasets in sentiment analysis? Do they count separately to different items in Figure 1?", "paper_topic_and_main_contributions": "The paper describes work on surveying a large body of published sentiment analysis papers in literature to provide a sociotechnical perspective on recurring weaknesses and challenges present in the field. The author/s’ emphasize the motivation of the study by highlighting that the progress of sentiment analysis over the years cuts across beyond computing areas and into interdisciplinary sections such as finance, medicine, government, etc. One of the main challenges the survey highlights is the lack of concrete definitions and characterization of what a ‘sentiment’ is and how this differs from concepts such as opinion mining, emotion recognition, and semantic orientation, to name a few. The study is split into different analyses covering aspects of various frameworks assumed by studies for sentiment analysis, (technical/social) perceptions of sentiment tackled, mentioned applications of sentiment analysis across different fields, demonstration of existing biases in current sentiment analysis tools, and existing challenges still faced by these systems including lack of standardization, limited context, and generalizability. The study then proposes an ethics checklist that can be used by future sentiment analysis studies to possibly alleviate some of the mentioned challenges, such as lack of definition and mentions of target users of the system.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "UBVfpG0WKv", "reviewer": "Reviewer_8ryf", "summary": "", "questions": "(A) In part 2 did you analyse 43 or 60 models because the figures given are inconsistent?\n(B) To clarify, were all the models that you analysed published in peer-reviewed articles, including those without source code or made available only behind a paywall. \n(C) Please clarify where the 189 papers mentioned in line 704 come from. Is this just all the papers you cite, papers found in your two systematic literature reviews, papers analysed across Studies 1-3?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We extend our sincere gratitude for your feedback. We truly appreciate the time and effort you dedicated to this evaluation, and we are fully committed to addressing the points you have raised. Your recognition of the significance of works that shed light on the complexities within the field of sentiment analysis resonates with our own mission, and we are grateful for your acknowledgment of our efforts in this regard.\n\nTo respond to your insights regarding the reviews, we wish to express our thanks for sharing references that illuminate the need for a clearer distinction between the concept of sentiment and the framework of sentiment analysis employed in NLP. The categorization of these aspects as conceptualization versus operationalization is a valuable perspective, and we are eager to incorporate this clarity into the introduction of our paper in the final draft.\n\nRegarding your second point, we wish to clarify that the ambiguity in distinguishing between various sentiment analysis frameworks simply reflects the existing ambiguity in the field. There is a need for future interdisciplinary research in order to develop standardized frameworks for sentiment analysis. Having said that, it is important to understand (as discussed in our work) that the diversified set of definitions that emerge from different fields leads to complexities in the standardization of frameworks. We attempt to underscore how existing frameworks often overlap and lack well-defined boundaries, resulting in a somewhat fluid landscape. These ad-hoc creations can pose challenges when applied beyond the confines of sentiment analysis. Your suggestion to include examples to emphasize this point is well-received, and we commit to incorporating such illustrations into the paper.\n\nIn response to your specific queries:\n\nA. Study 2 consists of 43 papers focused on sentiment analysis models and 19 papers focused on sentiment analysis datasets, totaling 62 papers. We acknowledge the need to edit this detail within the section.\n\nB. Yes, all the papers we collected for our study were published in peer-reviewed journals or conference proceedings. We have included the comprehensive collection of these papers in the Appendix and supplementary material. Furthermore, we intend to make this collection publicly accessible alongside the final version of our paper.\n\nC. We appreciate your inquiry regarding the sources underpinning our analysis of the weaknesses in the field of sentiment analysis, as encapsulated in our paper and the three individual studies presented. To clarify, we conducted multiple literature reviews, each aligned with specific facets of sentiment analysis i.e., Sentiment Analysis Applications, Sentiment Analysis Models, Sentiment Analysis Datasets, Surveys and Meta-Analysis, and Frameworks (as shown in Table 1). These sources have been meticulously cited throughout our work and are reclassified in the Appendix for reference. We will ensure that this point is articulated more explicitly in our paper to enhance its clarity.\n\nOnce again, we express our appreciation for your positive feedback and recognition of our work. Your insights as well as the additional references have been invaluable in shaping the refinement of our paper, and we look forward to incorporating these enhancements in our final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This is a confidently written paper which highlights serious murkiness in the way sentiment analysis is currently conceptualised and conducted. The authors succinctly summarise the NLP and social science literature to demonstrate where lack of shared understanding, or even clear definitions of key concepts leads to issues related to validity, comparability and usefulness of measures. \n\nThe authors provide useful guidance for practitioners - I think the recommendations drawing attention to the impact of use cases, and cultural or contextual variations in sentiment interpretation, are particularly valuable for the community.", "reasons_to_reject": "I found the definition of “framework” provided to be slightly too broad to be useful. For example, the distinction between providing a definition of sentiment, and defining the sentiment framework (which they say encompasses “a set of principles… for… defining a specific concept”), is unclear. I believe the distinction may be clearer if cast in the light of the measurement modelling framework which distinguishes conceptualisation and operationalisation of a concept such as sentiment (see Adcock and Collier, 2001; Jacobs and Wallach, 2021).\n\nI also found the differences between the different technical frameworks for sentiment analysis difficult to distinguish, particularly “object’s orientation” vs “opinions or evaluation”, although the issue of conflicting definitions still comes across. I recommend the authors expand on why adopting one sentiment framework to a new domain “may not suffice” (L314), with concrete examples. \n\nI found the limitations section to be superficial: it was more a statement of the fact the work is not exhaustive (what work is?) rather than an engagement with i.e. which areas the work fails to encompass, how the authors’ preconceptions may have limited the work etc", "questions_to_authors": "(A) In part 2 did you analyse 43 or 60 models because the figures given are inconsistent?\n(B) To clarify, were all the models that you analysed published in peer-reviewed articles, including those without source code or made available only behind a paywall. \n(C) Please clarify where the 189 papers mentioned in line 704 come from. Is this just all the papers you cite, papers found in your two systematic literature reviews, papers analysed across Studies 1-3?", "paper_topic_and_main_contributions": "In this paper the authors highlight current issues with the way sentiment is conceptualised and measured in papers on “sentiment analysis” (SA). They demonstrate conflict both within the NLP literature, and between papers on SA and the conceptualisation of sentiment (and the related concepts of emotions, opinions and subjectivity) in the social science literature. They demonstrate that only half of papers applying SA, and less than half of papers proposing a model and dataset, provide a clear indication of which framework of SA they use, and only a handful actually define sentiment. The authors highlight issues of bias in existing SA models. They summarise the key weaknesses of SA - in addition to conflicting or missing definitions, there are the issues of subjective interpretation; lack of nuance amongst others. The authors then propose 4 recommendations and 10 related questions which can guide practitioners proposing new SA artefacts to be theoretically grounded and robust, and provide sufficient detail.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "L63 - missing parenthesis\nL185 - commonly linked to sentiment in the technical literature (?)\nL198 - a → as\nL351 - Q1- → Q1:\nL438 - superfluous punctuation\nL568 - repetition of tone"}, "extra_text": {"Missing_References": "Measurement modelling framework references given above. \n\nThe authors may want to consider referencing some of the work on aspect-based sentiment analysis. \n\nPaper on bias in proprietary sentiment analysis models against LGBTQ+ community - Ungless, E. L., Ross, B., & Belle, V. (2023). Potential Pitfalls With Automatic Sentiment Analysis: The Example of Queerphobic Bias."}}]

Author rebuttal:

Meta-review: This work provides a survey of papers on sentiment analysis and provides a sociotechnical perspective on the area, discussing the issues and challenges. These challenges include the lack of a concrete definition of “sentiment analysis” used within NLP and across NLP and social science, challenges in existing sentiment analysis models, such as biases and limited generalizability among others. Lastly, it provides four pieces of advice and 10 related questions to mitigate the challenges it discusses.

Reviewers appreciated that this work provides a succinct summary of the task of sentiment analysis, a careful discussion of its issues, and a practical guideline for (at least partially) mitigating the issues. The presentation is also clear, which is especially important for this type of paper. 

The reviewers do not seem to have major concerns but noted that a main figure presenting an overview of the survey would be helpful, some terms like “framework” can be better clarified, and the limitations section can provide a deeper discussion, among others.
Decision: Accept-Main