# Weighted Joint Sentiment-Topic Model for Sentiment Analysis Compared to ALGA: Adaptive Lexicon Learning Using Genetic Algorithm

Authors: Amjad Osmani; Jamshid Bagherzadeh Mohasefi
Journal: Computational Intelligence and Neuroscience
 (2022)

---

## Abstract

Latent Dirichlet Allocation (LDA) is an approach to unsupervised learning that aims to investigate the semantics among words in a document as well as the influence of a subject on a word. As an LDA-based model, Joint Sentiment-Topic (JST) examines the impact of topics and emotions on words. The emotion parameter is insufficient, and additional parameters may play valuable roles in achieving better performance. In this study, two new topic models, Weighted Joint Sentiment-Topic (WJST) and Weighted Joint Sentiment-Topic 1 (WJST1), have been presented to extend and improve JST through two new parameters that can generate a sentiment dictionary. In the proposed methods, each word in a document affects its neighbors, and different words in the document may be affected simultaneously by several neighbor words. Therefore, proposed models consider the effect of words on each other, which, from our view, is an important factor and can increase the performance of baseline methods. Regarding evaluation results, the new parameters have an immense effect on model accuracy. While not requiring labeled data, the proposed methods are more accurate than discriminative models such as SVM and logistic regression in accordance with evaluation results. The proposed methods are simple with a low number of parameters. While providing a broad perception of connections between different words in documents of a single collection (single-domain) or multiple collections (multidomain), the proposed methods have prepared solutions for two different situations (single-domain and multidomain). WJST is suitable for multidomain datasets, and WJST1 is a version of WJST which is suitable for single-domain datasets. While being able to detect emotion at the level of the document, the proposed models improve the evaluation outcomes of the baseline approaches. Thirteen datasets with different sizes have been used in implementations. In this study, perplexity, opinion mining at the level of the document, and topic_coherency are employed for assessment. Also, a statistical test called Friedman test is used to check whether the results of the proposed models are statistically different from the results of other algorithms. As can be seen from results, the accuracy of proposed methods is above 80% for most of the datasets. WJST1 achieves the highest accuracy on Movie dataset with 97 percent, and WJST achieves the highest accuracy on Electronic dataset with 86 percent. The proposed models obtain better results compared to Adaptive Lexicon learning using Genetic Algorithm (ALGA), which employs an evolutionary approach to make an emotion dictionary. Results show that the proposed methods perform better with different topic number settings, especially for WJST1 with 97% accuracy at |Z| = 5 on the Movie dataset.


---

## Body

## 1. Introduction

Opinion extraction is one of the main branches of natural language processing (NLP) research. Comment extraction (emotion analysis) now is widely used in websites containing different types of merchandise. Online product reviews can help customers buy a product and help manufacturers discover new opportunities by analyzing user feedback. Consequently, automated analysis of reviews is critical. Emotion Analyzer can browse comments on the web and categorize many comments as positive or negative tags. This research is important because it makes managing customer requests easier and more efficient because product owners automatically extract customer feedback and use customer feedback to sell products. There are different methods for extracting opinions and analyzing them, and in this research, an intelligent method has been used [1–7]. Topic modeling presumes that the input text document set contains several unknown subjects that need recognition. Each subject (topic) is an unknown distribution of words, and each review (text document) is a distribution of subjects. The aim is to detect concealed knowledge in textual data related to the user’s comments. Several methods perform subject modelings, such as Latent Dirichlet Allocation (LDA) and Probabilistic Latent Semantics Analysis (PLSA). PLSA is a method that can produce the data perceived in a document-term matrix. LDA is a probabilistic method because it is exhibited in a probabilistic language, and it is a generative model because it is about ensuring that documents are produced. LDA has based on the premise that a review is a combination of subjects in which each topic is distributed over words. The linear growth of PLSA parameters indicates that the method is prone to overfitting. LDA can be easily extended to new documents. In addition, increasing the training data size does not lead to the growth of LDA-related parameters [7].In LDA, subjects are related to documents, and words are related to subjects. To model the emotion of reviews, Joint Sentiment-Topic (JST) [8] establishes an extra layer of emotion between the layers of document and subject, where the emotion labels are related to the documents, the subjects are related to the emotion labels, and words are tagged with emotions and related topics. This study assumes that each word in a document affects its neighbors, and different words in the document may be affected simultaneously by several neighbor words. Thus, the proposed models consider the effect of words on each other. The proposed models add two parameters (weight and window) to JST. The window parameter represents the range of the effect of a word, and the weight parameter represents the strength of the effect of the word. These two parameters play an important role in better classification, as seen in the evaluation section. Using the parameters weight and window, two new methods are introduced that have revealed notable dominance over the baseline algorithms, such as JST, Topic Sentiment modeling (TS) [9], Reverse-JST (RJST) [10], and Tying-JST model (TJST) [8].More and more improved algorithms and strategies are used to solve sentiment analysis problems. However, none of the researchers have improved the accuracy besides generating a sentiment dictionary. Different from other related studies, in this study, the proposed models improve topic-model-based sentiment classification using two parameters (weight and window). The proposed models consider the effect of words on each other. They can also generate a sentiment dictionary that includes words and scores that specify positive and negative labels and their weight. Accuracy is calculated using two formulas. Finally, by evaluating the proposed methods and the comparison with other algorithms on thirteen datasets of different sizes, the results show that the algorithms presented in this study are superior to the compared algorithms in terms of accuracy, perplexity, and topic_coherency.The rest of this article is arranged as follows: Section2 shows a summarized overview of previous works in emotion analysis and the use of topic modeling in emotion analysis. The proposed models are provided in Section 3. The evaluation results are discussed in Section 4, and Section 5 concludes this article.


## 2. Related Works

The value of emotion analysis may be highlighted by analyzing customer happiness from online services like email. It is also feasible to employ emotion mining to evaluate the opinions of various people in order to make them aware of things that have favorable reviews. Major types of classification in emotion analysis are document, sentence, and aspect. An opinion is a quadrilateral (g, s, h, t), where g is the target, s is sentiment, h is the author’s opinion, and t is the opinion expression time [11, 12, 13]. Many attempts have been made to detect emotions and explore the knowledge embedded in text data. Topic modeling obtains concealed subjects of documents. In topic modeling, the aim is to discover the best set of hidden variables that can express the observed data. LDA has been used as a topic model to effectively explore subjects in the documents [7]. LDA has motivated countless algorithms to expand to solve different problems [14–17]. In [18], the authors exhibit three topic models which make better LDA using date, helpfulness, and subtopic parameters. Articles [8, 10, 19] describe the methodology JST. This model expands LDA using a sentiment layer. This method cannot accurately identify the different emotions and is used as a baseline method in most articles. Several methods are similar to JST [8, 10, 20]). The aspect and Sentiment Unification Model (ASUM) [20] is similar to JST. JST assumes that each word represents an aspect, but ASUM assumes that each sentence represents a description of an aspect. A variation of the JST model is TJST [8]. The main difference between JST and TJST is that to sample a word in a document during the generative process of documents, JST selects a subject-document distribution for each document, whereas TJST uses one subject-document distribution for all documents. According to [10], the emotion influences the subject in JST, whereas in RJST, the subject influences the emotion. According to [9], there is only one topic-sentiment distribution for all documents in the TS, while there is a distribution for each document in RJST.Several methods have been introduced for text emotion analysis that uses topic modeling [21–23, 78]. In [24], the authors introduce an algorithm that creates a review containing both shared subjects and subjects distributed over words as special data. Two topic models are proposed in [79]: Multilabel Supervised Topic Model (MSTM) and Sentiment Latent Topic Model (SLTM). Both methods could be used to categorize social emotions. In [25], the authors introduce a Sentiment Enriched and Latent-Dirichlet-Allocation-based review rating Prediction (SELDAP) to predict ratings using topics and sentiments of reviews. In [26], the authors introduce a method named Hierarchical Clinical Embeddings combined with Topic modeling (HCET), which can integrate five types of Electronic Health Record (EHR) data over several visits to predict depression. The authors of [80] presented the word Sense aware LDA (SLDA) approach that uses word sense in topic formation. In [27], the authors introduce a survey of different short text topic modeling methods. They provide a detailed analysis of algorithms and discuss their performance. The authors proposed a segment-level joint topic-sentiment model (STSM) in [81], where each sentence is divided into parts by conjunctions, and the assumption that all terms in a section convey the same emotion is presented. In [28], the authors provided a thorough examination of subject modeling methods.Deep learning provides an approach to utilizing large volumes of calculation and data using little manual engineering. Recently, deep learning approaches to analyzing emotions have reached a considerable triumph [29, 30, 47, 77]. Optimization methods have developed significantly in recent years [31–37]. Optimization methods are widely used in the feature section, notably for text. In [38], the authors proposed a multiobjective-grey wolf-optimization algorithm to categorize sentiments. In [39], the authors proposed a binary grey wolf optimizer method to classify labels in the text. In the following article [40], the authors introduced a new optimization method that mimics the model of a successful person in society. Their article used this method to categorize emotions, which achieved very good results. There are several works on using user behavior for sentiment analysis. Tag sentiment aspect (TSA) framework, a new probabilistic generative topic framework, was presented by [48] with three implementation editions. TSA is on the basis of LDA. In [41], the authors concentrate on user-based methods on social networks, where users create text data to show their views on different topics and make connections with other users to create a social network. In [42], the authors used a signed social network to detect the emotions of reviews as an unsupervised approach. Various works use other techniques for sentiment analysis problems [43–45]. In Adaptive Lexicon learning using a Genetic Algorithm (ALGA) [46], some emotion dictionaries for a dataset in the training stage are constructed using the genetic method. These sets are utilized in the testing stage. Each lexicon comprises both words and their scores. A chromosome is modeled as a vector of emotional words and scores in the genetic approach. Scores are in the range of (the lowest score of an emotional word, the highest score of an emotional word). The main goal of ALGA is to create a lexicon that minimizes the error in the training stage.In [47], the authors proposed a deep learning-based topic-level opinion mining method. The approach is novel in that it works at thelevel of the sentence to explore the subject using online latent semanticindexing and then employs a subject-level attention method in an extendedshort-term memory network to detect emotion. In [62], the authors proposed a joint aspect-based sentiment topic model that extracts multigrained aspects and emotions. In [49], parts-of-speech (POS) tagging is performed via a hidden Markov model, and unigrams, bigrams, and bi-tagged features are extracted. Also, the nonparametric hierarchical Dirichlet process is employed to extract the joint sentiment-topic features. In [50], the authors used an unsupervised machine learning method to extract emotion at the document and word levels. In [51], the authors proposed a new framework for joint sentiment-topic modeling based on the Restricted Boltzmann Machine (RBM), a type of neural network. In [52], the authors proposed a probabilistic method to incorporate textual reviews and overall ratings, considering their natural connection for a joint sentiment-topic prediction. In [53], the authors proposed a hybrid topic model-based method for aspect extraction and emotion categorization of reviews. LDA is used for aspect extraction and two-layer bidirectional long short-term memory for emotion categorization. In [54], the authors proposed a joint sentiment-topic model that uses Markov Random Field Regularizer and can extract more coherent and diverse topics from short texts. In [55], the authors proposed a topic model with a new document-level latent sentiment variable for each topic, which moderates the word frequency within a topic. In [56], the authors proposed a new method for text emotion detection, aiming to improve the LSTM network by integrating emotional intelligence and attention mechanism. In [57], the authors proposed a new model for aspect-based emotion detection. The model is a novel adaptation of the LDA algorithm for product aspect extraction.In [58], the authors introduced a new deep learning-based algorithm for emotion detection, using available ratings as weak supervision signals. In [59], the authors introduced a new deep learning-based algorithm for emotion detection, using two hidden layers. The first layer learns sentence vectors to represent the semantics of sentences, and in the second layer, the relations of sentences are encoded. In [60], the authors introduced a transformer-based model for emotion detection that encodes representation from a transformer and applies deep embedding to improve the quality of tweets. In [61], the authors introduced an attention-based deep method using two independent layers. By having to consider temporal information flow in two directions, it will retrieve both past and future contexts.In this study, the proposed methods have tried to increase the accuracy with fewer parameters and, at the same time, simplicity compared to the existing methods. The proposed methods analyze emotions at the document-level and create an emotional dictionary. They are also the first methods that create an emotional dictionary through a topic modeling technique automatically and accurately. The proposed methods are the first methods that consider the words in the text and their effect on each other in a dynamic and weighty way.Table1 compares a number of articles presented in recent years in emotion analysis in terms of method, language, and dataset. In the method column, as can be seen, the combination of topic modeling and deep learning methods has recently been considered. In the language column, it is specified in which language the proposed method has been tested. The name of the dataset that has been tested can also be seen in the dataset column.Table 1
A general comparison of similar methods in recent years.
ReferencesMethodLanguageDatasetGeneral resultPathak et al. [47]Deep learning + topic modelingEnglishFacebook, Ethereum, Bitcoin, SemEval-2017Facebook-0.79, Ethereum-0.844, Bitcoin-0.817, SemEval-2017-0.889Tang et al.[62]Topic modelingEnglishAmazon, YelpAmazon-0.82, Yelp-0.84Kalarani and Selva Brunda [49]Joint sentiment-topic features + POS tagging + SVM and ANNEnglishBalanced dataset, unbalanced dataSVM-0.84, ANN-0.87Farkhod et al. [50]Topic modelingEnglishIMDBIMDB-F1 score-70.0Fatemi and Safayani [51]Topic modeling + restricted Boltzmann machineEnglish20-Newsgroups (20NG), movie review (MR), multidomain sentiment (MDS)Perplexity: MR: 406.74Pathik and Shukla [53]Deep learning + topic modelingEnglishYelp, Amazon, IMDBYelp- 0.75, Amazon-0.76, IMDB- 0.82Sengupta et al.[54]Topic modelingEnglishMovies, TwitterPerplexity: Movies- 3834.7, Twitter- 280.75Huang et al.[56]Deep learningEnglishIMDB, YelpIMDB-0.963, Yelp-0.735Özyurt and Akcayol [57]Topic modelingEnglish + TurkishUser reviews in Turkish language about smartphones, SemEval-2016, Task-5 Turkish restaurant reviewsPrecision-81.36Recall-83.43F-score-82.39Zhao et al. [58]Deep learningEnglishAmazon reviewCNN-87.7, LSTM-87.9Rao et al. [59]Deep learningEnglishYelp 2014, 2015, IMDbYelp2014–63.9 Yelp2015–63.8, IMDb-44.3Naseem et al. [60]Deep learningEnglishAirline datasetAirline dataset = 0.95Basiri et al. [61]Attention-based deep learningEnglishSentiment140, Airline, Kindle dataset, movie reviewKindle dataset = 0.93, Airline = 0.92, movie review = 0.90, Sentiment140 = 0.81The proposed models are deeply described step-by-step in the next section.


## 3. Proposed Models

This study proposes two novel topic sentiment models called Weighted Joint Sentiment-Topic (WJST) and Weighted Joint Sentiment-Topic 1 (WJST1). The proposed models improve JST using two extra parameters (weight and window).According to Figure1, the data type of the dataset is text. Preprocessing is performed by lowercasing all words, removing the stop words and words with too low and too high frequency, stemming, removing digits, and removing nonalphabetic characters such as (#, ! …). Proposed models can be summarized as follows: (1) in the Generative Model part, the procedure of generating a word in a document under a topic model is illustrated. (2) In the Plate Notation part, a graphical representation of the subject model is provided (in the style of plate notation). (3) In the Model Inference part, Gibbs sampling will be used (to fulfill approximation inference). In the Evaluation phase, the model’s performance is evaluated using accuracy, perplexity, and topic_coherency.Figure 1
The framework chart of the proposed methods.


### 3.1. Motivation

The proposed models add two parameters to JST as latent variables in this study. From our view, it is assumed that the words in the documents affect their neighbors, and different words in the document may be affected simultaneously by several neighbor words. For example, in the sentence “My phone has a small memory, and its pictures quality is low,” the unigramsmall affects the unigram memory, and the bigram small memory affects the unigrams phone and pictures. So, unigram small affects unigrams memory, phone, and picture.According to Figure2, the reviews as input text data types are used for sentiment classification. The proposed models consider the effect of words on each other. They adopt Gibbs sampling to perform approximate inference of distributions. After completing the sampling in the Gibbs sampling algorithm, latent variables’ distribution can be calculated. Sentiment classification at the document-level is calculated based on the probability of a sentiment label given to a document.Figure 2
The general architecture of the proposed methods.Like the above example, a word can affect neighbor words in many sentences. So, in the proposed models, we consider the effect of words on each other using two parameters. The window parameter represents the range of the impact of a word, and the weight parameter represents the strength of the effect of the word. In the proposed models, each word has a weight, a sentiment label, and a topic and affects its neighbors as much as its window size, which means that each word has a window. For instance, as can be seen in Figure3, word w3 has the window size equal to 1 and affects words w2 and w4, and w6 has the window size equal to 2 and affects words w4, w5,w7,andw8. If word w3 had weight h and negative sentiment, words w2 and w4 would have weight h and negative sentiment as well. Each word is affected by its neighbors. So, different words in a document may be affected simultaneously by several neighbor words.Figure 3
An example of a sentence with different windows.


### 3.2. The Problem Statement

In this study, given a corpus of|R| documents, R=r1,r2,r3,…,rR, a document, r, consists of w1,w2,w3,…,wNr words, and each word belongs to a vocabulary set with a |V| distinct element. Furthermore, |Q| is the number of separate windows, |E| is the number of distinct weights, |S| is the number of distinct sentiment labels, and |Z| is the number of distinct topics. In the present study, five sets θ,φ,π,ψ,andξ require to be inferred which are latent variables. The hyperparameters α,β,γ,δ,andμ are given based on the experience, which can be the prior observation counts before observing any actual words, where α is Dirichlet prior distribution for θ, β is Dirichlet prior distribution for φ, γ is Dirichlet prior distribution for π, δ is Dirichlet prior distribution for ψ, and μ is Dirichlet prior distribution for ξ. The latent parameters z,s,q,e,φ,θ,π,ξ,andψ require to be approximated using observed variables, where z is topic variable, s is sentiment variable, q is window variable, and e is weight variable. The proposed models demonstrate the process of generating words in documents. Furthermore, they can approximate the latent variables. In the present study, the main aim of the proposed topic models is to categorize sentiments at the document-level.


#### 3.2.1. The Problem We Are Trying to Solve or Improve

Analyzing user satisfaction with various services, products, or movies is the main problem in this study, mainly reflected in users’ comments. A user’s comment is formed by a message as text on the Internet which can be a tweet or a simple message on a website. So, for example, it is feasible to employ emotion mining to evaluate the opinions of various people in order to make them aware of things that have favorable reviews.


#### 3.2.2. The Solution to the Problem

Many attempts have been made to detect emotions and explore the knowledge embedded in text data. Topic modeling as a known method can obtain concealed subjects of documents. LDA has been used as a topic model to effectively explore issues in the documents. As an LDA-based model, JST examines the impact of topics and emotions on words. The emotion parameter is insufficient, and additional parameters may play valuable roles in achieving better performance.This study presents two new topic models that extend and improve JST through two new parameters and generate a sentiment dictionary. The proposed models consider the effect of words on each other, which, from our view, is an important factor and can increase the performance of baseline methods. Several methods have been introduced for text emotion analysis that uses topic modeling. However, none of the researchers have improved the accuracy besides generating a sentiment dictionary. Different from other related studies, in this study, the proposed models improve topic-model-based sentiment classification using two parameters (weight and window). The proposed models are deeply described step-by-step in the following sections.


### 3.3. The General Structure of WJST

This subsection introduces a new model named WJST, which improves JST using two parameters (weight and window). The primary goal of WJST is to classify sentiments at the document-level. A summary of symbols applied in the model is prepared in Table2. The process of generating a word of a document in WJST can be outlined as follows: (1) for each document, an author first decides the distribution of sentiments. For example, sentiments are 70% positive and 30% negative, so the proposed model chooses a sentiment label from the per-document sentiment distribution. (2) After determining the sentiment label, the author writes a review about a product according to the distribution of topics. For example, topics are 70% about memory, 20% about speed, and 10% about battery, so WJST chooses a topic from the per-document topic distribution that depends on the sentiment label. (3) After determining the sentiment label and the topic, the author decides the distribution of weights and the distribution of windows. WJST then chooses a weight from the per-document weight distribution that depends on the sentiment label and topic. WJST chooses a window size from the per-document window distribution that depends on the topic. (4) Finally, the author chooses some words to express an opinion under the identified topic, sentiment label, weight, and window. So, WJST draws a word from the per-corpus word distribution that depends on the topic, sentiment label, weight, and window. The words with different topics may have different window sizes. For example, a word with topic memory has a smaller window size than a word with topic mobile because topic mobile is more general than topic memory which can cover topic memory. So, the topic affects window size.Table 2
A summary of notations used in WJST.
SymbolDescriptionCollectionsRSet of all documentsVVocabulary setQSet of all distinct windows (with different sizes)ZSet of all topicsESet of all distinct weightsSSet of all sentiment labelsInit parametersqWindow variableeWeight variablerDocument variablezTopic variablewWord variablesSentiment variableDistributionsθProbability ofz given s and rφProbability ofw given z, s, q, and eπProbability ofs given r (s0 = Positive label, s1 = Negative label)ψProbability ofe given z, s, and rξProbability ofq given z and rHyper parametersαDirichlet prior distribution forθβDirichlet prior distribution forφγDirichlet prior distribution forπδDirichlet prior distribution forψμDirichlet prior distribution forξThe words with different topics may have different weights. For example, wordsize in topic memory is significant and considerable weight because all customers like memories with larger capacity. Word size in topic mobile is not important as word size in topic memory, and it has a small weight in topic mobile because some customers may like mobile phones with small size (iPhone 6s), and some customers may like the mobile phones with large size (iPhone 6s+). So, the topic affects weight. The words with different sentiment labels may have different weights. For example, suppose that topic memory contains two words size and cost. If the word size is positive, positive size will be more important than the word cost, and its weight will be larger than the cost. If word size is negative, the positive cost will be more important than the word size, and its weight will be larger than the size. Positive size means using words like large and big because customers like memories with larger capacity sizes. Negative size means using words like small because costumers do not like memories with smaller capacity size. Positive cost means using words like low and cheap because costumers like low-priced memories. Negative cost means using words like high and expensive because costumers do not like high-priced memories. So, sentiment label affects weight. The proposed model is parametric in this study [63]. Furthermore, the number of topics is constant. The generative model of WJST is demonstrated in Figure 4.Figure 4
The formal definition of the process of generating words in WJST.The symbols ofMulti and Dir demonstrate distributions of Multinomial and Dirichlet, respectively. Five sets of latent variables θ,φ,π,ψ,andξ require to be inferred which are latent variables. The hyperparameters α,β,γ,δ,andμ are given based on the experience, which can be the prior observation counts before observing any actual words. The latent parameters z,s,q,e,θ,φ,π,ψ,andξ require to be approximated using observed variables. The plate notation of WJST is exhibited in Figure 5. The plate notation is a method for expressing variables repeating in a graphical model. Furthermore, a probabilistic model shows the conditional dependency layout among the random variables as a graph.Figure 5
The plate notation of WJST.According to Figure5, the joint probability distributions for the model WJST can be factored as follows:(1)Pw,z,s,q,e=Pw|z,s,q,e×Pz|s,r×Ps|r×Pq|z,r×Pe|z,s,r,where by integrating out φ, we achieve:(2)Pw|z,s,q,e=ΓV×βΓβVZ×S×Q×E∏z∏s∏q∏e∏wΓNw,z,s,q,e+βΓNz,s,q,e+V×β,where |V| is the vocabulary size, |S| is the number of sentiment labels, |Z| is the number of topics, |Q| is the number of distinct windows, and |E| is the number of weights. The symbol Nw,z,s,q,e is the number of times the word w has been assigned to topic z, window q, weight e, and sentiment s. The symbol Nz,s,q,e is the number of words with topic z, window q, weight e, and sentiment s.The symbolβ is Dirichlet prior to φ. The symbolΓ is the gamma function. In addition, by integrating out θ, we achieve:(3)Pz|s,r=ΓZ×αΓαZS×R∏r∏s∏zΓNz,s,r+αΓNs,r+Z×α,where R is the number of documents and Nz,s,r is the number of words with topic z with sentiment s in document r. The symbol Ns,r is the number of words with sentiment s in document r. The symbol α is Dirichlet before θ. And by integrating out π, we achieve:(4)Ps|r=ΓS×γΓγSR∏r∏sΓFs,r+γΓFr+S×γ,where Fs,r is the effect of words with sentiment s in document r, which is equal to ∑w∈rew,s,r×1+2×qw,s,r where ew,s,r is the weight of word w with sentiment s in document r and qw,s,r is the window size of word w with sentiment s in document r. The symbol Fr is the sum of the effect of words with different sentiments (positive and negative) in document r, which is equal to ∑s∈positive,negativeFs,r.The symbolγ is Dirichlet before π. And by integrating out ξ, we achieve:(5)Pq|z,r=ΓQ×μΓμQZ×R∏r∏z∏qΓNq,z,r+μΓNz,r+Q×μ,where |Q| is the number of distinct windows. The symbol Nq,z,r is the number of words with topic z and window q in document r. The symbol Nz,r is the number of words with topic z in document r. The symbol μ is Dirichlet before ξ. And by integrating out ψ, we achieve:(6)Pe|z,s,r=ΓE|×δΓδEZ×S×R∏s∏r∏z∏eΓNe,z,s,r+δΓNz,s,r+E×δ,where |E| is the number of weights, Ne,z,s,r is the number of words with topic z, weight e, and sentiment s in document r. The symbol Nz,s,r is the number of words with sentiment s and topic z in document r. The symbolδ is Dirichlet before ψ. To estimate the parameters φ,θ,π,ξ,andψ, we need to evaluate the above distributions. These distributions are difficult to assess directly, so we adopt Gibbs sampling to perform approximate inference. Gibbs sampling is a widely used inference technique and is a popular approach for parameter estimation and inference in many topic models such as LDA [7]. The advantage of using the Gibbs sampling method is that it is simple and easy to implement. In this study, Gibbs sampling is used to estimate the distributions of the latent variables. The pseudocode of the Gibbs sampling algorithm is given in Figure 6 for the proposed model, and the meanings of all variables are seen in Table 2. The algorithm will sample each variable (z, s, q, and e) based on the following formula by canceling terms in equations (2)–(6) (by replacing terms in (1) with those in equations (2)–(6):(7)Pzr,i=z,sr,i=s,qr,i=q,er,i=e|zr,i,sr,i,qr,i,er,i,β,α,γ,μ,δ∝Nw,z,s,q,e+βNz,s,q,e+V×β×Nz,s,r+αNs,r+Z×α×Fs,r+γFr+S×γ×Nq,z,r+μNz,r+Q×μ×Ne,z,s,r+δNz,s,r+E×δ,where zr,i,sr,i,qr,i, and er,i are topic, sentiment, window, and weight assignments, respectively, for all the words in the collection, except for the word considered at position i in document r. Posterior inference of parameters is performed using Gibbs sampling, as demonstrated in Figure 6.Figure 6
Adopted Gibbs sampling for WJST1.In the section of initialization, the method randomly sets the parameters. A sentiment dictionary is employed for initializing sentiment labels. The sentiment dictionary contains words and scores that specify positive and negative labels and their weight. In this study, AFINN [64] is used as a sentiment dictionary, improving the model’s accuracy. At the end of the sampling algorithm, each word has a weight and a sentiment label. Therefore, a dictionary can generate sentiment scores (weights and sentiment labels) and words. The scores are extracted from a dataset based on P(w| s, e). Each word’s weight and sentiment with the most probability are selected as sentiment scores among all documents. Adaptive Lexicon learning using Genetic Algorithm (ALGA) [46] uses the genetic algorithm to generate a sentiment dictionary. However, we use topic modeling in WJST, to generate this dictionary. In WJST, the window size is different for various words. At each step of the sampling algorithm, count variables such as Fs,r and Fr are updated after sampling sentiment label, weight, and window size. After completing the sampling, the distribution of latent variables (φ,θ,π,ξ,andψ) can be calculated as follows:(8)φ=Nw,z,s,q,e+βNz,s,q,e+V×β,(9)θ=Nz,s,r+αNs,r+Z×α,(10)π=Fs,r+γFr+S×γ,(11)ξ=Nq,z,r+μNz,r+Q×μ,ψ=Ne,z,s,r+δNz,s,r+E×δ.The probability of a word given a topic would be equal to∑s,q,ePw|z,s,q,e, and the probability of a sentiment label given a document for sentiment classification at the document-level is calculated using π.The time complexity of the proposed method quantifies the amount of time taken by the Gibbs sampling algorithm to run as the main function. Given the number of words in all documentswALL (wALL=∑r∈RNr, where Nr is the number of words in document r), the number of topics |Z|, the number of distinct windows |Q|, the number of weights |E|, and the total number of sentiment labels |S|, the time complexity of each Gibbs sampling iteration would be O(wALL·|S|·|Z|·|Q|·|E|). Furthermore, given the number of iterations G, the total time complexity of WJST would be O(G·wALL·|S|·|Z|·|Q|·|E|). Table 3 compares different methods in terms of time complexity.Table 3
The time complexity of different models.
ModelTime complexityJST, RJST, TJST, and TSO(G·wALL·|S|·|Z|)WJSTO(G·wALL·|S|·|Z|·|Q|·|E|)


### 3.4. The General Structure of WJST1

A version of WJST called WJST1 is presented in Figure7. The distributions θ,ξ,andψ in WJST depend on the document, but in WJST1, the distributions θ,ξ,andψ do not rely on the document. Dependency between documents of a domain is more than documents in different domains. A pattern in documents of a domain may not exist in documents of other domains. So, calculations on multidomain datasets should be local and not cover all domains. For example, considering the distributions Pz|s and Pz|s,r, where z is topic, s is sentiment, and r documents, in the first state Pz|s, topic depends on sentiment. The distribution covers all documents in different domains. Perhaps a topic is positive in one domain and negative in another domain. So, it is better to depend the topic on the documents of a domain, not all domains. Thus, the topic is limited to the document (and domain), and contradiction between different domains is eliminated. So, WJST is suitable for multidomain datasets, and WJST1 is a version of WJST suitable for single-domain datasets. According to Figure 7, ξ is the probability of q given z, θ is the probability of z given s, and ψ is the probability of e given z and s, and the joint probability distribution for WJST1 can be factored as follows:(12)Pw,z,s,q,e=Pw|z,s,q,e×Pz|s×Ps|r×Pq|z×Pe|z,swhere by integrating out θ, we achieve:(13)Pz|s=ΓZ×αΓαZS∏s∏zΓNz,s+αΓNs+Z×α,whereNz,s is the number of words with topic z and sentiment s. The symbol Ns is the number of words with sentiment s. The symbol α is Dirichlet before θ. And by integrating out ξ, we achieve:(14)Pq|z=ΓQ×μΓμQZ∏z∏qΓNq,z+μΓNz+Q×μ,whereNq,z is the number of words with topic z and window q. The symbol Nz is the number of words with topic z. The symbol μ is Dirichlet before ξ. And by integrating out ψ, we achieve:(15)Pe|z,s=ΓE×δΓδEZ×S∏s∏z∏eΓNe,z,s+δΓNz,s+E|×δ,whereNe,z,s is the number of words with topic z, weight e, and sentiment s. The symbol Nz,s is the number of words with sentiment s and topic z. The symbol δ is Dirichlet before ψ. The symbols Pw|z,s,q,eandPs|r are calculated using equations (2) and (4), respectively. After completing the sampling, the distribution of latent variables (θ,ξ,andψ) is calculated as follows:(16)θ=Nz,s+αNs+Z×α,ξ=Nq,z+μNz+Q×μ,ψ=Ne,z,s+δNz,s+E×δ.Figure 7
The graphical model of WJST1.Andφandπ are computed through equations (8) and (10), respectively. Experimental results are demonstrated in the next section.


## 3.1. Motivation

The proposed models add two parameters to JST as latent variables in this study. From our view, it is assumed that the words in the documents affect their neighbors, and different words in the document may be affected simultaneously by several neighbor words. For example, in the sentence “My phone has a small memory, and its pictures quality is low,” the unigramsmall affects the unigram memory, and the bigram small memory affects the unigrams phone and pictures. So, unigram small affects unigrams memory, phone, and picture.According to Figure2, the reviews as input text data types are used for sentiment classification. The proposed models consider the effect of words on each other. They adopt Gibbs sampling to perform approximate inference of distributions. After completing the sampling in the Gibbs sampling algorithm, latent variables’ distribution can be calculated. Sentiment classification at the document-level is calculated based on the probability of a sentiment label given to a document.Figure 2
The general architecture of the proposed methods.Like the above example, a word can affect neighbor words in many sentences. So, in the proposed models, we consider the effect of words on each other using two parameters. The window parameter represents the range of the impact of a word, and the weight parameter represents the strength of the effect of the word. In the proposed models, each word has a weight, a sentiment label, and a topic and affects its neighbors as much as its window size, which means that each word has a window. For instance, as can be seen in Figure3, word w3 has the window size equal to 1 and affects words w2 and w4, and w6 has the window size equal to 2 and affects words w4, w5,w7,andw8. If word w3 had weight h and negative sentiment, words w2 and w4 would have weight h and negative sentiment as well. Each word is affected by its neighbors. So, different words in a document may be affected simultaneously by several neighbor words.Figure 3
An example of a sentence with different windows.


## 3.2. The Problem Statement

In this study, given a corpus of|R| documents, R=r1,r2,r3,…,rR, a document, r, consists of w1,w2,w3,…,wNr words, and each word belongs to a vocabulary set with a |V| distinct element. Furthermore, |Q| is the number of separate windows, |E| is the number of distinct weights, |S| is the number of distinct sentiment labels, and |Z| is the number of distinct topics. In the present study, five sets θ,φ,π,ψ,andξ require to be inferred which are latent variables. The hyperparameters α,β,γ,δ,andμ are given based on the experience, which can be the prior observation counts before observing any actual words, where α is Dirichlet prior distribution for θ, β is Dirichlet prior distribution for φ, γ is Dirichlet prior distribution for π, δ is Dirichlet prior distribution for ψ, and μ is Dirichlet prior distribution for ξ. The latent parameters z,s,q,e,φ,θ,π,ξ,andψ require to be approximated using observed variables, where z is topic variable, s is sentiment variable, q is window variable, and e is weight variable. The proposed models demonstrate the process of generating words in documents. Furthermore, they can approximate the latent variables. In the present study, the main aim of the proposed topic models is to categorize sentiments at the document-level.


### 3.2.1. The Problem We Are Trying to Solve or Improve

Analyzing user satisfaction with various services, products, or movies is the main problem in this study, mainly reflected in users’ comments. A user’s comment is formed by a message as text on the Internet which can be a tweet or a simple message on a website. So, for example, it is feasible to employ emotion mining to evaluate the opinions of various people in order to make them aware of things that have favorable reviews.


### 3.2.2. The Solution to the Problem

Many attempts have been made to detect emotions and explore the knowledge embedded in text data. Topic modeling as a known method can obtain concealed subjects of documents. LDA has been used as a topic model to effectively explore issues in the documents. As an LDA-based model, JST examines the impact of topics and emotions on words. The emotion parameter is insufficient, and additional parameters may play valuable roles in achieving better performance.This study presents two new topic models that extend and improve JST through two new parameters and generate a sentiment dictionary. The proposed models consider the effect of words on each other, which, from our view, is an important factor and can increase the performance of baseline methods. Several methods have been introduced for text emotion analysis that uses topic modeling. However, none of the researchers have improved the accuracy besides generating a sentiment dictionary. Different from other related studies, in this study, the proposed models improve topic-model-based sentiment classification using two parameters (weight and window). The proposed models are deeply described step-by-step in the following sections.


## 3.2.1. The Problem We Are Trying to Solve or Improve

Analyzing user satisfaction with various services, products, or movies is the main problem in this study, mainly reflected in users’ comments. A user’s comment is formed by a message as text on the Internet which can be a tweet or a simple message on a website. So, for example, it is feasible to employ emotion mining to evaluate the opinions of various people in order to make them aware of things that have favorable reviews.


## 3.2.2. The Solution to the Problem

Many attempts have been made to detect emotions and explore the knowledge embedded in text data. Topic modeling as a known method can obtain concealed subjects of documents. LDA has been used as a topic model to effectively explore issues in the documents. As an LDA-based model, JST examines the impact of topics and emotions on words. The emotion parameter is insufficient, and additional parameters may play valuable roles in achieving better performance.This study presents two new topic models that extend and improve JST through two new parameters and generate a sentiment dictionary. The proposed models consider the effect of words on each other, which, from our view, is an important factor and can increase the performance of baseline methods. Several methods have been introduced for text emotion analysis that uses topic modeling. However, none of the researchers have improved the accuracy besides generating a sentiment dictionary. Different from other related studies, in this study, the proposed models improve topic-model-based sentiment classification using two parameters (weight and window). The proposed models are deeply described step-by-step in the following sections.


## 3.3. The General Structure of WJST

This subsection introduces a new model named WJST, which improves JST using two parameters (weight and window). The primary goal of WJST is to classify sentiments at the document-level. A summary of symbols applied in the model is prepared in Table2. The process of generating a word of a document in WJST can be outlined as follows: (1) for each document, an author first decides the distribution of sentiments. For example, sentiments are 70% positive and 30% negative, so the proposed model chooses a sentiment label from the per-document sentiment distribution. (2) After determining the sentiment label, the author writes a review about a product according to the distribution of topics. For example, topics are 70% about memory, 20% about speed, and 10% about battery, so WJST chooses a topic from the per-document topic distribution that depends on the sentiment label. (3) After determining the sentiment label and the topic, the author decides the distribution of weights and the distribution of windows. WJST then chooses a weight from the per-document weight distribution that depends on the sentiment label and topic. WJST chooses a window size from the per-document window distribution that depends on the topic. (4) Finally, the author chooses some words to express an opinion under the identified topic, sentiment label, weight, and window. So, WJST draws a word from the per-corpus word distribution that depends on the topic, sentiment label, weight, and window. The words with different topics may have different window sizes. For example, a word with topic memory has a smaller window size than a word with topic mobile because topic mobile is more general than topic memory which can cover topic memory. So, the topic affects window size.Table 2
A summary of notations used in WJST.
SymbolDescriptionCollectionsRSet of all documentsVVocabulary setQSet of all distinct windows (with different sizes)ZSet of all topicsESet of all distinct weightsSSet of all sentiment labelsInit parametersqWindow variableeWeight variablerDocument variablezTopic variablewWord variablesSentiment variableDistributionsθProbability ofz given s and rφProbability ofw given z, s, q, and eπProbability ofs given r (s0 = Positive label, s1 = Negative label)ψProbability ofe given z, s, and rξProbability ofq given z and rHyper parametersαDirichlet prior distribution forθβDirichlet prior distribution forφγDirichlet prior distribution forπδDirichlet prior distribution forψμDirichlet prior distribution forξThe words with different topics may have different weights. For example, wordsize in topic memory is significant and considerable weight because all customers like memories with larger capacity. Word size in topic mobile is not important as word size in topic memory, and it has a small weight in topic mobile because some customers may like mobile phones with small size (iPhone 6s), and some customers may like the mobile phones with large size (iPhone 6s+). So, the topic affects weight. The words with different sentiment labels may have different weights. For example, suppose that topic memory contains two words size and cost. If the word size is positive, positive size will be more important than the word cost, and its weight will be larger than the cost. If word size is negative, the positive cost will be more important than the word size, and its weight will be larger than the size. Positive size means using words like large and big because customers like memories with larger capacity sizes. Negative size means using words like small because costumers do not like memories with smaller capacity size. Positive cost means using words like low and cheap because costumers like low-priced memories. Negative cost means using words like high and expensive because costumers do not like high-priced memories. So, sentiment label affects weight. The proposed model is parametric in this study [63]. Furthermore, the number of topics is constant. The generative model of WJST is demonstrated in Figure 4.Figure 4
The formal definition of the process of generating words in WJST.The symbols ofMulti and Dir demonstrate distributions of Multinomial and Dirichlet, respectively. Five sets of latent variables θ,φ,π,ψ,andξ require to be inferred which are latent variables. The hyperparameters α,β,γ,δ,andμ are given based on the experience, which can be the prior observation counts before observing any actual words. The latent parameters z,s,q,e,θ,φ,π,ψ,andξ require to be approximated using observed variables. The plate notation of WJST is exhibited in Figure 5. The plate notation is a method for expressing variables repeating in a graphical model. Furthermore, a probabilistic model shows the conditional dependency layout among the random variables as a graph.Figure 5
The plate notation of WJST.According to Figure5, the joint probability distributions for the model WJST can be factored as follows:(1)Pw,z,s,q,e=Pw|z,s,q,e×Pz|s,r×Ps|r×Pq|z,r×Pe|z,s,r,where by integrating out φ, we achieve:(2)Pw|z,s,q,e=ΓV×βΓβVZ×S×Q×E∏z∏s∏q∏e∏wΓNw,z,s,q,e+βΓNz,s,q,e+V×β,where |V| is the vocabulary size, |S| is the number of sentiment labels, |Z| is the number of topics, |Q| is the number of distinct windows, and |E| is the number of weights. The symbol Nw,z,s,q,e is the number of times the word w has been assigned to topic z, window q, weight e, and sentiment s. The symbol Nz,s,q,e is the number of words with topic z, window q, weight e, and sentiment s.The symbolβ is Dirichlet prior to φ. The symbolΓ is the gamma function. In addition, by integrating out θ, we achieve:(3)Pz|s,r=ΓZ×αΓαZS×R∏r∏s∏zΓNz,s,r+αΓNs,r+Z×α,where R is the number of documents and Nz,s,r is the number of words with topic z with sentiment s in document r. The symbol Ns,r is the number of words with sentiment s in document r. The symbol α is Dirichlet before θ. And by integrating out π, we achieve:(4)Ps|r=ΓS×γΓγSR∏r∏sΓFs,r+γΓFr+S×γ,where Fs,r is the effect of words with sentiment s in document r, which is equal to ∑w∈rew,s,r×1+2×qw,s,r where ew,s,r is the weight of word w with sentiment s in document r and qw,s,r is the window size of word w with sentiment s in document r. The symbol Fr is the sum of the effect of words with different sentiments (positive and negative) in document r, which is equal to ∑s∈positive,negativeFs,r.The symbolγ is Dirichlet before π. And by integrating out ξ, we achieve:(5)Pq|z,r=ΓQ×μΓμQZ×R∏r∏z∏qΓNq,z,r+μΓNz,r+Q×μ,where |Q| is the number of distinct windows. The symbol Nq,z,r is the number of words with topic z and window q in document r. The symbol Nz,r is the number of words with topic z in document r. The symbol μ is Dirichlet before ξ. And by integrating out ψ, we achieve:(6)Pe|z,s,r=ΓE|×δΓδEZ×S×R∏s∏r∏z∏eΓNe,z,s,r+δΓNz,s,r+E×δ,where |E| is the number of weights, Ne,z,s,r is the number of words with topic z, weight e, and sentiment s in document r. The symbol Nz,s,r is the number of words with sentiment s and topic z in document r. The symbolδ is Dirichlet before ψ. To estimate the parameters φ,θ,π,ξ,andψ, we need to evaluate the above distributions. These distributions are difficult to assess directly, so we adopt Gibbs sampling to perform approximate inference. Gibbs sampling is a widely used inference technique and is a popular approach for parameter estimation and inference in many topic models such as LDA [7]. The advantage of using the Gibbs sampling method is that it is simple and easy to implement. In this study, Gibbs sampling is used to estimate the distributions of the latent variables. The pseudocode of the Gibbs sampling algorithm is given in Figure 6 for the proposed model, and the meanings of all variables are seen in Table 2. The algorithm will sample each variable (z, s, q, and e) based on the following formula by canceling terms in equations (2)–(6) (by replacing terms in (1) with those in equations (2)–(6):(7)Pzr,i=z,sr,i=s,qr,i=q,er,i=e|zr,i,sr,i,qr,i,er,i,β,α,γ,μ,δ∝Nw,z,s,q,e+βNz,s,q,e+V×β×Nz,s,r+αNs,r+Z×α×Fs,r+γFr+S×γ×Nq,z,r+μNz,r+Q×μ×Ne,z,s,r+δNz,s,r+E×δ,where zr,i,sr,i,qr,i, and er,i are topic, sentiment, window, and weight assignments, respectively, for all the words in the collection, except for the word considered at position i in document r. Posterior inference of parameters is performed using Gibbs sampling, as demonstrated in Figure 6.Figure 6
Adopted Gibbs sampling for WJST1.In the section of initialization, the method randomly sets the parameters. A sentiment dictionary is employed for initializing sentiment labels. The sentiment dictionary contains words and scores that specify positive and negative labels and their weight. In this study, AFINN [64] is used as a sentiment dictionary, improving the model’s accuracy. At the end of the sampling algorithm, each word has a weight and a sentiment label. Therefore, a dictionary can generate sentiment scores (weights and sentiment labels) and words. The scores are extracted from a dataset based on P(w| s, e). Each word’s weight and sentiment with the most probability are selected as sentiment scores among all documents. Adaptive Lexicon learning using Genetic Algorithm (ALGA) [46] uses the genetic algorithm to generate a sentiment dictionary. However, we use topic modeling in WJST, to generate this dictionary. In WJST, the window size is different for various words. At each step of the sampling algorithm, count variables such as Fs,r and Fr are updated after sampling sentiment label, weight, and window size. After completing the sampling, the distribution of latent variables (φ,θ,π,ξ,andψ) can be calculated as follows:(8)φ=Nw,z,s,q,e+βNz,s,q,e+V×β,(9)θ=Nz,s,r+αNs,r+Z×α,(10)π=Fs,r+γFr+S×γ,(11)ξ=Nq,z,r+μNz,r+Q×μ,ψ=Ne,z,s,r+δNz,s,r+E×δ.The probability of a word given a topic would be equal to∑s,q,ePw|z,s,q,e, and the probability of a sentiment label given a document for sentiment classification at the document-level is calculated using π.The time complexity of the proposed method quantifies the amount of time taken by the Gibbs sampling algorithm to run as the main function. Given the number of words in all documentswALL (wALL=∑r∈RNr, where Nr is the number of words in document r), the number of topics |Z|, the number of distinct windows |Q|, the number of weights |E|, and the total number of sentiment labels |S|, the time complexity of each Gibbs sampling iteration would be O(wALL·|S|·|Z|·|Q|·|E|). Furthermore, given the number of iterations G, the total time complexity of WJST would be O(G·wALL·|S|·|Z|·|Q|·|E|). Table 3 compares different methods in terms of time complexity.Table 3
The time complexity of different models.
ModelTime complexityJST, RJST, TJST, and TSO(G·wALL·|S|·|Z|)WJSTO(G·wALL·|S|·|Z|·|Q|·|E|)


## 3.4. The General Structure of WJST1

A version of WJST called WJST1 is presented in Figure7. The distributions θ,ξ,andψ in WJST depend on the document, but in WJST1, the distributions θ,ξ,andψ do not rely on the document. Dependency between documents of a domain is more than documents in different domains. A pattern in documents of a domain may not exist in documents of other domains. So, calculations on multidomain datasets should be local and not cover all domains. For example, considering the distributions Pz|s and Pz|s,r, where z is topic, s is sentiment, and r documents, in the first state Pz|s, topic depends on sentiment. The distribution covers all documents in different domains. Perhaps a topic is positive in one domain and negative in another domain. So, it is better to depend the topic on the documents of a domain, not all domains. Thus, the topic is limited to the document (and domain), and contradiction between different domains is eliminated. So, WJST is suitable for multidomain datasets, and WJST1 is a version of WJST suitable for single-domain datasets. According to Figure 7, ξ is the probability of q given z, θ is the probability of z given s, and ψ is the probability of e given z and s, and the joint probability distribution for WJST1 can be factored as follows:(12)Pw,z,s,q,e=Pw|z,s,q,e×Pz|s×Ps|r×Pq|z×Pe|z,swhere by integrating out θ, we achieve:(13)Pz|s=ΓZ×αΓαZS∏s∏zΓNz,s+αΓNs+Z×α,whereNz,s is the number of words with topic z and sentiment s. The symbol Ns is the number of words with sentiment s. The symbol α is Dirichlet before θ. And by integrating out ξ, we achieve:(14)Pq|z=ΓQ×μΓμQZ∏z∏qΓNq,z+μΓNz+Q×μ,whereNq,z is the number of words with topic z and window q. The symbol Nz is the number of words with topic z. The symbol μ is Dirichlet before ξ. And by integrating out ψ, we achieve:(15)Pe|z,s=ΓE×δΓδEZ×S∏s∏z∏eΓNe,z,s+δΓNz,s+E|×δ,whereNe,z,s is the number of words with topic z, weight e, and sentiment s. The symbol Nz,s is the number of words with sentiment s and topic z. The symbol δ is Dirichlet before ψ. The symbols Pw|z,s,q,eandPs|r are calculated using equations (2) and (4), respectively. After completing the sampling, the distribution of latent variables (θ,ξ,andψ) is calculated as follows:(16)θ=Nz,s+αNs+Z×α,ξ=Nq,z+μNz+Q×μ,ψ=Ne,z,s+δNz,s+E×δ.Figure 7
The graphical model of WJST1.Andφandπ are computed through equations (8) and (10), respectively. Experimental results are demonstrated in the next section.


## 4. Experimental Results

The present study executes the methods on a computer with an Intel Core i7 CPU and 8 GB RAM. Proposed models are compared on 13 datasets. 4 datasets crawled from Amazon (https://www.amazon.com) opinions include Electronic, Movie, Android, and Automotive. 2 MDS datasets [65] contain Magazines and Sports. A dataset crawled from the IMDB movie archive [3] is MR. 3 UCI datasets [66] include Amazon, Yelp, and IMDB. 3 Twitter datasets [46] include STS-Test, SOMD, and Sanders. Data preprocessing contains (1) lowercasing all words, (2) removing digits, nonalphabetic characters, stop words, and words with too low and too high frequency, and (3) stemming. The details of the datasets are provided in Table 4.Table 4
Description of datasets.
#DatasetNumber of reviewsVocabulary sizeNumber of words1Movie4006592415402Electronic4004501291173Automotive4003590197334Android400217397235STS359148937846SOMD916201377727Sanders12243221141008Magazines180080401253879Sports2000858211392110MR20003305473302211Amazon10001521729612IMDB10002556970613Yelp100016797726The number of topics is unknown, provided as a constant amount at the beginning of the Gibbs sampling algorithm. In this study,α,γ,β,andδ specific distributions are symmetric, and we empirically set the value of parameters, and this setting demonstrates fairly good performance in our experiments. Table 5 exhibits the initialization of parameters used in different algorithms.Table 5
Initial values of parameters.
ModelParametersJSTMax_iteration:5000; |Z| = 5,10,15,20;α=0.1; γ=0.016×average document length; β=0.01;RJSTMax_iteration:5000; |Z| = 5,10,15,20; α=0.1; γ=0.016×average document length; β=0.01;TJSTMax_iteration:5000; |Z| = 5,10,15,20; α=0.1; γ=0.016×average document length; β=0.01;TSMax_iteration:5000; |Z| = 5,10,15,20; α=0.1; γ=0.016×average document length; β=0.01;WJSTMax_iteration:5000; |Z| = 5,10,15,20; α=0.3; γ=0.016×average document length; β=0.01; μ=3; δ=9; E = [−5, +5]; Q = {1,2,3,4,5,6};WJST1Max_iteration:5000; |Z| = 5,10,15,20; α=0.3; γ=0.016×average document length; β=0.01;μ=3; δ=9; E = [−5, +5]; Q = {1,2,3,4,5,6};A sentiment dictionary is employed for initializing sentiment labels. Sentiment dictionaries such as AFINN [64], IMDB [67], 8-K [67], and Bing Liu [68, 69] contain words and scores that specify positive and negative labels as well as their weight. In the present study, AFINN is used as a sentiment dictionary which improves the model’s accuracy. Sentiment detection at the document-level, perplexity, and topic_coherency are used to compare the efficacy of proposed models as three standard parameters which are used in different papers [7, 70, 71–73].In the present study, the Accuracy parameter uses the formula of ((TP + TN))⁄((TP + FP + TN + FN)), where TP is the number of true positives, TN is the number of true negatives, FP is the number of false positives, and FN is the number of false negatives.π distribution equation (10) determines how likely each comment is positive or negative. For example, if the value of P(+) is more significant than the value of P(−) (for a comment), the comment will be positive. The Accuracy’s formula uses π distribution (equation (10)) to calculate TP, TN, FP, and FN values. For example, if a comment is positive and detected as positive (by the proposed methods), a unit is added to TP.So, sentiment analysis (sentiment detection) at the document-level is realized usingπ distribution (equation (10)), and the formula of ((TP + TN))⁄((TP + FP + TN + FN)) is used to compute the Accuracy.The error formula can be calculated using the formula of (1-Accuracy). Accuracy, perplexity, and topic_coherency are used for evaluations in the present study. Further study can investigate more parameters such as MSE, MAE, and RMSE for future research.Furthermore, Better methods have lower perplexity and also higher topic_coherency. Given a test datasetDTest, the perplexity is computed through(17)PerplexityDTest=exp−∑r=1RlogPwr∑r=1RNr,where wr are the words in document r, Nr is the length of document r, and Pwr is the probability of words in document r. The lower value of the formula over a held-out document demonstrates Better generalization efficacy. The evaluation results are shown in Tables 6–8, 9–14, and the proposed models demonstrate better results. In the report of Tables 6–8, 9–14, the perplexity of proposed methods is lower than that of baseline models. In the report of Tables 9–12, the perplexity is reduced with an increase in topics. Topic_coherency is also calculated using(18)Average Topic−CoherencyZ=∑i=1ZCVziZ=∑i=1Z∑m=2M∑n=1m−1logCODFvmzi,vnzi+1/DFvnziZ.where Vzi=v1zi,…,vMzi is the list of M words that have a high probability in the topic zi, CVzi is topic_coherency for the topic zi, Z is the set of all topics, |Z| is the number of distinct topics, DF is the document frequency, and CODF is the co-occurrence of two words in different documents. A smoothing count of 1 is included to avoid taking the logarithm of zero. In the present study, topic_coherency is computed through (18), equal to the average of topic_coherency values in Z. Furthermore, a higher value of topic_coherency reflects the better quality of the detected topics. M is equal to 10, and results are demonstrated in Tables 6–8, 9–14. A different number of topics (5, 10, 15, and 20) and different distinct windows (1, 2, 3, 4, 5, and 6) are applied for evaluating models. In this part, baseline methods include JST [8], RJST [10], TJST [8], and TS [9]. In the present section, the Friedman test [74, 75] is used to examine the achievements of the comparison methods. The Friedman test is a nonparametric multiple comparison test utilized to examine the differences between algorithms by assigning the lowest rank to the best approach in minimization problems and the highest rank to the best approach in maximization problems.Table 6
Sentiment classification on Android, Automotive, Electronic, and Movie datasets.
AndroidMetric\ modelRNDAFINNRND + AFINNJSTTJSTRJSTTSWJSTWJST1Accuracy10.480.69750.580.6250.7650.58250.54250.7950.865Accuracy2———————0.78250.8525Perplexity———17.458119.718517.442617.870614.39614.7631Topic_coherency———−2.0645−0.8536−1.9914−2.373−0.5547−0.187AutomotiveAccuracy10.49250.6250.5350.65750.76750.6150.55250.7550.8Accuracy2———————0.74750.795Perplexity———22.683824.038521.804422.487818.461219.0627Topic_coherency———−1.0158−0.4712−1.4986−0.9008−0.9311−0.326ElectronicAccuracy10.4650.6750.520.70250.760.55250.54750.86250.8475Accuracy2———————0.8750.855Perplexity———23.358624.302423.47124.023919.299920.2452Topic_coherency———−1.5892−1.0482−1.2996−1.2719−0.5322−1.1683MovieAccuracy10.5250.5950.5550.75750.94750.620.54250.84750.97Accuracy2———————0.83250.9675Perplexity———25.278726.581325.168425.448821.049422.1082Topic_coherency———−0.4089−0.111−1.0947−1.0214−0.0602−0.1329Table 7
Sentiment classification on Magazine, Sport, MR, Amazon, IMDB, and Yelp datasets.
MagazineMetric\ modelRNDAFINNRND + AFINNJSTTJSTRJSTTSWJSTWJST1Accuracy10.5150.65220.58220.67050.7050.54110.50220.83550.81Accuracy2———————0.83720.8083Perplexity———21.850623.034921.459321.382819.991421.2095Topic_coherency———−0.0548−0.0348−0.0946−0.0561−0.132−0.0077SportAccuracy10.52850.6860.57250.6530.7090.55650.51550.7980.802Accuracy2———————0.7820.795Perplexity———22.87423.135622.026422.336121.96821.4821Topic_coherency———−0.2234−0.0876−0.1369−0.0544−0.1406−0.2242MRAccuracy10.48950.6010.54550.6130.620.510.50.8210.8445Accuracy2———————0.8180.843Perplexity———33.866335.069535.235934.670433.22233.7698Topic_coherency———−0.021−0.0139−0.0012−0.0409−0.001−0.0106AmazonAccuracy10.4910.7310.5740.6110.6450.6090.540.7790.796Accuracy2———————0.8290.798Perplexity———12.644213.731613.34914.239710.921112.9946Topic_coherency———−0.8318−4.3775−0.4224−0.5411−0.5874−0.2696IMDBAccuracy10.4980.6980.5750.6050.6160.5460.5450.760.77Accuracy2———————0.7610.774Perplexity———21.105320.441920.854319.712414.671918.9035Topic_coherency———−1.3334−1.4868−0.8853−1.1246−0.9666−0.9438YelpAccuracy10.5060.6890.5590.5790.6140.5610.5470.7370.773Accuracy2———————0.7260.769Perplexity———15.456516.716915.596515.361412.214513.3453Topic_coherency———−2.1865−2.5609−1.9632−1.1714−2.0815−2.3715Table 8
Sentiment classification on different datasets based on different situations (AFINN and NO_AFINN).
AndroidMetricMetric\DicAFINNNO_AFINNWJSTAccuracy10.74250.5725Accuracy20.73750.58Perplexity15.5316.1551Topic_Coh−2.2654−1.7346WJST1Accuracy10.8550.81Accuracy20.84750.8075Perplexity16.339916.0482Topic_Coh−2.2228−0.1295AutomotiveWJSTAccuracy10.71250.6025Accuracy20.70250.6075Perplexity20.448820.4065Topic_Coh−3.2282−1.6628WJST1Accuracy10.79250.7025Accuracy20.790.7125Perplexity20.521321.1296Topic_Coh−0.326−1.1809ElectronicWJSTAccuracy10.85250.705Accuracy20.84250.6825Perplexity20.057920.2615Topic_Coh−0.5322−0.5926WJST1Accuracy10.84750.76Accuracy20.8550.765Perplexity21.819521.5739Topic_Coh−1.5586−1.6968MovieWJSTAccuracy10.84750.71Accuracy20.83250.715Perplexity22.458822.6124Topic_Coh−0.9342−0.4637WJST1Accuracy10.95750.485Accuracy20.9450.4875Perplexity23.566222.8134Topic_Coh−1.2359−1.3717Table 9
Sentiment classification on the Android dataset according to the different number of topics.
ModelMetric\topic5101520RNDAccuracy0.480.480.480.48AFINNAccuracy0.69750.69750.69750.6975AFINN + RNDAccuracy0.580.580.580.58Bing_LiuAccuracy0.69750.69750.69750.6975Bing_Liu + RNDAccuracy0.57750.57750.57750.5775IMDBAccuracy0.70250.70250.70250.7025IMDB + RNDAccuracy0.61250.61250.61250.61258KAccuracy0.54250.54250.54250.54258K + RNDAccuracy0.5150.5150.5150.515JSTAccuracy0.6250.61750.62250.6125Perplexity19.672619.518719.18217.4581Topic_Coh−4.6026−2.5848−2.2753−2.0645TJSTAccuracy0.75750.71750.7650.7475Perplexity21.148720.372620.151619.7185Topic_Coh−0.8536−1.568−3.4285−2.8739RJSTAccuracy0.58250.540.5550.5325Perplexity19.942919.191518.13717.4426Topic_Coh−3.3792−1.9914−3.403−3.4386TSAccuracy0.54250.52750.530.5175Perplexity20.493419.176218.383317.8706Topic_Coh−3.9618−3.2137−2.373−2.569WJSTAccuracy10.79250.74250.7950.79Accuracy20.77750.73750.7750.7825Perplexity16.730315.5314.635114.396Topic_Coh−0.5547−2.2654−2.9122−2.6465WJST1Accuracy10.810.8550.8650.85Accuracy20.79250.84750.85250.8375Perplexity16.678716.339915.666214.7631Topic_Coh−0.187−2.2228−1.5703−2.0204Table 10
Sentiment classification on the Automotive dataset according to the different number of topics.
ModelMetric\topic5101520RNDAccuracy0.49250.49250.49250.4925AFINNAccuracy0.6250.6250.6250.625AFINN + RNDAccuracy0.5350.5350.5350.535Bing_LiuAccuracy0.640.640.640.64Bing_Liu + RNDAccuracy0.5350.5350.5350.535IMDBAccuracy0.590.590.590.59IMDB + RNDAccuracy0.48250.48250.48250.48258KAccuracy0.50250.50250.50250.50258K + RNDAccuracy0.480.480.480.48JSTAccuracy0.62750.65750.63250.5975Perplexity24.715423.696123.2722.6838Topic_Coh−1.486−2.3443−1.1354−1.0158TJSTAccuracy0.760.73750.76750.7275Perplexity25.563725.063324.574924.0385Topic_Coh−0.59−0.4712−1.6252−1.9377RJSTAccuracy0.6150.550.540.535Perplexity25.265423.831622.390521.8044Topic_Coh−1.5779−1.4986−2.0833−2.9936TSAccuracy0.54250.53750.55250.55Perplexity25.070523.850422.936422.4878Topic_Coh−2.5941−0.9008−6.2652−4.6902WJSTAccuracy10.7550.71250.740.745Accuracy20.74750.70250.7450.735Perplexity21.200820.448818.704918.4612Topic_Coh−1.6542−3.2282−1.6783−0.9311WJST1Accuracy10.800.79250.79250.7925Accuracy20.790.790.790.795Perplexity21.435720.521320.048119.0627Topic_Coh−1.1883−0.326−1.084−0.8349Table 11
Sentiment classification on the Electronic dataset according to the different number of topics.
ModelMetric\topic5101520RNDAccuracy0.4650.4650.4650.465AFINNAccuracy0.6750.6750.6750.675AFINN + RNDAccuracy0.520.520.520.52Bing_LiuAccuracy0.6950.6950.6950.695Bing_Liu + RNDAccuracy0.5350.5350.5350.535IMDBAccuracy0.63750.63750.63750.6375IMDB + RNDAccuracy0.54750.54750.54750.54758KAccuracy0.50750.50750.50750.50758K + RNDAccuracy0.50750.50750.50750.5075JSTAccuracy0.6750.70250.64750.6275Perplexity25.22724.671924.611523.3586Topic_Coh−1.5892−4.1751−2.6007−2.1239TJSTAccuracy0.750.740.730.76Perplexity25.209124.802824.409224.3024Topic_Coh−1.0482−1.8084−1.6694−1.8297RJSTAccuracy0.55250.550.53750.53Perplexity25.311324.177923.837823.471Topic_Coh−1.2996−1.3991−4.472−5.4363TSAccuracy0.540.53750.54750.515Perplexity26.229524.88524.504824.0239Topic_Coh−1.2719−1.6586−2.5403−3.2174WJSTAccuracy10.86250.85250.76750.675Accuracy20.8750.84250.7550.665Perplexity20.548920.057919.900919.2999Topic_Coh−2.0442−0.5322−1.3702−1.0887WJST1Accuracy10.790.84750.80750.8275Accuracy20.800.8550.81250.8225Perplexity22.601221.819520.855120.2452Topic_Coh−1.1683−1.5586−1.6646−1.4681Table 12
Sentiment classification on the Movie dataset according to the different number of topics.
ModelMetric\topic5101520RNDAccuracy0.5250.5250.5250.525AFINNAccuracy0.5950.5950.5950.595AFINN + RNDAccuracy0.5550.5550.5550.555Bing_LiuAccuracy0.6350.6350.6350.635Bing_Liu + RNDAccuracy0.5650.5650.5650.565IMDBAccuracy0.64250.64250.64250.6425IMDB + RNDAccuracy0.59750.59750.59750.59758KAccuracy0.50250.50250.50250.50258K + RNDAccuracy0.510.510.510.51JSTAccuracy0.75750.63750.71750.6325Perplexity27.311126.914526.246325.2787Topic_Coh−1.0123−0.4089−1.2434−1.226TJSTAccuracy0.9150.94250.94750.9325Perplexity27.79127.412226.99326.5813Topic_Coh−0.111−1.6632−0.199−0.8503RJSTAccuracy0.620.540.51750.5175Perplexity27.628426.517226.148925.1684Topic_Coh−2.6713−1.0947−1.979−2.3544TSAccuracy0.54250.5150.51750.5175Perplexity27.670726.478626.113825.4488Topic_Coh−1.5025−1.3799−1.0214−4.6598WJSTAccuracy10.72250.84750.75250.6975Accuracy20.710.83250.75750.6875Perplexity23.114522.458821.298221.0494Topic_Coh−0.0751−0.9342−0.0602−0.4873WJST1Accuracy10.970.95750.96750.9675Accuracy20.96250.9450.96750.96Perplexity24.630223.566222.718022.1082Topic_Coh−0.1348−1.2359−0.1329−0.7102Table 13
Sentiment classification on different datasets according to the different number of distinct windows (before random selection).
AndroidModelMetric\window123456WJSTAccuracy10.83750.780.79250.74250.7250.9075Accuracy20.830.770.77750.7350.7250.9075Perplexity18.394816.726616.730316.16615.949915.1128Topic_Coh−1.4063−0.9279−0.5547−1.2929−1.7475−0.746WJST1Accuracy10.89750.85250.810.86750.7550.8025Accuracy20.88250.830.79250.86750.750.7875Perplexity19.373418.003116.678716.851816.101516.5461Topic_Coh−2.2084−1.6272−0.1870−0.7753−1.328−1.7836AutomotiveWJSTAccuracy10.77750.740.7550.7350.73750.69Accuracy20.77250.7450.74750.7450.7250.685Perplexity23.636521.771221.200820.968820.509519.6748Topic_Coh−1.1684−1.7095−1.6542−0.2824−1.7604−0.1311WJST1Accuracy10.8050.81250.80.75750.780.7725Accuracy20.8050.79750.790.7550.780.7725Perplexity23.268422.260121.435720.809220.509120.2379Topic_Coh−1.7318−0.5912−1.1883−0.7637−0.62−0.5134ElectronicWJSTAccuracy10.8450.76750.86250.73250.77750.7325Accuracy20.8450.75750.8750.72750.78250.7225Perplexity22.56122.046220.548920.795220.749520.0283Topic_Coh−0.8471−0.6207−2.0442−1.4345−0.786−0.9412WJST1Accuracy10.80250.8750.790.86750.86250.835Accuracy20.79750.870.80.8650.86250.8375Perplexity23.590323.54622.601222.855921.889721.4464Topic_Coh−0.8251−0.9581−1.1683−1.2189−0.8492−0.3402MovieWJSTAccuracy10.8550.8150.72250.65750.70.765Accuracy20.8450.79750.710.66250.6850.765Perplexity24.577924.515323.114523.120923.270922.2328Topic_Coh−0.4063−0.0216−0.0751−2.5351−0.0888−0.0791WJST1Accuracy10.97250.97750.970.9650.5750.595Accuracy20.960.9650.96250.9550.58750.5725Perplexity26.11725.0924.630223.977822.706822.3572Topic_Coh−0.1348−0.1348−0.1348−0.0315−0.0378−0.0106Average sectionWJSTAccuracy10.82870.77560.78310.71680.7350.7737Accuracy20.82310.76750.77750.71750.72930.77Perplexity22.292521.264820.398620.262720.119919.2621Topic_Coh−0.957−0.8199−1.082−1.3862−1.0956−0.4743WJST1Accuracy10.86930.87930.84250.86430.74310.7512Accuracy20.86120.86560.83620.86060.7450.7425Perplexity23.087222.224821.336421.123620.301720.1469Topic_Coh−1.225−0.8278−0.6696−0.6973−0.7087−0.6619Table 14
Sentiment classification on different datasets according to the different number of distinct windows (after random selection).
AndroidModelMetric\window123456WJSTAccuracy10.710.7750.70250.690.71750.675Accuracy20.70.7650.69750.680.7050.6675Perplexity17.79115.854416.112415.693213.890414.1876Topic_Coh−1.6−1.9691−1.4526−4.6888−2.6353−1.2267WJST1Accuracy10.80250.850.8450.80250.86750.76Accuracy20.79250.830.840.78750.8650.7525Perplexity18.931917.784917.614515.801715.400415.0917Topic_Coh−1.2666−1.6744−3.027−2.3428−1.1215−2.0183AutomotiveWJSTAccuracy10.74250.73750.680.68250.67250.64Accuracy20.740.730.67750.680.66250.6325Perplexity20.843320.721219.684320.080119.169118.7129Topic_Coh−1.4406−2.0489−2.6701−2.2155−1.1019−1.1651WJST1Accuracy10.78250.760.7550.77250.74750.7675Accuracy20.7850.75750.74750.7750.75250.76Perplexity22.185621.360120.605719.891519.40219.0827Topic_Coh−1.5638−1.474−1.1274−1.7761−1.2042−1.1755ElectronicWJSTAccuracy10.79750.720.760.68750.76750.6825Accuracy20.7950.7150.74750.660.7650.665Perplexity21.917121.050920.485820.2619.408919.5549Topic_Coh−0.8282−1.0407−0.9593−0.9779−1.0413−0.6259WJST1Accuracy10.74750.74250.77750.74250.77250.78Accuracy20.7450.74750.780.740.76250.785Perplexity23.226922.457321.720321.459820.868320.5084Topic_Coh−1.7905−1.5082−1.4854−0.9143−1.2881−1.3489MovieWJSTAccuracy10.74250.770.80750.6150.70.595Accuracy20.74250.7650.78750.590.690.595Perplexity23.752923.32222.105621.857320.846720.9653Topic_Coh−1.0908−0.9134−0.9145−0.6129−1.1099−0.716WJST1Accuracy10.97250.9650.960.96750.960.9625Accuracy20.9750.960.9550.9550.95750.9575Perplexity26.179725.313824.514924.111623.502523.3023Topic_Coh−0.0218−0.2225−1.9033−0.0897−0.2896−0.0569Average sectionWJSTAccuracy10.74810.75060.73750.66870.71430.6481Accuracy20.74430.74370.72750.65250.70560.64Perplexity21.07620.237119.59719.472618.328718.3551Topic_Coh−1.2399−1.493−1.4991−2.1237−1.4721−0.9334WJST1Accuracy10.82620.82930.83430.82120.83680.8175Accuracy20.82430.82370.83060.81430.83430.8137Perplexity22.63121.72921.113820.316119.793319.4962Topic_Coh−1.1606−1.2197−1.8857−1.2807−0.9758−1.1499There are several methods for the validation of classification and topic modeling-based problems. Still, the methods used in this study are the most common and are used in most articles related to our article for evaluation. Also, there are various methods for validation that we will try to use in a future study to evaluate the proposed methods. The following is the reason for choosing the validation methods used in this study:We chose accuracy, perplexity, and coherence score as evaluation metrics because of their popularity in classification and topic modeling problems. Perplexity is an essential metric that, in theory, represents how well a model behaved on unseen data and is provided using the normalized log-likelihood technique. Meanwhile, the coherence score measures the degree of semantic similarity between high-scoring words and helps distinguish the semantical interpretation of topics based on statistical inference.The main question we want to answer is whether the proposed methods can improve the performance of text sentiment classification. This study compares proposed methods with different baselines, including JST and recently representative approaches. Consider a Confusion Matrix for a classification problem that predicts whether a comment has positive sentiment or not. The total number of correctly detected cases is one of the more obvious measures. When all of the classes are equally important, it is typically utilized. When True positives and True negatives are more significant, accuracy is employed. According to the accuracy criterion, one can immediately know whether the model is adequately trained or not and how it works in general. The most popular measurement for classification issues is accuracy, which is the proportion of correctly predicted cases to all cases. This metric's opposite, or error, can be calculated as 1-accuracy. In machine learning, an accuracy parameter is an excellent option for sentiment classification when the classes in the dataset are almost evenly distributed. Also, we will try to use various metrics such as recall and precision in future studies to evaluate the proposed methods.We use the Friedman test to compare the results produced by the proposed methods and the competitors to verify the classification performance. Friedman’s test is used to examine the achievements of the comparison methods. The Friedman test is a nonparametric multiple comparison test that is utilized to explore the differences between algorithms by assigning the lowest rank to the best approach in minimization problems and the highest rank to the best approach in maximization problems.Topic modeling is one of the most important NLP fields. It aims to explain a textual dataset by decomposing it into two distributions: topics and words. A topic modeling algorithm is a mathematical or statistical model used to infer what the issues that better represent the data are. Human judgment-based review techniques can yield good results but are expensive and time-consuming. Human judgment is also not well defined.In contrast, the appeal of quantitative metrics such as perplexity is the ability to standardize, automate, and scale the evaluation of topic models. In natural language processing, perplexity is a traditional metric for evaluating topic models. The lower value of the formula over a held-out document demonstrates better generalization efficacy.Perplexity's inability to capture context and the relationships between words within a topic or across topics within a document is one of its drawbacks. For human understanding, semantic context is important. Approaches like topic coherency have been designed to tackle this problem by capturing the context between words in a subject. Extracting topic words is one of the main tasks in topic modeling. In most articles about topic modeling, topic_coherency is shown as a number that represents the overall topics’ interpretability and is used to assess the topics’ quality. The higher the topic_coherency value, the better the quality of the subjects extracted.


### 4.1. Sentiment Scores for the Words in a Dataset

In this section, a dictionary is generated, including sentiment scores (weights and sentiment labels) and words. The scores are extracted from datasets based on P(w| s, e). The weight and sentiment with the most probability are selected for each word as a sentiment score. The extracted scores for some phrases in the form of unigram can be seen in Tables 15 and 16. ALGA [46] uses the genetic algorithm to generate a sentiment dictionary; however, we use topic modeling in the proposed models to create this dictionary. According to Tables 15 and 16, ten words from each dataset are selected and scored by the proposed models. For example, the word nice obtains a score of 4 in WJST and obtains a score of 5 in WJST1. The scores are different in the proposed methods; for example, the word serious achieves a score of 1 in WJST and a score of -2 in WJST1. Table 15 is related to Android, Automotive, Electronic, and Movie datasets. Table 16 is associated with STS, Sanders, and SOMD datasets.Table 15
Sentiment scores, for some instance, words related to Android, Automotive, Electronic, and Movie datasets.
DatasetAndroidAutomotiveElectronicMovieModelWordScoreWordScoreWordScoreWordScoreWJST1Nice5Much5Satisfy5See5Cute4Use4Crew4Father4Favorit3Long3Way3Pray3Perfect2Expens2Fluid2Human2Great1Stuff1Feel1Event1Type−1Fals−1Pull−1Terribl−1Wast−2Serious−2Side−2Sens−2Everi−3Extens−3Nervous−3Lost−3Unknown−4Space−4Even−4Sure−4Everyth−5Know−5Extend−5Injur−5WJSTNice4Much3Satisfy−2See−4Cute1Use2Crew2Father5Favorit2Long4Way4Pray−3Perfect2Expens−4Fluid1Human5Great2Stuff−5Feel−3Event1Type1Fals−5Pull−4Terribl4Wast−5Serious1Side−1Sens−4Everi4Extens5Nervous3Lost3Unknown5Space−5Even−2Sure2Everyth2Know4Extend−5Injur−5Table 16
Sentiment scores for some instance words related to STS, Sanders, and SOMD datasets.
DatasetSTSSandersSOMDModelWordScoreScoreScoreWJST1Much−442Good54−5Bad−5−1−5Nice−355Hate−5−51Love553WJSTMuch−5−5−2Good−4−13Bad−4−13Nice321Hate−5−42Love4−2−3


### 4.2. Topic Discovery

The topics are extracted from datasets based onP(w|z) in this section. A topic is a multinomial distribution over words based on topics, sentiments, weights, and window sizes. The top words could approximately reflect the meaning of a topic. Tables 17–19 show some examples of topics extracted from Movie, Android, and Electronic datasets by different models. Each row shows the top 10 words for the corresponding topic and sentiment label. The top 10 words from each topic were extracted and then used for topic_coherency. Extracting topic words is one of the main tasks in topic modeling. This section lists the top 10 words in three examples for Movie, Android, and Electronic datasets. The listed words for each topic describe the topic. The listed words for the proposed methods have a better topic_coherency value than baseline methods because they have a higher value of topic_coherency. The higher the topic_coherency value, the better the quality of the subjects extracted.Table 17
Top 10 words extracted from the Movie dataset.
ModelSentimentTop 10 wordsWJST+jesu, film, God, love, mel, Christian, life, suffer, believ, roman−movi, godzilla, bad, dvd, origin, horror, buy, version, worst, actorWJST1+Jesu, mel, passion, mother, stori, realli, great, everyon, God, like−godzilla, monster, go, time, star, know, kill, make, militari, AmericanJST+mel, stori, mother, two, realli, becom, anoth, God, like, back−godzilla, look, monster, american, militari, like, worst, zellweg, emmerich, quitTable 18
Top 10 words extracted from Android dataset.
ModelSentimentTop 10 wordsWJST+app, game, sudoku, play, version, enjoy, option, want, hint, like−work, app, would, fire, live, station, tri, say, select, kindl, load, userWJST1+sudoku, tri, love, game, time, easi, tablet, star, call, make−close, tablet, seem, get, year, download, much, station, time, androidJST+station, want, even, peopl, work, avail, version, puzzl, custom, believ−use, app, find, review, great, got, total, new, night, fakeTable 19
Top 10 words extracted from Electronic dataset.
ModelSentimentTop 10 wordsWJST+read, book, screen, touch, kindl, page, better, wifi, ebook, like−work, went, new, servic, need, bad, system, hous, number, mineWJST1+googl, amazon, book, color, store, kindl, download, small, pdf−time, work, two, much, one, power, comput, phone, go, unitJST+book, touch, read, page, free, librari, touch, screen, much, pdf−plug, work, could, devic, comput, charger, router, cabl, item, design


### 4.3. Sentiment Classification at Document-Level

In this section, the number of distinct windows is three, and the models use the AFINN sentiment dictionary in the initialization section of the Gibbs sampling algorithm. A document is classified based onP(s| r), which is the probability of a sentiment given by a document. A document is classified as negative if P(+|r) < P(−| r) and vice versa. Determining sentiment is important which is calculated using two formulas in this paper. In the first formula, Ps|r=Ns,r/Nr where Ns,r is the number of words with sentiment s in document r and Nr is the number of words in document r. In the second formula, Ps|r=Fs,r/Fr where Fs,r is the effect of words with sentiment s in document r and Fr is equal to the sum of the effect of words with different sentiments in document r. In all evaluations, accuracy1 is calculated based on the first formula, and accuracy2 is calculated based on the second formula. As shown in Figure 8, the document is negative according to the first formula, and the document is positive according to the second formula, and the weight of positive words is more than negative ones, although the number of negative words is more than positive ones, and positive words can affect sentiment analysis at document-level.Figure 8
An example of calculating the sentiment of a document using two formulas.In this section, the best values for each method (the highest accuracy, the lowest perplexity, and the highest topic_coherency) are selected from Tables9–12 and are listed in Tables 6 and 7. Table 6 compares the models based on four datasets (Android, Automotive, Movie, and Electronic) and Table 7 compares the models based on six datasets (Magazine, Sports, MR, Amazon, IMDB, and Yelp). The results of Tables 6 and 7 are evaluated on unigram words. AFINN method classifies each document according to the Ps|r=Ns,r/Nr where the word sentiment label is directly obtained from the AFINN sentiment lexicon. The RND method classifies each document according to the Ps|r=Ns,r/Nr where the word sentiment label is determined randomly, and in the AFINN + RND method, the algorithm uses both AFINN and RND methods. The improvement over these methods will reﬂect how much the proposed methods and baseline methods can learn from a dataset. The report in Tables 6 and 7 shows that the proposed models perform better than JST. Based on the results, the proposed methods have a signiﬁcant improvement over AFINN and the baseline methods on all datasets. As seen from AFINN-based methods results, the results calculated based on the sentiment lexicon are below 70% for most datasets. In this study, parameters perplexity and topic_coherency are not calculated for AFINN, RND, and AFINN + RND methods. TS and RJST methods have lower accuracy than other methods on all datasets, but JST and TJST achieve better performance. As can be seen from the results, TJST outperforms JST on all datasets because, in JST, the distribution θ depends on the document, but in TJST, the distribution θ does not depend on the document and is generally estimated because it uses all documents for computations. According to Tables 6 and 7, WJST1 has higher accuracy than other methods. WJST1 outperforms WJST because, in WJST, the distributions θ,ξ,andψ depend on the document, but in WJST1, the distributions θ,ξ,andψ do not depend on the document and are generally estimated because they use all documents for computations. The perplexity value varies on different datasets because the size of datasets is different, according to Table 4.The analysis of the Friedman test on the results of Tables6 and 7 demonstrates that there is a statistically significant difference between the performances of the algorithms in terms of accuracy with χ210=92.091andp<0.01, in terms of perplexity with χ25=38.629andp<0.01, and in terms of topic_coherency with χ25=5.508andp>0.1. The mean rank of the algorithms based on the Friedman test, which is demonstrated in Figure 9, indicates that WJST1 ranks first among all the algorithms in 7 in terms of accuracy and topic_coherency. According to Figure 9, if the experiment intends to find the minimum value (perplexity), the Friedman test assigns the lowest rank to the best-performing algorithm. If the problem intends to find the maximum value (accuracy and topic_coherency), the Friedman test assigns the highest rank to the best-performing algorithm. According to Figure 9, -1 is accuracry1 and -2 is accuracy2. As shown in Figure 10, average values of accuracy, perplexity, and topic_coherency are equal to the average values in each column of Tables 6 and 7 for each method, in which the values are calculated on Android, Automotive, Electronic, Movie, Magazine, Sport, MR, Amazon, IMDB, and Yelp datasets. According to the results, WJST has a lower perplexity value than other methods. WJST1 outperforms WJST and baseline methods in terms of accuracy and topic_coherency. According to Figure 10, -1 is accuracry1 and -2 is accuracy2.Figure 9
According to the Friedman test, the mean rank of algorithms in Tables6 and 7.Figure 10
Average of sentiment classification values calculated on Android, Automotive, Electronic, Movie, Magazine, Sport, MR, Amazon, IMDB, and Yelp datasets (based on Tables6 and 7), in terms of accuracy (a), perplexity (b), and topic_coherency (c).
(a)(b)(c)


### 4.4. Evaluation Results According to the Different Situations, with AFINN and NO_AFINN States

In this section, the study aims to examine the impact of the AFINN dictionary in the initialization part of Gibbs sampling on the proposed models. The results of the evaluation are shown in Table8. In this section, the number of distinct windows is three, and the number of topics is ten. The most effective is visible in WJST1 on the Movie dataset, where the accuracy in the NO_AFINN state is equal to 0.48 and is equal to 0.95 in the AFINN state. Prior sentiment information affects perplexity and topic_coherency lower than accuracy. According to Table 8, it can be seen that using the AFINN dictionary is more effective than using the NO_AFINN state. In the NO_AFINN state, prior sentiment information was not incorporated into the models for sentiment words in the initialization section of the Gibbs sampling algorithm.


### 4.5. Evaluation Results According to the Different Sentiment Dictionaries

In this subsection, the study compares different dictionaries achieved by the proposed models. The output of WJST and WJST1 can be a weighted sentiment dictionary. Using the obtained dictionary by each method on each dataset, other datasets will be evaluated. Each document will be classified according toP(s|r), where the word sentiment label is directly obtained from the dictionary. Tables 20 and 21 are related to WJST and WJST1, respectively. The impact of using different dictionaries achieved by WJST and WJST1 is presented in Tables 20 and 21. The methods AFINN + w, Android + w, ELEC + w, Auto + w, and MOV + w classify each document according to Ps|r=Fs,r/Fr, where the weight and sentiment label is directly obtained from AFINN, Android, Electronic, Automotive, and Movie lexicons, and window size is considered to be one for all words in all documents. Methods Bing_Liu, 8K, Android, Automotive, ELEC, MOV, and IMDB classify each document according to Ps|r=Ns,r/Nr where the word sentiment label is directly obtained from Bing_Liu, 8K, Android, Automotive, Electronic, Movie, and IMDB lexicons, respectively. The Bing_Liu + RND method uses both Bing_Liu and RND methods. In the IMDB + RND method, the algorithm uses both IMDB and RND methods. The 8K + RND method utilizes both 8K and RND methods. In the Android + RND method, the algorithm uses both Android and RND methods. In the Auto + RND method, the algorithm uses both Auto and RND methods. In the ELEC + RND method, the algorithm uses both ELEC and RND methods. In the MOV + RND method, the algorithm uses both MOV and RND methods. According to Table 20, the AFINN method achieves the highest accuracy on one dataset. Proposed methods achieve the highest accuracy on six datasets. According to the results in Table 21, the proposed methods achieve the highest accuracy on seven datasets. In AFINN, Bing Liu, IMDB, and 8-k dictionaries, sentiment and score values are set manually for each word, but proposed models use topic modeling to generate dictionaries. Proposed methods such as MOV and ELEC perform well on the datasets on which they are created based on. The dictionaries achieved by the proposed models are dependent on the application domain.Table 20
Sentiment classification using different sentiment dictionaries achieved by WJST.
Model\datasetAndroidAutoELECMOVSTSSandersSOMDRND0.480.49250.4650.5250.53460.48520.4847AFINN0.69750.6250.6750.5950.7340.6740.3395AFINN + w0.6850.580.63750.595———AFINN + RND0.580.5350.520.5550.60380.54240.4475Bing_Liu0.69750.640.6950.6350.6980.68130.322Bing_Liu + RND0.57750.5350.5350.5650.62880.54160.4388IMDB0.70250.590.63750.64250.55120.59880.4617IMDB + RND0.61250.48250.54750.59750.55950.51960.47488K0.54250.50250.50750.50250.49860.53510.39958K + RND0.5150.480.50750.510.53730.4950.4814Android0.85250.620.6450.61250.60230.60780.4712Android + w0.83750.590.630.62———Android + RND0.85250.62750.66750.610.59950.58650.4832Auto0.570.7050.63750.6050.60230.57670.6011Auto + w0.57750.70.650.6075———Auto + RND0.56250.7050.61750.58750.61060.55220.588ELEC0.65250.59750.84250.49250.60230.60290.5323ELEC + w0.66250.6050.82750.5025———ELEC + RND0.6450.60750.84250.47750.60230.60370.5585MOV0.660.59250.63750.810.62440.5350.5127MOV + w0.67250.590.64750.8025———MOV + RND0.65250.58750.620.810.62170.54160.5105STS0.510.580.5750.6150.6240.47620.5312STS + RND0.55750.56250.52750.57750.6240.51380.54Sanders0.590.60.5650.55750.56070.70880.5105Sanders + RND0.59250.59250.58250.54250.57460.70880.5443SOMD0.55250.52750.470.46250.48590.53340.6093SOMD + RND0.53750.50.48750.47250.54410.52360.6093Table 21
Sentiment classification using different sentiment dictionaries achieved by WJST1.
Model\datasetAndroidAutoELECMOVSTSSandersSOMDRND0.480.49250.4650.5250.53460.48520.4847AFINN0.69750.6250.6750.5950.7340.6740.3395AFINN + w0.6850.580.63750.595———AFINN + RND0.580.5350.520.5550.60380.54240.4475Bing_Liu0.69750.640.6950.6350.6980.68130.322Bing_Liu + RND0.57750.5350.5350.5650.62880.54160.4388IMDB0.70250.590.63750.64250.55120.59880.4617IMDB + RND0.61250.48250.54750.59750.55950.51960.47488K0.54250.50250.50750.50250.49860.53510.39958K + RND0.5150.480.50750.510.53730.4950.4814Android0.8550.60250.5650.650.58570.58650.4985Android + w0.8350.60.60250.6625———Android + RND0.8550.590.5750.61250.59670.55790.505Auto0.63750.7450.61750.630.5940.62330.4766Auto + w0.63250.73750.650.635———Auto + RND0.63750.7450.620.62250.58290.61180.4875ELEC0.650.54750.70750.58250.5940.61920.457ELEC + w0.630.57750.71250.63———ELEC + RND0.64250.55750.70750.57750.61340.59140.4744MOV0.630.55250.58250.63750.56630.57670.4603MOV + w0.64250.54750.64250.94———MOV + RND0.6350.5650.610.63750.57180.5710.4493STS0.5850.54750.5950.57750.74310.59390.4231STS + RND0.58750.5650.59750.57250.74310.57430.4395Sanders0.6050.5550.60.53750.66320.75380.421Sanders + RND0.60750.550.60250.53750.67710.75380.4559SOMD0.6150.5550.550.51250.57730.60780.594SOMD + RND0.610.52750.54250.53250.5940.57670.594The analysis of the Friedman test on the results of Table20 demonstrates that there is a statistically significant difference between the performances of competitors in terms of accuracy with χ227=70.070andp<0.01. The analysis also shows that there is a statistically significant difference between the performances of the algorithms in Table 21 in terms of accuracy with χ227=79.740andp<0.01. The mean rank of the algorithms can be seen in Figure 11. As shown in Figure 12, Average1 is equal to the average of values in each row for each method, in which the values are calculated on datasets Android. Automotive, Electronic, Movie, STS, Sanders, and SOMD based on Tables 20 and 21. Furthermore, Average2 is equal to the average of values in each row for each method, in which the values are calculated on Android, Automotive, Electronic, and Movie based on Tables 20 and 21. According to the results, Bing_Liu achieves the highest value on column Average1. Furthermore, the Android, MOV, and Bing_Liu methods have higher accuracy than other methods.Figure 11
The mean rank according to the Friedman test based on results of Tables20 and 21.Figure 12
Average accuracy values are calculated using different sentiment dictionaries on Android, Automotive, Electronic, Movie, STS, Sanders, and SOMD datasets (based on Tables20 and 21).


### 4.6. Evaluation Results According to the Different Number of Topics

In this subsection, the proposed models are examined based on the different topics (5, 10, 15, and 20). The AFINN dictionary is utilized in methods, and the number of distinct windows is three. Evaluations results are shown in Tables9–12. The proposed methods are better than the baseline methods based on the results. The results show that increasing the number of topics will decrease the perplexity value. WJST1 achieves the highest accuracy on the Movie dataset with 97 percent, but the highest accuracy value on the Movie dataset in WJST is equal to 84 percent. Results show that the proposed methods perform better with different topic number settings, especially for WJST1 with 97% accuracy at |Z| = 5 on the Movie dataset. Based on the results, WJST has a lower perplexity than other methods. WJST1 outperforms WJST and baseline methods in terms of accuracy and topic_coherency. TJST performs better than the WJST method in terms of accuracy, but WJST achieves higher accuracy than JST and other baseline methods. This observation shows that modeling the parameters weight and window improves sentiment classification at the document-level. According to (18), a lower topic_coherency value suggests that the retrieved subjects are of worse quality than one with a highertopic_coherency. The words in a subject accurately describe the subject and have a stronger association with one another.


### 4.7. Evaluations Results According to the Different Number of Distinct Windows

In this subsection, the proposed models are evaluated according to the different number of separate windows (1, 2, 3, 4, 5, and 6), which are effective for improving the proposed models. In this experiment, the number of topics is five, and the models use the AFINN sentiment dictionary. Based on Table13, the proposed methods are compared according to accuracy, perplexity, and topic_coherency. The results show that increasing the number of distinct windows will decrease the perplexity value. In the report in Table 13, an increase in the size of a window will reduce the accuracy because it will increase the number of words in the window, and each term may not affect all neighbors in its window.For instance, as shown in Figure13, the word terrible has a window size equal to 3. In Table 13, it is assumed that each word affects all neighbors in its window, so Table 13 takes that the unigram terrible effects unigrams film, last, season, sophie, best, and actress. As shown in Figure 13, the word terrible can affect unigrams film, last, and season, but it is not about unigrams sophie, best, and actress. So, finding the words that can be involved in a window is a new challenge that we introduce in this study, and two methods are presented. The first method assumes that each word affects all neighbors in its window, and the second method assumes that each word affects some random neighbors in its window. So, the first method selects all neighbors, but the second method selects some neighbors randomly. In this study, all evaluations are calculated based on the first method, and the second method is considered for the evaluation in Table 14.Figure 13
An example for showing the effect of each word on neighbors in its window.As shown in Table13, accuracy, perplexity, and topic_coherency values are calculated on Android, Automotive, Electronic, and Movie datasets before random selection using the first method. As shown in Table 14, accuracy, perplexity, and topic_coherency values are calculated on Android, Automotive, Electronic, and Movie datasets after random selection using the second method. So, in Table 13, it is assumed that each word affects all neighbors in its window, but in Table 14, it is assumed that each word affects some random neighbors in its window. As shown in Tables 13 and 14, average values of accuracy, perplexity, and topic_coherency are equal to average values in each column of Tables 13 and 14 for each window size. The values are calculated on Android Automotive, Electronic, and Movie datasets. According to the results, the second method is more stable than the first method in terms of accuracy, but the first method has higher accuracy than the second method. The second method outperforms the first method in terms of perplexity. The first method performs better than the second in terms of accuracy and topic_coherency.The analysis of the Friedman test on the results of Table13 demonstrates that there is a statistically significant difference between the performances of the algorithms in terms of accuracy with χ25=27.608andp<0.01, in terms of perplexity with χ25=35.143andp<0.01, and in terms of topic_coherency with χ25=6.232andp=0.284. The mean rank of the algorithms based on the Friedman test, which is demonstrated in Figure 14, indicates that (w = 1) outperforms other windows in terms of accuracy. Still, it has lower perplexity and topic_coherency values than other windows. (w = 6) outperforms other windows in perplexity and topic_coherency, but it has a lower accuracy value than other windows. (w = 3) provides a special situation for proposed algorithms in which accuracy, topic_coherency, and perplexity values are between the highest and lowest values (w = 1, 2, 4, 5, and 6). So, in this study, all evaluations are calculated based on (w = 3). The mean rank of the algorithms based on Table 14, which is demonstrated in Figure 15, indicates that (w = 1) ranks first among all the algorithms in Table 14 in terms of accuracy. The analysis of the Friedman test indicates that there is a statistically significant difference in terms of accuracy with χ25=17.550andp<0.01, in terms of perplexity with χ25=37.857andp<0.01, and in terms of topic_coherency with χ25=5.857andp=0.320.Figure 14
The mean rank of the seven algorithms is in Table13 according to the Friedman test.Figure 15
The mean rank of algorithms is in Table14 according to the Friedman test.


### 4.8. Sentiment Classification Using Proposed Methods in Comparison to ALGA

In this subsection, WJST and WJST1 are compared to ALGA [46]. Three datasets have been selected for evaluating the methods. Evaluations results are shown in Figure 16, which compares the results of models with each other according to accuracy, perplexity, and topic_coherency metrics. In ALGA [46], several sentiment lexicons are created for a dataset during the training stage using a genetic algorithm. During the testing process, these dictionaries are employed. Every dictionary has some words and scores. Each chromosome is represented as a vector of sentiment words and their scores in the genetic algorithm employed in the method. The scores are spread between a feeling word's lowest and maximum scores. The primary goal of ALGA is to create a dictionary that reduces errors on training datasets. The sum of scores for words of each instance Ti in dataset Dm using dictionary Lk is calculated using equation (19) and is treated as a feature [46]:(19)ALGADm,Ti,Lk=∑Wj∈TivkWj.Figure 16
Sentiment classification on SANDERS, SOMD, and STS datasets, in terms of accuracy (a), perplexity (b), and topic_coherency (c).
(a)(b)(c)Finding the values of words in the dictionaries (chromosomes) and adding them together is how the ALGA value for each instance is calculated. In (19), Wj represents the words of Ti, and vkWj shows the score of Wj in Lk. As mentioned in [46], ALGA will predict a positive instance when the ALGA feature is positive and a negative instance when the ALGA feature is negative. By dividing the number of correct predictions of instances of a given dataset by the total cases, ALGA's accuracy is calculated. In this subsection, proposed methods are compared with ALGA [46] because it can automatically generate a sentiment dictionary. ALGA generates a sentiment dictionary using the genetic algorithm, but proposed methods generate a sentiment dictionary using topic modeling. In proposed models, each document is classified based on P(s| r), the probability of sentiment label given a document. In proposed models, two labels (+, −) are considered, and a document is classified as negative if P(+|r) < P(−|r) and vice versa. Evaluations results can be seen in Figure 16. In this subsection, the number of distinct windows is three, and the number of topics is five. The models use the AFINN sentiment dictionary. According to Figures 16(a)–16(c), each column compares different methods on a dataset. The details of the datasets used in this section are illustrated in Table 4. In this subsection, only the accuracy is considered for the evaluation of ALGA. The ALGA-SW value is achieved by executing ALGA without taking stopwords into account. According to the results, WJST has higher accuracy than TJST on all datasets. WJST1 outperforms WJST and TJST on all datasets. ALGA and ALGA-SW perform better than other methods in terms of accuracy, but WJST1 achieves higher accuracy than ALGA and ALGA-SW on STS and Sanders datasets. The RND method achieves the lowest accuracy value on Sanders and STS datasets. The AFINN + RND method has higher accuracy than the RND method and has a lower accuracy than the AFINN method on Sanders and STS datasets. The RND method outperforms TJST, AFINN, and AFINN + RND methods on the SOMD dataset. In this study, parameters perplexity and topic_coherency are not calculated for ALGA, ALGA-SW, AFINN, RND, and AFINN + RND methods. According to Figure 16, -1 is accuracry1, and -2 is accuracy2.


### 4.9. Sentiment Classification Using Proposed Methods on Multidomain Datasets

In this subsection, the performance of the proposed methods is compared with baseline methods on a multidomain dataset. In this experiment, the number of distinct windows and topics is three and five, respectively, and the models use the AFINN sentiment dictionary. The multidomain dataset contains reviews taken from multiple domains (product types). The details of the multidomain dataset used in this section are illustrated in Table22.Table 22
Description of the multidomain dataset used in this section.
#DomainsNumber of reviewsVocabulary sizeNumber of words1Android, Automotive, Electronic, and Movie160011183100113As shown in Figure17, accuracy, perplexity, and topic_coherency values are calculated on a multidomain dataset that contains Android. Automotive, Electronic, and Movie domains. The methods WJST-dictionary and WJST1-dictionary classify each document according to Ps|r=Ns,r/Nr. The word sentiment label is directly obtained from WJST and WJST1 lexicons achieved by WJST and WJST1 on the multidomain dataset. According to Figure 17, -1 is accuracry1 and -2 is accuracy2. Based on the results, WJST has a lower perplexity value than other methods. WJST1 outperforms WJST in terms of topic_coherency. WJST performs better than the WJST1 method in terms of accuracy because the distributions θ,ξ,andψ in WJST depend on document, but in WJST1, the distributions θ,ξ,andψ do not depend on the document. Dependency between documents of a domain is more than documents in different domains. A pattern in the documents of a domain may not exist in the documents of other domains. Therefore, calculations on multidomain datasets should be local and not cover all domains. For example, considering the distributions Pz|s and Pz|s,r, where z is topic, s is the sentiment, and r is documents. In the first state (Pz|s), the topic depends on sentiment, and the distribution covers all documents in different domains. Perhaps a topic was positive in one domain and negative in another. Therefore, it is better to depend the topic on the documents of a domain, not all domains. Thus, the topic is limited to the document (and domain), and contradiction between different domains is eliminated. Therefore, WJST is suitable for multidomain datasets, and WJST1 is a version of WJST suitable for single-domain datasets. Sentiment classification on multidomain datasets is a challenge, and our solution in this study is using WJST, whose distributions (θ,ξ,andψ) depend on the document. Sentiment classification on multidomain datasets is a challenge, and further studies can be conducted to investigate this problem for future research.Figure 17
Sentiment classification values are calculated on a multidomain dataset in terms of accuracy (a), perplexity (b), and topic_coherency (c).
(a)(b)(c)


### 4.10. Comparison with Other Methods

In this subsection, the best performance of the proposed methods is compared with 57 competitors [13, 76];  [82–88], [8–10, 46] which is shown in Table 23. The details of the datasets used in this section are illustrated in Table 4.Table 23
Sentiment classification on MR, Sanders, SOMD, STS, Amazon, IMDB, and Yelp datasets.
Method/datasetMRSandersSOMDSTSAmazonIMDBYelpALGA [46]—0.80670.81470.7668———ALGA-SW [46]—0.78680.78770.7886———BPSO (Shang et al., 2016)————0.74390.790.789BICA (Mirhosseini et al., 2017)————0.7930.7450.763BABC (Schiezaro et al., 2013)————0.75090.740.736MaxEnt (Saif et al., 2014)—0.8362—0.7782———NB (Saif et al., 2014)—0.8266—0.8106———LS-all [13]—0.8199—————SVM-all [13]—0.8214—————RMTL [13]—0.827875—————MTL-graph [13]—0.801725—————CMSC [13]—0.846325—————LSTM-all [13]—0.8063—————MTL-CNN [13]—0.829825—————MTL-DNN [13]—0.817—————ASP-MTL [13]—0.85125—————NeuroSent [13]—0.834575—————DAM [13]—0.863225—————SVM-BoW (Da Silva et al., 2014)—0.82430.7402————SVM-BoW + lex (Da Silva et al., 2014)—0.83980.7893————RF-BoW (Da Silva et al., 2014)—0.79240.7391————RF-BoW + lex (Da Silva et al., 2014)—0.82350.7936————LR-BoW (Da Silva et al., 2014)—0.77450.7238————LR-BoW + lex (Da Silva et al., 2014)—0.79490.7806————MNB-BoW (Da Silva et al., 2014)—0.79820.7543————MNB-BoW + lex (Da Silva et al., 2014)—0.83410.8013————ENS(LR + RF + MNB)-BoW (Da Silva et al., 2014)—0.82760.7555————ENS(LR + RF + MNB)-BoW + lex (Da Silva et al., 2014)—0.84890.8035————SVM-FH (Da Silva et al., 2014)—0.49750.5131————SVM-FH + lex (Da Silva et al., 2014)—0.75000.6299————RF-FH (Da Silva et al., 2014)—0.55640.6136————RF-FH + lex (Da Silva et al., 2014)—0.71630.7260————LR-FH (Da Silva et al., 2014)—0.56940.6529————LR-FH + lex (Da Silva et al., 2014)—0.75980.7303————MNB-FH (Da Silva et al., 2014)—0.54250.6070————MNB-FH + lex (Da Silva et al., 2014)—0.75080.7139————ENS(LR + RF + MNB)-FH (Da Silva et al., 2014)—0.57840.6517————ENS(LR + RF + MNB)-FH + lex (Da Silva et al., 2014)—0.76630.7456————WS-TSWE’ [76]0.841——————WS-TSWE [76]0.824——————TSWE-P [76]0.726——————TSWE + P [76]0.782——————JSTH [76]0.681——————HTSM [76]0.796——————SAE (Pagliardini et al., 2018)0.861——————ParagraphVec DBOW (Pagliardini et al., 2018)0.763——————ParagraphVec DM (Pagliardini et al., 2018)0.764——————IST (Pu et al., 2019)0.827——————UST (Pu et al., 2019)0.832——————UIST (Pu et al., 2019)0.845——————RND0.48950.48520.48470.53460.4910.4980.506AFINN0.6010.6740.33950.7340.7310.6980.689AFINN + RND0.54550.54240.44750.60380.5740.5750.559JST [8]0.6130.61760.49340.63980.6110.6050.579TJST [8]0.620.58980.46390.68140.6450.6160.614RJST [12]0.510.60860.49670.62320.6090.5460.561TS [9]0.50.56120.52510.5650.540.5450.547WJST-10.8210.7268.757860.71260.7790.760.737WJST-20.8180.72030.757750.71810.8290.7610.726WJST1-10.84450.8320.80.83170.7960.770.773WJST1-20.8430.83520.78590.83730.7980.7740.769


### 4.11. Comparison with Discriminative Models

The proposed methods are compared to baseline approaches such as logistic regression and SVM on four datasets in the following experiment. The multidomain dataset contains Android, Automotive, Electronic, and Movie domains. As shown in Table24, the accuracy value is calculated on four datasets. The results demonstrate that proposed methods have improved notably over AFINN and the baseline methods on all datasets. Based on Table 24, methods use two systems in preprocessing phase, which includes Bag of Word (BOW) and Term Frequency Inverse Document Frequency (TF-IDF). In the BOW system, more word frequency reflects more importance of the word. TF-IDF system believes that high frequency may not be able to provide much information. Furthermore, rare words contribute more weight to the method. According to evaluation results, the results of the TF-IDF system are better than the BOW system.Table 24
Sentiment classification in comparison with discriminative models on different datasets.
Method\DatasetAndroidAutomotiveElectronicMovieRND0.480.49250.4650.525AFINN0.69750.6250.6750.595RND + AFINN0.580.5350.520.555LOGISTIC REGRESSION(BOW)0.530.520.53380.57RANDOMFOREST(BOW)0.690.650.600.6575SVM(BOW)0.490.510.540.5675DECISIONTREE(BOW)0.710.700.560.665NAIVE_BAYES(BOW)0.510.550.47120.555KNEIGHBORS (N = 3, BOW)0.550.570.550.5725KNEIGHBORS (N = 4,BOW)0.560.5750.59150.585KNEIGHBORS (N = 5,BOW)0.570.57250.550.59KNEIGHBORS (N = 6,BOW)0.570.56750.560.61LOGISTIC REGRESSION (TF-IDF)0.610.630.830.59RANDOMFOREST (TF-IDF)0.5750.60.72430.5225SVM (TF-IDF)0.58250.590.800.54DECISIONTREE (TF-IDF)0.550.570.70430.55NAIVE_BAYES (TF-IDF)0.580.600.79450.58KNEIGHBORS (N = 3, TF-IDF)0.650.630.530.7125KNEIGHBORS (N = 4, TF-IDF)0.650.630.560.7125KNEIGHBORS (N = 5, TF-IDF)0.650.630.510.7125KNEIGHBORS (N = 6, TF-IDF)0.650.57250.530.7125JST0.6250.65750.70250.7575ASUM0.6130.63220.710.772TJST0.7650.76750.760.9475RJST0.58250.6150.55250.62TS0.54250.55250.54750.5425WJSTAccuracy10.7950.7550.86250.8475Accuracy20.78250.74750.8750.8325WJST1Accuracy10.8650.80.84750.97Accuracy20.85250.7950.8550.9675


### 4.12. Comparison with JST According to Extended Features

Suppose a unigram corresponds to the sentiment lexicon. In that case, its polarity will be equal to the subjectivity of the lexicon in order to identify the emotion label of the unigram for trying to prepare prior emotion information. The following technique is used to decide the emotion label of a bigram to prepare prior emotion information: If words of the bigram have the same polarity, the bigram’s polarity will be the same as that of the words. If one of the words is in the lexicon, the bigram’s polarity will equal the lexicon’s subjectivity. The bigram’s polarity will be opposed to the polarity of the second word if the first word is ‘not.’ The following methodology is used to decide the emotion label of a trigram in order to prepare prior emotion information. If words of the trigram have the same polarity, the trigram’s polarity will be the same as that of the words. If one of them is in the lexicon, the trigram’s polarity will equal the lexicon’s subjectivity. The trigram’s polarity will be opposed to the second or third word’s polarity if the first or second word is ‘not.’ The proposed methods are compared with JST on four datasets (single-domain) according to extended features (bigrams and trigrams) in the following experiment.As shown in Table25, the accuracy value is calculated on four datasets, and the experiment extends the features to bigrams and trigrams. According to the results, WJST1 outperforms WJST. According to evaluations results, proposed models outperform JST because the additional parameters can influence the process of producing words in a review appropriately. The perplexity value varies on different datasets because the size of datasets is different. As the number of grams increases, perplexity is increased because in each document, in addition to the unigrams (+bigrams), bigrams (+trigrams) are added to the data, and the size of the dataset is increased. As the number of grams increases, accuracy is improved. In some cases, it gets worse because higher grams (bigram or trigrams) are sometimes meaningless.Table 25
Sentiment classification according to extended features (bigrams and trigrams) on different datasets (single-domain).
ModelGramMetric\datasetAndroidAutomotiveElectronicMovieJSTU∗Accuracy0.6250.62750.6750.7575Perplexity19.672624.715425.22727.3111Topic_Coh−4.6026−1.486−1.5892−1.0123U + B∗Accuracy0.6550.70250.750.835Perplexity66.188286.689781.337598.9201Topic_Coh−2.4595−1.0417−0.9823−0.4165U + B + T∗Accuracy0.67750.58250.75250.7275Perplexity114.8812158.7113156.1383190.8403Topic_Coh−1.5776−2.3915−0.7030−0.1288WJSTUAccuracy10.79250.7550.86250.7225Accuracy20.77750.74750.8750.71Perplexity16.730321.200820.548923.1145Topic_Coh−0.5547−1.6542−2.0442−0.0751U + BAccuracy10.68250.630.670.65Accuracy20.680.6250.66750.6475Perplexity36.550549.858448.200858.0838Topic_Coh−1.3976−4.0214−1.2513−0.0423U + B + TAccuracy10.73250.68250.6350.6475Accuracy20.73250.68250.6150.6325Perplexity52.920572.762976.254588.1860Topic_Coh−1.3242−1.5525−1.2409−2.5170WJST1UAccuracy10.810.800.790.97Accuracy20.79250.790.800.9625Perplexity16.678721.435722.601224.6302Topic_Coh−0.187−1.1883−1.1683−0.1348U + BAccuracy10.70.76250.85750.935Accuracy20.70.76250.85250.935Perplexity38.560349.516754.704162.3459Topic_Coh−2.0962−1.8607−0.9744−0.0617U + B + TAccuracy10.830.76250.7450.9475Accuracy20.82750.76750.7450.9475Perplexity55.581975.362477.666096.3675Topic_Coh−1.3837−0.1840−0.9334−0.0333U∗: unigram. B∗: bigram. T∗: trigram.


### 4.13. Discussions on the Limitations of the Proposed Methods

Although the analysis of the results of the evaluation can demonstrate the best performance of proposed methods, proposed methods have some limitations, as follows:(1)
The first limitation is the time complexity of the proposed methods (O(G·wALL·|S|·|Z|·|Q|·|E|)) which is more than baseline methods (O(G·wALL·|S|·|Z|)) according to Table 3 in Section 3.3.(2)
The second limitation is the window size. On the report in Table13, an increase in the size of a window will decrease the accuracy because it will increase the number of words in the window, and each term may not affect all neighbors in its window. Therefore, finding the words that can be involved in a window is a new challenge that we introduce in this study, and two methods are presented in Section 4.7. The first method assumes that each word affects all neighbors in its window, and the second method assumes that each word affects some random neighbors in its window.Therefore, the first method selects all neighbors, but the second method selects some neighbors randomly. According to the results, the second method is more stable than the first method in terms of accuracy, but the first method has a higher accuracy value than the second method. The second method outperforms the first method in terms of perplexity. The first method performs better than the second in terms of accuracy and topic_coherency.


### 4.14. A Concise Description of the Proposed Solutions and the Results

The main problem in this study is to examine a user’s opinion about a product or movie, for example. This means identifying whether a user has a positive or negative idea about a subject (a product or movie).Two novel models have been proposed that use topic modeling to solve the above problem. So, our solution is using a technique named topic modeling. Proposed models extend and improve JST (as a topic model) through two new parameters. To improve JST, proposed models consider the effect of words on each other. The new parameters have an immense effect on model accuracy regarding evaluation results. According to evaluations results, the proposed models outperform JST because the additional parameters can influence the process of producing words in a review appropriately. They can improve sentiment classiﬁcation at the document-level. Also, in the evaluation results report, proposed methods are more accurate than discriminative models such as SVM and logistic regression. Proposed methods are more flexible than discriminative models because other information, such as the top 10 words, can be extracted from the heart of the data.


## 4.1. Sentiment Scores for the Words in a Dataset

In this section, a dictionary is generated, including sentiment scores (weights and sentiment labels) and words. The scores are extracted from datasets based on P(w| s, e). The weight and sentiment with the most probability are selected for each word as a sentiment score. The extracted scores for some phrases in the form of unigram can be seen in Tables 15 and 16. ALGA [46] uses the genetic algorithm to generate a sentiment dictionary; however, we use topic modeling in the proposed models to create this dictionary. According to Tables 15 and 16, ten words from each dataset are selected and scored by the proposed models. For example, the word nice obtains a score of 4 in WJST and obtains a score of 5 in WJST1. The scores are different in the proposed methods; for example, the word serious achieves a score of 1 in WJST and a score of -2 in WJST1. Table 15 is related to Android, Automotive, Electronic, and Movie datasets. Table 16 is associated with STS, Sanders, and SOMD datasets.Table 15
Sentiment scores, for some instance, words related to Android, Automotive, Electronic, and Movie datasets.
DatasetAndroidAutomotiveElectronicMovieModelWordScoreWordScoreWordScoreWordScoreWJST1Nice5Much5Satisfy5See5Cute4Use4Crew4Father4Favorit3Long3Way3Pray3Perfect2Expens2Fluid2Human2Great1Stuff1Feel1Event1Type−1Fals−1Pull−1Terribl−1Wast−2Serious−2Side−2Sens−2Everi−3Extens−3Nervous−3Lost−3Unknown−4Space−4Even−4Sure−4Everyth−5Know−5Extend−5Injur−5WJSTNice4Much3Satisfy−2See−4Cute1Use2Crew2Father5Favorit2Long4Way4Pray−3Perfect2Expens−4Fluid1Human5Great2Stuff−5Feel−3Event1Type1Fals−5Pull−4Terribl4Wast−5Serious1Side−1Sens−4Everi4Extens5Nervous3Lost3Unknown5Space−5Even−2Sure2Everyth2Know4Extend−5Injur−5Table 16
Sentiment scores for some instance words related to STS, Sanders, and SOMD datasets.
DatasetSTSSandersSOMDModelWordScoreScoreScoreWJST1Much−442Good54−5Bad−5−1−5Nice−355Hate−5−51Love553WJSTMuch−5−5−2Good−4−13Bad−4−13Nice321Hate−5−42Love4−2−3


## 4.2. Topic Discovery

The topics are extracted from datasets based onP(w|z) in this section. A topic is a multinomial distribution over words based on topics, sentiments, weights, and window sizes. The top words could approximately reflect the meaning of a topic. Tables 17–19 show some examples of topics extracted from Movie, Android, and Electronic datasets by different models. Each row shows the top 10 words for the corresponding topic and sentiment label. The top 10 words from each topic were extracted and then used for topic_coherency. Extracting topic words is one of the main tasks in topic modeling. This section lists the top 10 words in three examples for Movie, Android, and Electronic datasets. The listed words for each topic describe the topic. The listed words for the proposed methods have a better topic_coherency value than baseline methods because they have a higher value of topic_coherency. The higher the topic_coherency value, the better the quality of the subjects extracted.Table 17
Top 10 words extracted from the Movie dataset.
ModelSentimentTop 10 wordsWJST+jesu, film, God, love, mel, Christian, life, suffer, believ, roman−movi, godzilla, bad, dvd, origin, horror, buy, version, worst, actorWJST1+Jesu, mel, passion, mother, stori, realli, great, everyon, God, like−godzilla, monster, go, time, star, know, kill, make, militari, AmericanJST+mel, stori, mother, two, realli, becom, anoth, God, like, back−godzilla, look, monster, american, militari, like, worst, zellweg, emmerich, quitTable 18
Top 10 words extracted from Android dataset.
ModelSentimentTop 10 wordsWJST+app, game, sudoku, play, version, enjoy, option, want, hint, like−work, app, would, fire, live, station, tri, say, select, kindl, load, userWJST1+sudoku, tri, love, game, time, easi, tablet, star, call, make−close, tablet, seem, get, year, download, much, station, time, androidJST+station, want, even, peopl, work, avail, version, puzzl, custom, believ−use, app, find, review, great, got, total, new, night, fakeTable 19
Top 10 words extracted from Electronic dataset.
ModelSentimentTop 10 wordsWJST+read, book, screen, touch, kindl, page, better, wifi, ebook, like−work, went, new, servic, need, bad, system, hous, number, mineWJST1+googl, amazon, book, color, store, kindl, download, small, pdf−time, work, two, much, one, power, comput, phone, go, unitJST+book, touch, read, page, free, librari, touch, screen, much, pdf−plug, work, could, devic, comput, charger, router, cabl, item, design


## 4.3. Sentiment Classification at Document-Level

In this section, the number of distinct windows is three, and the models use the AFINN sentiment dictionary in the initialization section of the Gibbs sampling algorithm. A document is classified based onP(s| r), which is the probability of a sentiment given by a document. A document is classified as negative if P(+|r) < P(−| r) and vice versa. Determining sentiment is important which is calculated using two formulas in this paper. In the first formula, Ps|r=Ns,r/Nr where Ns,r is the number of words with sentiment s in document r and Nr is the number of words in document r. In the second formula, Ps|r=Fs,r/Fr where Fs,r is the effect of words with sentiment s in document r and Fr is equal to the sum of the effect of words with different sentiments in document r. In all evaluations, accuracy1 is calculated based on the first formula, and accuracy2 is calculated based on the second formula. As shown in Figure 8, the document is negative according to the first formula, and the document is positive according to the second formula, and the weight of positive words is more than negative ones, although the number of negative words is more than positive ones, and positive words can affect sentiment analysis at document-level.Figure 8
An example of calculating the sentiment of a document using two formulas.In this section, the best values for each method (the highest accuracy, the lowest perplexity, and the highest topic_coherency) are selected from Tables9–12 and are listed in Tables 6 and 7. Table 6 compares the models based on four datasets (Android, Automotive, Movie, and Electronic) and Table 7 compares the models based on six datasets (Magazine, Sports, MR, Amazon, IMDB, and Yelp). The results of Tables 6 and 7 are evaluated on unigram words. AFINN method classifies each document according to the Ps|r=Ns,r/Nr where the word sentiment label is directly obtained from the AFINN sentiment lexicon. The RND method classifies each document according to the Ps|r=Ns,r/Nr where the word sentiment label is determined randomly, and in the AFINN + RND method, the algorithm uses both AFINN and RND methods. The improvement over these methods will reﬂect how much the proposed methods and baseline methods can learn from a dataset. The report in Tables 6 and 7 shows that the proposed models perform better than JST. Based on the results, the proposed methods have a signiﬁcant improvement over AFINN and the baseline methods on all datasets. As seen from AFINN-based methods results, the results calculated based on the sentiment lexicon are below 70% for most datasets. In this study, parameters perplexity and topic_coherency are not calculated for AFINN, RND, and AFINN + RND methods. TS and RJST methods have lower accuracy than other methods on all datasets, but JST and TJST achieve better performance. As can be seen from the results, TJST outperforms JST on all datasets because, in JST, the distribution θ depends on the document, but in TJST, the distribution θ does not depend on the document and is generally estimated because it uses all documents for computations. According to Tables 6 and 7, WJST1 has higher accuracy than other methods. WJST1 outperforms WJST because, in WJST, the distributions θ,ξ,andψ depend on the document, but in WJST1, the distributions θ,ξ,andψ do not depend on the document and are generally estimated because they use all documents for computations. The perplexity value varies on different datasets because the size of datasets is different, according to Table 4.The analysis of the Friedman test on the results of Tables6 and 7 demonstrates that there is a statistically significant difference between the performances of the algorithms in terms of accuracy with χ210=92.091andp<0.01, in terms of perplexity with χ25=38.629andp<0.01, and in terms of topic_coherency with χ25=5.508andp>0.1. The mean rank of the algorithms based on the Friedman test, which is demonstrated in Figure 9, indicates that WJST1 ranks first among all the algorithms in 7 in terms of accuracy and topic_coherency. According to Figure 9, if the experiment intends to find the minimum value (perplexity), the Friedman test assigns the lowest rank to the best-performing algorithm. If the problem intends to find the maximum value (accuracy and topic_coherency), the Friedman test assigns the highest rank to the best-performing algorithm. According to Figure 9, -1 is accuracry1 and -2 is accuracy2. As shown in Figure 10, average values of accuracy, perplexity, and topic_coherency are equal to the average values in each column of Tables 6 and 7 for each method, in which the values are calculated on Android, Automotive, Electronic, Movie, Magazine, Sport, MR, Amazon, IMDB, and Yelp datasets. According to the results, WJST has a lower perplexity value than other methods. WJST1 outperforms WJST and baseline methods in terms of accuracy and topic_coherency. According to Figure 10, -1 is accuracry1 and -2 is accuracy2.Figure 9
According to the Friedman test, the mean rank of algorithms in Tables6 and 7.Figure 10
Average of sentiment classification values calculated on Android, Automotive, Electronic, Movie, Magazine, Sport, MR, Amazon, IMDB, and Yelp datasets (based on Tables6 and 7), in terms of accuracy (a), perplexity (b), and topic_coherency (c).
(a)(b)(c)


## 4.4. Evaluation Results According to the Different Situations, with AFINN and NO_AFINN States

In this section, the study aims to examine the impact of the AFINN dictionary in the initialization part of Gibbs sampling on the proposed models. The results of the evaluation are shown in Table8. In this section, the number of distinct windows is three, and the number of topics is ten. The most effective is visible in WJST1 on the Movie dataset, where the accuracy in the NO_AFINN state is equal to 0.48 and is equal to 0.95 in the AFINN state. Prior sentiment information affects perplexity and topic_coherency lower than accuracy. According to Table 8, it can be seen that using the AFINN dictionary is more effective than using the NO_AFINN state. In the NO_AFINN state, prior sentiment information was not incorporated into the models for sentiment words in the initialization section of the Gibbs sampling algorithm.


## 4.5. Evaluation Results According to the Different Sentiment Dictionaries

In this subsection, the study compares different dictionaries achieved by the proposed models. The output of WJST and WJST1 can be a weighted sentiment dictionary. Using the obtained dictionary by each method on each dataset, other datasets will be evaluated. Each document will be classified according toP(s|r), where the word sentiment label is directly obtained from the dictionary. Tables 20 and 21 are related to WJST and WJST1, respectively. The impact of using different dictionaries achieved by WJST and WJST1 is presented in Tables 20 and 21. The methods AFINN + w, Android + w, ELEC + w, Auto + w, and MOV + w classify each document according to Ps|r=Fs,r/Fr, where the weight and sentiment label is directly obtained from AFINN, Android, Electronic, Automotive, and Movie lexicons, and window size is considered to be one for all words in all documents. Methods Bing_Liu, 8K, Android, Automotive, ELEC, MOV, and IMDB classify each document according to Ps|r=Ns,r/Nr where the word sentiment label is directly obtained from Bing_Liu, 8K, Android, Automotive, Electronic, Movie, and IMDB lexicons, respectively. The Bing_Liu + RND method uses both Bing_Liu and RND methods. In the IMDB + RND method, the algorithm uses both IMDB and RND methods. The 8K + RND method utilizes both 8K and RND methods. In the Android + RND method, the algorithm uses both Android and RND methods. In the Auto + RND method, the algorithm uses both Auto and RND methods. In the ELEC + RND method, the algorithm uses both ELEC and RND methods. In the MOV + RND method, the algorithm uses both MOV and RND methods. According to Table 20, the AFINN method achieves the highest accuracy on one dataset. Proposed methods achieve the highest accuracy on six datasets. According to the results in Table 21, the proposed methods achieve the highest accuracy on seven datasets. In AFINN, Bing Liu, IMDB, and 8-k dictionaries, sentiment and score values are set manually for each word, but proposed models use topic modeling to generate dictionaries. Proposed methods such as MOV and ELEC perform well on the datasets on which they are created based on. The dictionaries achieved by the proposed models are dependent on the application domain.Table 20
Sentiment classification using different sentiment dictionaries achieved by WJST.
Model\datasetAndroidAutoELECMOVSTSSandersSOMDRND0.480.49250.4650.5250.53460.48520.4847AFINN0.69750.6250.6750.5950.7340.6740.3395AFINN + w0.6850.580.63750.595———AFINN + RND0.580.5350.520.5550.60380.54240.4475Bing_Liu0.69750.640.6950.6350.6980.68130.322Bing_Liu + RND0.57750.5350.5350.5650.62880.54160.4388IMDB0.70250.590.63750.64250.55120.59880.4617IMDB + RND0.61250.48250.54750.59750.55950.51960.47488K0.54250.50250.50750.50250.49860.53510.39958K + RND0.5150.480.50750.510.53730.4950.4814Android0.85250.620.6450.61250.60230.60780.4712Android + w0.83750.590.630.62———Android + RND0.85250.62750.66750.610.59950.58650.4832Auto0.570.7050.63750.6050.60230.57670.6011Auto + w0.57750.70.650.6075———Auto + RND0.56250.7050.61750.58750.61060.55220.588ELEC0.65250.59750.84250.49250.60230.60290.5323ELEC + w0.66250.6050.82750.5025———ELEC + RND0.6450.60750.84250.47750.60230.60370.5585MOV0.660.59250.63750.810.62440.5350.5127MOV + w0.67250.590.64750.8025———MOV + RND0.65250.58750.620.810.62170.54160.5105STS0.510.580.5750.6150.6240.47620.5312STS + RND0.55750.56250.52750.57750.6240.51380.54Sanders0.590.60.5650.55750.56070.70880.5105Sanders + RND0.59250.59250.58250.54250.57460.70880.5443SOMD0.55250.52750.470.46250.48590.53340.6093SOMD + RND0.53750.50.48750.47250.54410.52360.6093Table 21
Sentiment classification using different sentiment dictionaries achieved by WJST1.
Model\datasetAndroidAutoELECMOVSTSSandersSOMDRND0.480.49250.4650.5250.53460.48520.4847AFINN0.69750.6250.6750.5950.7340.6740.3395AFINN + w0.6850.580.63750.595———AFINN + RND0.580.5350.520.5550.60380.54240.4475Bing_Liu0.69750.640.6950.6350.6980.68130.322Bing_Liu + RND0.57750.5350.5350.5650.62880.54160.4388IMDB0.70250.590.63750.64250.55120.59880.4617IMDB + RND0.61250.48250.54750.59750.55950.51960.47488K0.54250.50250.50750.50250.49860.53510.39958K + RND0.5150.480.50750.510.53730.4950.4814Android0.8550.60250.5650.650.58570.58650.4985Android + w0.8350.60.60250.6625———Android + RND0.8550.590.5750.61250.59670.55790.505Auto0.63750.7450.61750.630.5940.62330.4766Auto + w0.63250.73750.650.635———Auto + RND0.63750.7450.620.62250.58290.61180.4875ELEC0.650.54750.70750.58250.5940.61920.457ELEC + w0.630.57750.71250.63———ELEC + RND0.64250.55750.70750.57750.61340.59140.4744MOV0.630.55250.58250.63750.56630.57670.4603MOV + w0.64250.54750.64250.94———MOV + RND0.6350.5650.610.63750.57180.5710.4493STS0.5850.54750.5950.57750.74310.59390.4231STS + RND0.58750.5650.59750.57250.74310.57430.4395Sanders0.6050.5550.60.53750.66320.75380.421Sanders + RND0.60750.550.60250.53750.67710.75380.4559SOMD0.6150.5550.550.51250.57730.60780.594SOMD + RND0.610.52750.54250.53250.5940.57670.594The analysis of the Friedman test on the results of Table20 demonstrates that there is a statistically significant difference between the performances of competitors in terms of accuracy with χ227=70.070andp<0.01. The analysis also shows that there is a statistically significant difference between the performances of the algorithms in Table 21 in terms of accuracy with χ227=79.740andp<0.01. The mean rank of the algorithms can be seen in Figure 11. As shown in Figure 12, Average1 is equal to the average of values in each row for each method, in which the values are calculated on datasets Android. Automotive, Electronic, Movie, STS, Sanders, and SOMD based on Tables 20 and 21. Furthermore, Average2 is equal to the average of values in each row for each method, in which the values are calculated on Android, Automotive, Electronic, and Movie based on Tables 20 and 21. According to the results, Bing_Liu achieves the highest value on column Average1. Furthermore, the Android, MOV, and Bing_Liu methods have higher accuracy than other methods.Figure 11
The mean rank according to the Friedman test based on results of Tables20 and 21.Figure 12
Average accuracy values are calculated using different sentiment dictionaries on Android, Automotive, Electronic, Movie, STS, Sanders, and SOMD datasets (based on Tables20 and 21).


## 4.6. Evaluation Results According to the Different Number of Topics

In this subsection, the proposed models are examined based on the different topics (5, 10, 15, and 20). The AFINN dictionary is utilized in methods, and the number of distinct windows is three. Evaluations results are shown in Tables9–12. The proposed methods are better than the baseline methods based on the results. The results show that increasing the number of topics will decrease the perplexity value. WJST1 achieves the highest accuracy on the Movie dataset with 97 percent, but the highest accuracy value on the Movie dataset in WJST is equal to 84 percent. Results show that the proposed methods perform better with different topic number settings, especially for WJST1 with 97% accuracy at |Z| = 5 on the Movie dataset. Based on the results, WJST has a lower perplexity than other methods. WJST1 outperforms WJST and baseline methods in terms of accuracy and topic_coherency. TJST performs better than the WJST method in terms of accuracy, but WJST achieves higher accuracy than JST and other baseline methods. This observation shows that modeling the parameters weight and window improves sentiment classification at the document-level. According to (18), a lower topic_coherency value suggests that the retrieved subjects are of worse quality than one with a highertopic_coherency. The words in a subject accurately describe the subject and have a stronger association with one another.


## 4.7. Evaluations Results According to the Different Number of Distinct Windows

In this subsection, the proposed models are evaluated according to the different number of separate windows (1, 2, 3, 4, 5, and 6), which are effective for improving the proposed models. In this experiment, the number of topics is five, and the models use the AFINN sentiment dictionary. Based on Table13, the proposed methods are compared according to accuracy, perplexity, and topic_coherency. The results show that increasing the number of distinct windows will decrease the perplexity value. In the report in Table 13, an increase in the size of a window will reduce the accuracy because it will increase the number of words in the window, and each term may not affect all neighbors in its window.For instance, as shown in Figure13, the word terrible has a window size equal to 3. In Table 13, it is assumed that each word affects all neighbors in its window, so Table 13 takes that the unigram terrible effects unigrams film, last, season, sophie, best, and actress. As shown in Figure 13, the word terrible can affect unigrams film, last, and season, but it is not about unigrams sophie, best, and actress. So, finding the words that can be involved in a window is a new challenge that we introduce in this study, and two methods are presented. The first method assumes that each word affects all neighbors in its window, and the second method assumes that each word affects some random neighbors in its window. So, the first method selects all neighbors, but the second method selects some neighbors randomly. In this study, all evaluations are calculated based on the first method, and the second method is considered for the evaluation in Table 14.Figure 13
An example for showing the effect of each word on neighbors in its window.As shown in Table13, accuracy, perplexity, and topic_coherency values are calculated on Android, Automotive, Electronic, and Movie datasets before random selection using the first method. As shown in Table 14, accuracy, perplexity, and topic_coherency values are calculated on Android, Automotive, Electronic, and Movie datasets after random selection using the second method. So, in Table 13, it is assumed that each word affects all neighbors in its window, but in Table 14, it is assumed that each word affects some random neighbors in its window. As shown in Tables 13 and 14, average values of accuracy, perplexity, and topic_coherency are equal to average values in each column of Tables 13 and 14 for each window size. The values are calculated on Android Automotive, Electronic, and Movie datasets. According to the results, the second method is more stable than the first method in terms of accuracy, but the first method has higher accuracy than the second method. The second method outperforms the first method in terms of perplexity. The first method performs better than the second in terms of accuracy and topic_coherency.The analysis of the Friedman test on the results of Table13 demonstrates that there is a statistically significant difference between the performances of the algorithms in terms of accuracy with χ25=27.608andp<0.01, in terms of perplexity with χ25=35.143andp<0.01, and in terms of topic_coherency with χ25=6.232andp=0.284. The mean rank of the algorithms based on the Friedman test, which is demonstrated in Figure 14, indicates that (w = 1) outperforms other windows in terms of accuracy. Still, it has lower perplexity and topic_coherency values than other windows. (w = 6) outperforms other windows in perplexity and topic_coherency, but it has a lower accuracy value than other windows. (w = 3) provides a special situation for proposed algorithms in which accuracy, topic_coherency, and perplexity values are between the highest and lowest values (w = 1, 2, 4, 5, and 6). So, in this study, all evaluations are calculated based on (w = 3). The mean rank of the algorithms based on Table 14, which is demonstrated in Figure 15, indicates that (w = 1) ranks first among all the algorithms in Table 14 in terms of accuracy. The analysis of the Friedman test indicates that there is a statistically significant difference in terms of accuracy with χ25=17.550andp<0.01, in terms of perplexity with χ25=37.857andp<0.01, and in terms of topic_coherency with χ25=5.857andp=0.320.Figure 14
The mean rank of the seven algorithms is in Table13 according to the Friedman test.Figure 15
The mean rank of algorithms is in Table14 according to the Friedman test.


## 4.8. Sentiment Classification Using Proposed Methods in Comparison to ALGA

In this subsection, WJST and WJST1 are compared to ALGA [46]. Three datasets have been selected for evaluating the methods. Evaluations results are shown in Figure 16, which compares the results of models with each other according to accuracy, perplexity, and topic_coherency metrics. In ALGA [46], several sentiment lexicons are created for a dataset during the training stage using a genetic algorithm. During the testing process, these dictionaries are employed. Every dictionary has some words and scores. Each chromosome is represented as a vector of sentiment words and their scores in the genetic algorithm employed in the method. The scores are spread between a feeling word's lowest and maximum scores. The primary goal of ALGA is to create a dictionary that reduces errors on training datasets. The sum of scores for words of each instance Ti in dataset Dm using dictionary Lk is calculated using equation (19) and is treated as a feature [46]:(19)ALGADm,Ti,Lk=∑Wj∈TivkWj.Figure 16
Sentiment classification on SANDERS, SOMD, and STS datasets, in terms of accuracy (a), perplexity (b), and topic_coherency (c).
(a)(b)(c)Finding the values of words in the dictionaries (chromosomes) and adding them together is how the ALGA value for each instance is calculated. In (19), Wj represents the words of Ti, and vkWj shows the score of Wj in Lk. As mentioned in [46], ALGA will predict a positive instance when the ALGA feature is positive and a negative instance when the ALGA feature is negative. By dividing the number of correct predictions of instances of a given dataset by the total cases, ALGA's accuracy is calculated. In this subsection, proposed methods are compared with ALGA [46] because it can automatically generate a sentiment dictionary. ALGA generates a sentiment dictionary using the genetic algorithm, but proposed methods generate a sentiment dictionary using topic modeling. In proposed models, each document is classified based on P(s| r), the probability of sentiment label given a document. In proposed models, two labels (+, −) are considered, and a document is classified as negative if P(+|r) < P(−|r) and vice versa. Evaluations results can be seen in Figure 16. In this subsection, the number of distinct windows is three, and the number of topics is five. The models use the AFINN sentiment dictionary. According to Figures 16(a)–16(c), each column compares different methods on a dataset. The details of the datasets used in this section are illustrated in Table 4. In this subsection, only the accuracy is considered for the evaluation of ALGA. The ALGA-SW value is achieved by executing ALGA without taking stopwords into account. According to the results, WJST has higher accuracy than TJST on all datasets. WJST1 outperforms WJST and TJST on all datasets. ALGA and ALGA-SW perform better than other methods in terms of accuracy, but WJST1 achieves higher accuracy than ALGA and ALGA-SW on STS and Sanders datasets. The RND method achieves the lowest accuracy value on Sanders and STS datasets. The AFINN + RND method has higher accuracy than the RND method and has a lower accuracy than the AFINN method on Sanders and STS datasets. The RND method outperforms TJST, AFINN, and AFINN + RND methods on the SOMD dataset. In this study, parameters perplexity and topic_coherency are not calculated for ALGA, ALGA-SW, AFINN, RND, and AFINN + RND methods. According to Figure 16, -1 is accuracry1, and -2 is accuracy2.


## 4.9. Sentiment Classification Using Proposed Methods on Multidomain Datasets

In this subsection, the performance of the proposed methods is compared with baseline methods on a multidomain dataset. In this experiment, the number of distinct windows and topics is three and five, respectively, and the models use the AFINN sentiment dictionary. The multidomain dataset contains reviews taken from multiple domains (product types). The details of the multidomain dataset used in this section are illustrated in Table22.Table 22
Description of the multidomain dataset used in this section.
#DomainsNumber of reviewsVocabulary sizeNumber of words1Android, Automotive, Electronic, and Movie160011183100113As shown in Figure17, accuracy, perplexity, and topic_coherency values are calculated on a multidomain dataset that contains Android. Automotive, Electronic, and Movie domains. The methods WJST-dictionary and WJST1-dictionary classify each document according to Ps|r=Ns,r/Nr. The word sentiment label is directly obtained from WJST and WJST1 lexicons achieved by WJST and WJST1 on the multidomain dataset. According to Figure 17, -1 is accuracry1 and -2 is accuracy2. Based on the results, WJST has a lower perplexity value than other methods. WJST1 outperforms WJST in terms of topic_coherency. WJST performs better than the WJST1 method in terms of accuracy because the distributions θ,ξ,andψ in WJST depend on document, but in WJST1, the distributions θ,ξ,andψ do not depend on the document. Dependency between documents of a domain is more than documents in different domains. A pattern in the documents of a domain may not exist in the documents of other domains. Therefore, calculations on multidomain datasets should be local and not cover all domains. For example, considering the distributions Pz|s and Pz|s,r, where z is topic, s is the sentiment, and r is documents. In the first state (Pz|s), the topic depends on sentiment, and the distribution covers all documents in different domains. Perhaps a topic was positive in one domain and negative in another. Therefore, it is better to depend the topic on the documents of a domain, not all domains. Thus, the topic is limited to the document (and domain), and contradiction between different domains is eliminated. Therefore, WJST is suitable for multidomain datasets, and WJST1 is a version of WJST suitable for single-domain datasets. Sentiment classification on multidomain datasets is a challenge, and our solution in this study is using WJST, whose distributions (θ,ξ,andψ) depend on the document. Sentiment classification on multidomain datasets is a challenge, and further studies can be conducted to investigate this problem for future research.Figure 17
Sentiment classification values are calculated on a multidomain dataset in terms of accuracy (a), perplexity (b), and topic_coherency (c).
(a)(b)(c)


## 4.10. Comparison with Other Methods

In this subsection, the best performance of the proposed methods is compared with 57 competitors [13, 76];  [82–88], [8–10, 46] which is shown in Table 23. The details of the datasets used in this section are illustrated in Table 4.Table 23
Sentiment classification on MR, Sanders, SOMD, STS, Amazon, IMDB, and Yelp datasets.
Method/datasetMRSandersSOMDSTSAmazonIMDBYelpALGA [46]—0.80670.81470.7668———ALGA-SW [46]—0.78680.78770.7886———BPSO (Shang et al., 2016)————0.74390.790.789BICA (Mirhosseini et al., 2017)————0.7930.7450.763BABC (Schiezaro et al., 2013)————0.75090.740.736MaxEnt (Saif et al., 2014)—0.8362—0.7782———NB (Saif et al., 2014)—0.8266—0.8106———LS-all [13]—0.8199—————SVM-all [13]—0.8214—————RMTL [13]—0.827875—————MTL-graph [13]—0.801725—————CMSC [13]—0.846325—————LSTM-all [13]—0.8063—————MTL-CNN [13]—0.829825—————MTL-DNN [13]—0.817—————ASP-MTL [13]—0.85125—————NeuroSent [13]—0.834575—————DAM [13]—0.863225—————SVM-BoW (Da Silva et al., 2014)—0.82430.7402————SVM-BoW + lex (Da Silva et al., 2014)—0.83980.7893————RF-BoW (Da Silva et al., 2014)—0.79240.7391————RF-BoW + lex (Da Silva et al., 2014)—0.82350.7936————LR-BoW (Da Silva et al., 2014)—0.77450.7238————LR-BoW + lex (Da Silva et al., 2014)—0.79490.7806————MNB-BoW (Da Silva et al., 2014)—0.79820.7543————MNB-BoW + lex (Da Silva et al., 2014)—0.83410.8013————ENS(LR + RF + MNB)-BoW (Da Silva et al., 2014)—0.82760.7555————ENS(LR + RF + MNB)-BoW + lex (Da Silva et al., 2014)—0.84890.8035————SVM-FH (Da Silva et al., 2014)—0.49750.5131————SVM-FH + lex (Da Silva et al., 2014)—0.75000.6299————RF-FH (Da Silva et al., 2014)—0.55640.6136————RF-FH + lex (Da Silva et al., 2014)—0.71630.7260————LR-FH (Da Silva et al., 2014)—0.56940.6529————LR-FH + lex (Da Silva et al., 2014)—0.75980.7303————MNB-FH (Da Silva et al., 2014)—0.54250.6070————MNB-FH + lex (Da Silva et al., 2014)—0.75080.7139————ENS(LR + RF + MNB)-FH (Da Silva et al., 2014)—0.57840.6517————ENS(LR + RF + MNB)-FH + lex (Da Silva et al., 2014)—0.76630.7456————WS-TSWE’ [76]0.841——————WS-TSWE [76]0.824——————TSWE-P [76]0.726——————TSWE + P [76]0.782——————JSTH [76]0.681——————HTSM [76]0.796——————SAE (Pagliardini et al., 2018)0.861——————ParagraphVec DBOW (Pagliardini et al., 2018)0.763——————ParagraphVec DM (Pagliardini et al., 2018)0.764——————IST (Pu et al., 2019)0.827——————UST (Pu et al., 2019)0.832——————UIST (Pu et al., 2019)0.845——————RND0.48950.48520.48470.53460.4910.4980.506AFINN0.6010.6740.33950.7340.7310.6980.689AFINN + RND0.54550.54240.44750.60380.5740.5750.559JST [8]0.6130.61760.49340.63980.6110.6050.579TJST [8]0.620.58980.46390.68140.6450.6160.614RJST [12]0.510.60860.49670.62320.6090.5460.561TS [9]0.50.56120.52510.5650.540.5450.547WJST-10.8210.7268.757860.71260.7790.760.737WJST-20.8180.72030.757750.71810.8290.7610.726WJST1-10.84450.8320.80.83170.7960.770.773WJST1-20.8430.83520.78590.83730.7980.7740.769


## 4.11. Comparison with Discriminative Models

The proposed methods are compared to baseline approaches such as logistic regression and SVM on four datasets in the following experiment. The multidomain dataset contains Android, Automotive, Electronic, and Movie domains. As shown in Table24, the accuracy value is calculated on four datasets. The results demonstrate that proposed methods have improved notably over AFINN and the baseline methods on all datasets. Based on Table 24, methods use two systems in preprocessing phase, which includes Bag of Word (BOW) and Term Frequency Inverse Document Frequency (TF-IDF). In the BOW system, more word frequency reflects more importance of the word. TF-IDF system believes that high frequency may not be able to provide much information. Furthermore, rare words contribute more weight to the method. According to evaluation results, the results of the TF-IDF system are better than the BOW system.Table 24
Sentiment classification in comparison with discriminative models on different datasets.
Method\DatasetAndroidAutomotiveElectronicMovieRND0.480.49250.4650.525AFINN0.69750.6250.6750.595RND + AFINN0.580.5350.520.555LOGISTIC REGRESSION(BOW)0.530.520.53380.57RANDOMFOREST(BOW)0.690.650.600.6575SVM(BOW)0.490.510.540.5675DECISIONTREE(BOW)0.710.700.560.665NAIVE_BAYES(BOW)0.510.550.47120.555KNEIGHBORS (N = 3, BOW)0.550.570.550.5725KNEIGHBORS (N = 4,BOW)0.560.5750.59150.585KNEIGHBORS (N = 5,BOW)0.570.57250.550.59KNEIGHBORS (N = 6,BOW)0.570.56750.560.61LOGISTIC REGRESSION (TF-IDF)0.610.630.830.59RANDOMFOREST (TF-IDF)0.5750.60.72430.5225SVM (TF-IDF)0.58250.590.800.54DECISIONTREE (TF-IDF)0.550.570.70430.55NAIVE_BAYES (TF-IDF)0.580.600.79450.58KNEIGHBORS (N = 3, TF-IDF)0.650.630.530.7125KNEIGHBORS (N = 4, TF-IDF)0.650.630.560.7125KNEIGHBORS (N = 5, TF-IDF)0.650.630.510.7125KNEIGHBORS (N = 6, TF-IDF)0.650.57250.530.7125JST0.6250.65750.70250.7575ASUM0.6130.63220.710.772TJST0.7650.76750.760.9475RJST0.58250.6150.55250.62TS0.54250.55250.54750.5425WJSTAccuracy10.7950.7550.86250.8475Accuracy20.78250.74750.8750.8325WJST1Accuracy10.8650.80.84750.97Accuracy20.85250.7950.8550.9675


## 4.12. Comparison with JST According to Extended Features

Suppose a unigram corresponds to the sentiment lexicon. In that case, its polarity will be equal to the subjectivity of the lexicon in order to identify the emotion label of the unigram for trying to prepare prior emotion information. The following technique is used to decide the emotion label of a bigram to prepare prior emotion information: If words of the bigram have the same polarity, the bigram’s polarity will be the same as that of the words. If one of the words is in the lexicon, the bigram’s polarity will equal the lexicon’s subjectivity. The bigram’s polarity will be opposed to the polarity of the second word if the first word is ‘not.’ The following methodology is used to decide the emotion label of a trigram in order to prepare prior emotion information. If words of the trigram have the same polarity, the trigram’s polarity will be the same as that of the words. If one of them is in the lexicon, the trigram’s polarity will equal the lexicon’s subjectivity. The trigram’s polarity will be opposed to the second or third word’s polarity if the first or second word is ‘not.’ The proposed methods are compared with JST on four datasets (single-domain) according to extended features (bigrams and trigrams) in the following experiment.As shown in Table25, the accuracy value is calculated on four datasets, and the experiment extends the features to bigrams and trigrams. According to the results, WJST1 outperforms WJST. According to evaluations results, proposed models outperform JST because the additional parameters can influence the process of producing words in a review appropriately. The perplexity value varies on different datasets because the size of datasets is different. As the number of grams increases, perplexity is increased because in each document, in addition to the unigrams (+bigrams), bigrams (+trigrams) are added to the data, and the size of the dataset is increased. As the number of grams increases, accuracy is improved. In some cases, it gets worse because higher grams (bigram or trigrams) are sometimes meaningless.Table 25
Sentiment classification according to extended features (bigrams and trigrams) on different datasets (single-domain).
ModelGramMetric\datasetAndroidAutomotiveElectronicMovieJSTU∗Accuracy0.6250.62750.6750.7575Perplexity19.672624.715425.22727.3111Topic_Coh−4.6026−1.486−1.5892−1.0123U + B∗Accuracy0.6550.70250.750.835Perplexity66.188286.689781.337598.9201Topic_Coh−2.4595−1.0417−0.9823−0.4165U + B + T∗Accuracy0.67750.58250.75250.7275Perplexity114.8812158.7113156.1383190.8403Topic_Coh−1.5776−2.3915−0.7030−0.1288WJSTUAccuracy10.79250.7550.86250.7225Accuracy20.77750.74750.8750.71Perplexity16.730321.200820.548923.1145Topic_Coh−0.5547−1.6542−2.0442−0.0751U + BAccuracy10.68250.630.670.65Accuracy20.680.6250.66750.6475Perplexity36.550549.858448.200858.0838Topic_Coh−1.3976−4.0214−1.2513−0.0423U + B + TAccuracy10.73250.68250.6350.6475Accuracy20.73250.68250.6150.6325Perplexity52.920572.762976.254588.1860Topic_Coh−1.3242−1.5525−1.2409−2.5170WJST1UAccuracy10.810.800.790.97Accuracy20.79250.790.800.9625Perplexity16.678721.435722.601224.6302Topic_Coh−0.187−1.1883−1.1683−0.1348U + BAccuracy10.70.76250.85750.935Accuracy20.70.76250.85250.935Perplexity38.560349.516754.704162.3459Topic_Coh−2.0962−1.8607−0.9744−0.0617U + B + TAccuracy10.830.76250.7450.9475Accuracy20.82750.76750.7450.9475Perplexity55.581975.362477.666096.3675Topic_Coh−1.3837−0.1840−0.9334−0.0333U∗: unigram. B∗: bigram. T∗: trigram.


## 4.13. Discussions on the Limitations of the Proposed Methods

Although the analysis of the results of the evaluation can demonstrate the best performance of proposed methods, proposed methods have some limitations, as follows:(1)
The first limitation is the time complexity of the proposed methods (O(G·wALL·|S|·|Z|·|Q|·|E|)) which is more than baseline methods (O(G·wALL·|S|·|Z|)) according to Table 3 in Section 3.3.(2)
The second limitation is the window size. On the report in Table13, an increase in the size of a window will decrease the accuracy because it will increase the number of words in the window, and each term may not affect all neighbors in its window. Therefore, finding the words that can be involved in a window is a new challenge that we introduce in this study, and two methods are presented in Section 4.7. The first method assumes that each word affects all neighbors in its window, and the second method assumes that each word affects some random neighbors in its window.Therefore, the first method selects all neighbors, but the second method selects some neighbors randomly. According to the results, the second method is more stable than the first method in terms of accuracy, but the first method has a higher accuracy value than the second method. The second method outperforms the first method in terms of perplexity. The first method performs better than the second in terms of accuracy and topic_coherency.


## 4.14. A Concise Description of the Proposed Solutions and the Results

The main problem in this study is to examine a user’s opinion about a product or movie, for example. This means identifying whether a user has a positive or negative idea about a subject (a product or movie).Two novel models have been proposed that use topic modeling to solve the above problem. So, our solution is using a technique named topic modeling. Proposed models extend and improve JST (as a topic model) through two new parameters. To improve JST, proposed models consider the effect of words on each other. The new parameters have an immense effect on model accuracy regarding evaluation results. According to evaluations results, the proposed models outperform JST because the additional parameters can influence the process of producing words in a review appropriately. They can improve sentiment classiﬁcation at the document-level. Also, in the evaluation results report, proposed methods are more accurate than discriminative models such as SVM and logistic regression. Proposed methods are more flexible than discriminative models because other information, such as the top 10 words, can be extracted from the heart of the data.


## 5. Conclusion

In this study, two new models called WJST and WJST1 have been presented that extend JST and improve accuracy metrics. Reviewing the various articles about sentiment analysis indicates that the proposed models are associated with innovation and lead to remarkable results compared to the baseline methods. The proposed models can generate a sentiment dictionary. According to evaluations results, the proposed models consider the effect of words on each other using the extra parameters, which are important and influential. The evaluation results indicate that the accuracy has been improved compared to the baseline methods such as JST, RJST, TS, TJST, and ALGA. Results show that the proposed methods perform better with different topic number settings. WJST1 outperforms other methods in terms of accuracy, demonstrating its effectiveness of that. Prior sentiment information affects perplexity and topic_coherency lower than accuracy.According to evaluations results, using the AFINN dictionary as prior sentiment information is more effective than using the NO_AFINN state. ALGA uses the genetic algorithm to generate a sentiment dictionary; however, proposed methods use topic modeling to generate this dictionary. According to the evaluation results, the proposed models outperform JST because the additional parameters could influence the process of producing words in a review appropriately. They have the potential to increase the emotion detection's accuracy at the level of the document. The proposed methods are unsupervised, and no labeled data is required. Proposed methods can automatically assess web comments and categorize reviews as positive or negative. The proposed methods have tried to increase the accuracy with fewer parameters and, at the same time, simplicity compared to the existing methods. The proposed methods both analyze emotions at the document-level and create an emotional dictionary. They are also the first methods to create an emotional dictionary through a topic modeling technique and in an automatic and accurate way. The proposed methods are the first methods that consider the words in the text and their effect on each other in a dynamic and weighty way. Also, they are parametric.


## 6. Future Work

The proposed models are parametric in the present study, and further studies will be conducted to investigate nonparametric models. Sentiment classification on multidomain datasets is a challenge, and further studies can be conducted to investigate this problem for future research. In future research, the proposed methods can be evaluated on more datasets. More parameters can also be assessed. Twitter social network data have obtained significant attention in natural language processing studies, with certain conditions, such as short data length. In future articles, the proposed methods can be modified to analyze emotions with specific Twitter data.



---

*Source: 7612276-2022-07-31.xml*