# Big Data Aspect-Based Opinion Mining Using the SLDA and HME-LDA Models

Authors: Ling Yuan; JiaLi Bin; YinZhen Wei; Fei Huang; XiaoFei Hu; Min Tan
Journal: Wireless Communications and Mobile Computing
 (2020)

---

## Abstract

In order to make better use of massive network comment data for decision-making support of customers and merchants in the big data era, this paper proposes two unsupervised optimized LDA (Latent Dirichlet Allocation) models, namely, SLDA (SentiWordNet WordNet-Latent Dirichlet Allocation) and HME-LDA (Hierarchical Clustering MaxEnt-Latent Dirichlet Allocation), for aspect-based opinion mining. One scheme of each of two optimized models, which both use seed words as topic words and construct the inverted index, is designed to enhance the readability of experiment results. Meanwhile, based on the LDA topic model, we introduce new indicator variables to refine the classification of topics and try to classify the opinion target words and the sentiment opinion words by two different schemes. For better classification effect, the similarity between words and seed words is calculated in two ways to offset the fixed parameters in the standard LDA. In addition, based on the SemEval2016ABSA data set and the Yelp data set, we design comparative experiments with training sets of different sizes and different seed words, which prove that the SLDA and the HME-LDA have better performance on the accuracy, recall value, and harmonic value with unannotated training sets.


---

## Body

## 1. Introduction

With the development of the Internet, almost all the things of human living have become digitized. The information in network is in the form of structured data and unstructured data [1]. Big data analysis and mining is aimed at discovering implicit, previously unknown, and potentially useful information and knowledge from big databases that contain high volumes of valuable veracious data collected or generated at a high velocity from a wide variety of data sources [2], which is called “4V” of big data. In fact, the deeper mining of big data is to mine the user demand and other deep information; the text mining that this paper studied is one of the ways to mine valid information from big data of text. Therefore, the study proposes two optimized opinion mining methods for customers and merchants to extract valid information they need from massive textual data that satisfies the “4V” [2].For example, when purchasing a product, people usually refer to others’ comments in the specialized product comment area first. Although comments on different platforms have different forms of display, most of them are text-based. Due to most product comments on Taobao only have positive, neutral, and negative labels, what users can directly refer to is just the number of positive and negative comments. Since different users have different needs for the same product, they need to know which attributes of it perform well and which perform poorly. However, these attributes are not shown in detail in the comment interface. It is impossible and uneconomic for users to read more comments to find the attribute they need, because of the massive quantity and continuous growth of product comments. Thus, in order to assist clients and merchants for better decision-making, conducting opinion mining and sentiment analysis of big data is necessary, which will bring huge profits to some markets.Data mining and analysis have been used in the tourism industry [3], groundwater potential mapping [4], and so on; it becomes more and more important in modern life. Liu and Zhang [5] divided the text mining and analysis tasks into three levels of granularity: chapter level, sentence level, and attribute level. The chapter level, whose research unit is a document, usually uses algorithms to show whether the opinions expressed by the author are negative or positive, and it is often used to analyse blogs and news. While the sentence-level sentiment analysis is often used to analyse whether the expression of microblogs, tweets, etc., in social networks is negative or positive. Obviously, it is impossible to perform opinion mining at the chapter level and sentence level when analysing user comments. Please consider the following review:The location of this restaurant is relatively remote, but the waiter has a good attitude and the dishes taste good.In fact, this sentence contains three opinions. Therefore, we ought not to determine the sentiment polarity of this comment simply and should get more specific results, such as <location, remote, negative>, <service, good, positive>, <dish, good, positive>, i.e., the aspect-based opinion mining. In 2015 and 2016, SEMEVAL released the research topic of ABSA (Aspect-Based Sentiment Analysis), triggering a study boom among scholars. SEMEVAL divides ABSA into three subtasks: identifying entities and attributes, identifying the expression of opinions, and identifying the sentiment polarity of opinions. SEMEVAL also gives the annotated training sets and test sets, such as catering, hotel, and laptop.In recent years, people devote a lot of energy to analyse comments on the Internet for obtaining the detailed opinions of users [6, 7]. The aspect-based opinion mining of online comments can be divided into two subtasks, namely, the opinion target extraction and the classification of sentiment polarity of comments. Figure 1 gives the overview of the study method classification of the aspect-based opinion mining.Figure 1
The classification of aspect-based opinion mining.With the development of supervised models, the deep learning, and the conditional random field, CNN (Convolutional Neural Networks) and RNN (Recurrent Neural Networks) have been widely used in NLP (Natural Language Processing) [8], which have achieved good results in ABSA research. Some other supervised learning methods [9, 10] such as the BMAM [11] which was proposed in recent time also have achieved good results. In addition, Chen et al. [12] and Araque et al. [13] introduced deep learning into the ABSA and achieved satisfactory results. However, compared with the models proposed in this paper based on LDA, all the above models lack the adaptability in various fields and have high manpower costs for annotation. For example, the effects of these models will be greatly reduced when the aspect category of the comment is transferred from the food and beverage to the laptop. Moreover, supervised models such as BMAM [11] need a lot more manpower than the models proposed in this paper to annotate data due to the small number of annotated training sets given. In addition, the time complexity of SLDA and HME-LDA is better than the popular models above. The time complexity of SLDA and HME-LDA is just related to the number of documents, topics, and words, while the time complexity of the popular models such as RNN is related to the multilayer neural network structure, especially the fully connected layer, and the result of the previous time sequence, leading to a really lower speed than the SLDA and HME-LDA.As for the unsupervised model, there are two basic models for latent semantic analysis: the probabilistic latent semantic analysis (PLSA) [14] model and the latent Dirichlet allocation (LDA) [15] model, which can be applied to extract attributes, assuming that each comment is a combination of attribute words and opinion. Mei et al. [16] proposed a topic-sentiment hybrid model based on the PLSA to extract aspect opinion target words and sentiment words from a group of blogs. Li et al. [17] proposed two kinds of joint models, sentiment LDA and dependence-sentiment LDA, to find positive or negative aspects of sentiment words. Due to the flexibility of the LDA topic model, it is extended and combined with other methods to obtain a topic model [18, 19], which can improve the result topics or the additional information of the model.In view of the short content, wide coverage and the small number of the annotated corpus of the network comment and its need for aspect-based mining, this paper proposes two schemes based on the LDA topic model that have unsupervised features and good extensibility, making it possible for network comments to perform aspect-based opinion mining with as little annotated data as possible. As for the data cleaning, the amount of data, correctness, completeness, and time correlation [20] are all good evaluation indicators of data quality. As for the data amendment, the Markov Random Field performs well [21].This paper regards opinion targets, aspects, and opinion expressions as aspect opinion targets that refer to entities or properties to which sentiment words modifies, and sentiment words related to aspect opinion targets are called sentiment opinion words or opinion words.Moreover, the aspect-based opinion mining schemes proposed in this paper only require users to set corresponding seed words and introduce the classification layer to classify the opinion target words and sentiment opinion words. Meanwhile, in order to improve the effects of the models in schemes, we are biasing the parameters of the LDA model by calculating the similarity between the words in the corpus and the seed words we set.To sum up, the main study contents of this paper are as follows:(1)
Introduce the NLP tools, WordNet and SentiWordNet, into the standard LDA model to design an optimized LDA-based topic model(2)
Introduce a maximum entropy classifier into the standard LDA model to design another optimized LDA-based topic model(3)
Implement the above two optimized models and design experiments to verify the feasibility and superiority of optimized modelsThe rest of the paper is organized as follows. Next, we will describe two schemes with two optimized LDA-based models in detail. Then, the experimental results will be given in the next section. At the same time, we will give some analysis about the results. Finally, we give the conclusion.


## 2. Two Optimized LDA Models for Aspect-Based Opinion Mining

This paper proposes two schemes for aspect-based opinion mining. The first scheme is based on the inverted list and the SLDA (SentiWordNet WordNet-Latent Dirichlet Allocation) model proposed in this paper. The second scheme is based on the inverted list and the HME-LDA (Hierarchical Clustering MaxEnt-Latent Dirichlet Allocation) model proposed in this paper.The SLDA model is an optimized LDA model based on the WordNet and SentiWordNet, where the WordNet is for the similarity calculation of words and seed words and the SentiWordNet is for the separation of the opinion target words and the opinion words, while the HME-LDA model is an optimized LDA model based on SLDA and MaxEnt-LDA [22]. In fact, SLDA still has the disadvantage of relying on dictionary tools (WordNet and SentiWordNet), leading to the application failure of SLDA in other languages. Luckily, the HME-LDA model solves the problem.Next, we will illustrate the two optimized models of schemes in detail.


### 2.1. Optimized Model, SLDA, Based on SentiWordNet and WordNet

Different from the text messages such as documents, blogs, and news, network comments tend to be shorter and often appear in the form of sentences. Please consider the following restaurant comments:(1)
“We, there were four of us, arrived at noon - the place was empty and the staff acted like we were imposing on them and they were very rude.”(2)
“Everything is always cooked to perfection, the service is excellent, the decor cool and understated.”In nonaspect opinion mining, the only thing you need to do is to analyse the sentiment polarity of sentences. For example, the word “rude” in the first sentence is negative, so the sentiment polarity of the first sentence is negative. In the second one, the sentiment polarity of the word “perfection” is positive, so the sentiment polarity of the second sentence is positive. This way that only judges the sentiment polarity of sentences does not apply to network comments. More meaningful information should be specific to the word pairs of <Aspect Opinion Target-Opinion> such as <staff-rude>, <cook-perfect>, and <service-excellent>, while the algorithm based on the topic model lacks readability and appointed key words, where the relevant original content fails to be directly found by the final result.The SLDA is an optimized LDA model based on SentiWordNet (a sentiment dictionary based on WordNet) and WordNet (a large database of English words). Since it is impossible for the LDA itself to separate opinions from opinion targets, this chapter adds a classification layer of opinion words and opinion target words based on the LDA to realize the separation of opinions and aspect opinion targets. The similarity between the word and the seed word in the text, which is reflected on LDA parameters, is calculated by setting the seed word and using the calculation tools of the vocabulary similarity in WordNet. Meanwhile, the opinion target is separated from the sentiment opinion words using tools that calculate the vocabulary sentiment in SentiWordNet. Aiming at the lack of readability of LDA results, we establish a belonging relationship among the clustering results, seed words, and original texts. Also, the SLDA model needs to set seed words, but has no need for the additional annotated data sets.In order to achieve the goal that users can quickly find what they want from massive comments by inquiring the index rather than by reading, there are three steps to do:(1)
Construct an inverted index to number sentences and words(2)
Determine the aspect category of comments, separate the aspect-based attribute words from the sentiment opinion words, and determine the aspect category to which they belong(3)
Enhance the readability of results. Users can see an overview of Domain-Aspect-Opinion and find the specific sentence by the inverted index of a wordIt is worth noting that the premise of our study is that the aspect category of the original corpus is known. The aspect category, which has the belonging relationship with the aspect opinion target, is usually given in advance by the original corpus, which can be learned by the user guide of the corpus. For example, the aspect opinion target “steak” belongs to the aspect category “Food.” In addition, both words and sentences can belong to an aspect category. Thus, what only need to do is to label aspect opinion target words and sentences with a specific aspect category.


#### 2.1.1. Implementation Process

Figure2 is an overall flow chart of the scheme that conducts aspect-based opinion mining based on SLDA and the inverted index.
(1)
Construct an inverted index. The words in the corpus are numbered in the form of binary group<a, b>, where a is the serial number of the sentence and b is the serial number of the words in the sentence. In addition, the generation of the inverted list requires the removal of duplicate word and the recording of their numbers. The inverted list reserves the sentence number that contains the word and the position information of the word in the sentence, making it easier to retrieve with context information from the original corpus later(2)
Data preprocessing, whose main tasks are to extract the data required and remove stop words from the original corpus for making a training set. The formats of the original data sets are XML and CSV. It is necessary to extract comment statements by the corresponding labels and fields. And the text in the corpus contains many useless stop words, such as “is” and “a,” which should be removed before further processing to avoid interference with the training of the SLDA model(3)
Introduce preprocessed data into SLDA for model training and get clustering results. The setup of seed words, as well as the assist of WordNet and SentiWordNet, is required in this process(4)
Process the clustering results for better readability. The results in the topic model are probability matrix, whose readability is poor. To solve this problem, it is necessary to find the word corresponding to the result with higher probability and find its original sentence by the inverted indexFigure 2
The overall flow chart of the scheme that is based on SLDA and inverted index.In short, the SLDA model, trained with the preprocessed data of step 2 above, queries the original sentences that contain keywords from the original corpus by the inverted index of step 1 above.


#### 2.1.2. The Optimized Direction of LDA

The expectation of every random variableμi of the Dirichlet distribution can be expressed as Eμi. The value of Eμi can be calculated by equation (1), where α represents the parameter of the Dirichlet distribution and K represents the number of topics:
(1)Eμi=αi∑i=1Kαi,whereα is a fixed value and the Eμi of each topic is same. Based on this, a biasing method of α can be explored to make the expectations of the corresponding probabilities of each topic different, so as to generate the topic bias. More visually, when α is fixed, it means that we fail to know which topic word to use before generating the document. When α is biased, the ideal topic is determined before the document is generated.Actually, the topic numberZm,n can be used as an indicator variable in the LDA model to control the selection of topic-word distribution. Based on this, more indicator variables can be introduced to refine the topic and obtain more topic-word distributions.From the above, there are two aspect opinion targets that the LDA model can optimize: the first is to bias the parameters,α and β, which can generate topic biases to improve the classification effect; the second is to introduce more indicator variables like Zm,n, which can generate more detailed topic classifications.


#### 2.1.3. The Description of the SLDA Model

The standard LDA uses the document as the unit of topic allocation, while in the aspect-based opinion mining of the network comments, the sentence is often used as the unit of topic allocation, because there is no document with large contents in network comments and the topic allocation of vocabulary in network comments is actually more meaningful. In the aspect-based opinion mining of network comments, it is necessary to extract the aspect category of the comment, the opinion target, and the comment opinion (sentiment polarity) from the text. For example, in restaurant comments, “food,” “service,” and “ambiance” are the aspect categories of comments. In the “food” category, “steak” is the opinion target, and the evaluation of “good” for “steak” is the opinion of the comments (sentiment polarity).In the SLDA model, seed words are directly used as aspect categories of comments, while the opinion targets of comments and the comment opinions are separated by the introduced sentiment layer, and the positive and negative polarities of comments are classified as well. The PGM (Probabilistic Graphical Model) of SLDA is shown in Figure3.Figure 3
The probabilistic graphical model of SLDA.


#### 2.1.4. The Generation Process of the SLDA Model

Only the parametersα and β, which, respectively, belong to the document-topic Dirichlet distribution and the topic-word Dirichlet distribution, are introduced into the standard LDA. In the SLDA model, seed words have been identified as the aspect category of comments. The variable yd∈A,O is introduced to represent the separation of the opinion target and the comment opinion. When yd=A, the current word is the opinion target of comments. When yd=O, the current word is the comment opinion. Meanwhile, the variable vd∈P,N is introduced into SLDA. When vd=P, the sentiment polarity of the current vocabulary is positive, and when vd=N, the sentiment polarity of the current vocabulary is negative. Both yd and vd are determined by the corresponding algorithms based on the WordNet and SentiWordNet. In the standard LDA model, the values of parameters, α and β, are fixed. In the SLDA model, different αd will be set for each sentence, and the parameters, βt,A, βt,P, and βt,N, will be set for the opinion target, the positive comments and the negative comments, respectively.In SLDA, the first step is to generate the sentence-topic distribution and determine a topic for each sentence by the multinomial distribution. Then, two influence factors,yd and vd, are determined by WordNet and SentiWordNet to indicate the aspect categories of words. By the SLDA generation method above, we finally select a topic-word distribution and determine the final word.The concrete generation process of SLDA is as follows:Firstly, the distributions of the opinion targetφt,A, the positive opinion word φt,P, and the negative opinion word φt,N are, respectively, extracted from the parameters, βt,A, βt,P, βt,N, where φt,A~Dirβt,A, φt,P~Dirβt,P, and φt,N~Dirβt,N.Similar to the standard LDA, for each sentence (document in the standard LDA), a topic distribution,θd~Dirαd, is extracted from the Dirichlet distribution with the parameter αd.Then, the wordwd,n in the sentence d extracts a topic number t, i.e., extracts zd,n~Multiθd.After determining the topic number of the wordwd,n, the classification of it remains to be determined. After determining the topic t in SLDA, in order to further classify the word as the opinion target word or the opinion of comments, the variables, yd,n∈A,O and vd,n∈P,N, are introduced into the SLDA model. The yd,n and vd,n point to the n-th word wd,n in the sentence d jointly. The yd,n is used to indicate that the word wd,n is the opinion target of comments or the comment opinion, which is extracted from the Bernoulli distribution on 0,1 with the parameter πd,n. The vd,n is used to indicate that the word wd,n is a positive or a negative comment opinion, which is extracted from the Bernoulli distribution on 0,1 with the parameter Ωd,n. The above two Bernoulli distributions are calculated by the WordNet and SentiWordNet. Finally, the word wd,n can be extracted according to equation (2):
(2)Wd,n~Multiφt,A,ifyd,n=A,Multiφt,A,ifyd,n=O,andvd,n=P,Multiφt,A,ifyd,n=O,andvd,n=N.Table1 gives the description of the related symbols in the SLDA model, which is useful for readers in reading.Table 1
The description of related symbols in the SLDA model.
SymbolDescriptionDThe total number of comments in the corpus. The unit of corpus is the sentenceTThe number of topicsVThe number of words in the corpuswd,nThen-th word in the d-th comment in the corpuszd,nThe topic ofd-th comment. The value is 1,⋯,Tyd,nAn indicator variable. The value isA,O. It is used to indicate the opinion target words and the sentiment opinion words.vd,nAn indicator variable. The value isP,N. It is used to indicate positive and negative sentiment opinion words.A,O,P,NThe opinion target, the sentiment opinion word, the positive sentiment word, the negative sentiment wordφt,AThe distribution of the opinion target generated by a priori Dirichlet distribution with the parameterβt,Aφt,PThe distribution of positive comment words generated by a priori Dirichlet distribution with parametersβt,Pφt,NThe distribution of negative comment words generated by a priori Dirichlet distribution with parametersβt,NθdThe distribution of sentence topic terms generated by a priori Dirichlet distribution with parameterαd


#### 2.1.5. The Inference Process of the SLDA Model

The SLDA model consists of two major parts. One is the classification of the opinion target words and sentiment opinion words composed of the WordNet and SentiWordNet as well as the classification of positive and negative sentiment opinion words. The other is the LDA topic model. Besides, the seed words should be set before the inference of SLDA. Next, several modules will be introduced in turn.(1) The Setting of Seed Words. In SLDA, seed words are set as aspect categories of comments. For example, in the classic English comment set of the Restaurant, the aspect categories are food, service, ambiance, etc. If the corpus is the Restaurant English comment set, the seed words can be directly set as food, service, and ambience. The seed word is recorded as wt, where t∈1,⋯,T, that is to say, the number of seed words determines the number of topics in the SLDA model.(2) The Inference of the Word Classification Model Based on the WordNet and SentiWordNet. In the SLDA model, the Bernoulli distribution with parameter πd,n and Bernoulli distribution with parameter Ωd,n, which are, respectively, used to separate the opinion target words from sentiment opinion words and separate positive sentiment opinion words from negative sentiment opinion words, are related to the WordNet and SentiWordNet. The calculation of πd,n depends on the seed word wt.The words in WordNet have the feature of polysemy. To calculate the similarity between words, it is necessary to determine the exact meaning of a word. Therefore, before the model inference, it is necessary to determine the semantic interpretationst,k0 of the seed word wt in the WordNet. When the current word is wd,n, its semantic interpretation in the WordNet is sd,n,k, where k∈1,⋯,K and K is the number of semantic interpretations. After determining the semantics of the seed words, we can regard the seed word as the topic and the aspect category of the final opinion target. All nonsentiment opinion words will be grouped into a certain aspect category of comments. Therefore, the similarity between the semantics of the current word and the semantics of the seed word can be calculated, and the semantics with the greatest similarity can be determined as the meaning expressed by the word in the sentence finally. The degree of semantic similarity between different words in the WordNet is recorded as Sims1,s2; then, the semantic similarity, Simsd,n,k,st,k0, between the sd,n,k of wd,n and the st,k0 of each seed word wt can be calculated, where k∈1,⋯,K. Besides, K is the number of semantic interpretations, t∈1,⋯,T, and T is the number of seed words. In all calculation results, we choose the semantics with the max value of the similarity result and determine the largest result k′ as the semantics sd,n,k′ to which the current word wd,n belongs.After determining the semanticsd,n,k′ of the word wd,n, the sentiment polarity of sd,n,k′ can be queried in the SentiWordNet. In the SentiWordNet, the semantics of a word has three sentiment propensity probabilities: ρo indicates the probability that the semantics is objective (excluding sentiment polarity), pP indicates the probability that the semantics is positive, and ρN indicates the probability that the semantics is negative. Besides, po+pP+pN=1. It is believed that if the semantics is objective, the word is the opinion target, whose corresponding probability is po; otherwise, it is a sentiment opinion word. The sentiment scores of the semantics sd,n,k′ of the word wd,n are recorded as pd,n,ko, pd,n,kP, and pd,n,kN. In Section 2.1.4, the Bernoulli distributions with parameter πd,n and parameter Ωd,n can be determined by equation (3) and equation (4):
(3)πd,n=pd,n,ko,(4)Ωd,n=pd,n,kPpd,n,kP+pd,n,kN.So far, the model inference based on the WordNet and SentiWordNet has been completed. Algorithm1 is the pseudocode for calculating πd,n, Ωd,n.Algorithm 1. The calculation ofπd,n, Ωd,n.
1: Query the semantic listSlist of wd,n in WordNet2://Record the semantic value of the max similarity3:sw=0;4://Record the maximum similarity5:simmax=0;6://Each semanticsofwd,n7:fors∈slistdo8:forst∈stlistdo9://Sim() is a function provided by WordNet10:ifsimmax<Sims,stthen11:sw=s;12:sim_max=Sims,st;13;end if14:end for15:end for16: Use SentiWordNet to query the sentiment polarity of semanticSw17: Calculateπd,n using equation (3)18: CalculateΩd,n using equation (4)19:returnsw20:returnπd,n21:returnΩd,n(3) The Inference of the SLDA Model. In the standard LDA, the parameters, α and β, of the Dirichlet distribution are fixed. While in the SLDA model, the parameters, α and β, are biased by calculating the similarity between the input corpus and seed words. The fixed parameters are recorded as αbase and βbase, and the biased parameters are recorded as αd, βt,A, βt,P, and βt,N. In this paragraph, the semantic similarity between the word w and the seed word t is recorded as simw,t. The probability that w is a positive word is recorded as simw,P. The probability that w is a negative word is recorded as simw,N. The w is recorded as simw,A if it belongs to the opinion target word. The w is recorded as simw,O if it belongs to the sentiment opinion word.In the standard LDA, the parameterα is used to control the topic distribution probability of the document. For all documents in the corpus, the values of α are the same, leading to determine which topic word will be the document topic before generating the document hardly. However, the parameter αd in SLDA, which can be calculated by equation (5), will be set separately for each document (for each sentence in SLDA) based on the similarity between vocabulary and seed words, leading to determine a more ideal topic in advance before generating the document.
(5)αd=∑iNdsimwd,i,t∑t′T∑iNdsimwd,i,t′×αbase.In equation (5), Nd is the number of all words in the current sentence, T is the number of topics, wd,i is the i-th word in the current sentence, and t is the seed word.In the standard LDA, the parameterβ is used to control the word distribution of each topic, and the value of β is the same for each topic. In SLDA, parameters which can be calculated by equation (6), equation (7), and equation (8) will be set separately for each topic based on the similarity between vocabulary and seed words.
(6)βt,A=simw,A×βbase,(7)βt,P=simw,P×βbase,(8)βt,N=simw,N×βbase.Similar to the standard LDA, SLDA is solved by the Gibbs sampling method. The variables involved in the solution process are explained in Table2.Table 2
The partial symbolic description of SLDA model inference.
SymbolDescriptionnvt,AThe number of wordsv in topic t and category And,tThe number of words with the topict in the d-th sentenceβvt,uThe number of wordsv in topic t and category AEquation (9) is used to sample the topic of each sentence.
(9)pZd,n=tz¬d,n,y¬d,n,v¬d,n,⋅∝nwd,nt,A+βwd,nt,A∑vVnvt,A+βvt,A×nwd,nt,P+βwd,nt,P∑vVnvt,P+βvt,P×nwd,nt,N+βwd,nt,N∑vVnvt,N+βvt,N×nd,t+αd,t.Equation (10) and equation (11) are used to sample yd,n and vd,n.
(10)pyd,n=uzd,n=t,⋅αnwd,nt,u+βwd,nt,u×simwd,n,u∑vVnvt,u+βvt,u,u∈A,O,(11)pvd,n=qzd,n=t,⋅∝nwd,nt,q+βwd,nt,q×simwd,n,q∑vVnvt,q+βvt,q,q∈P,N.In the corpus, the approximate probability of the topict and the sentence d can be calculated by equation (12).
(12)θd=nd,t+αd,tnd+∑t′Tαd,t′.Witht as the topic, the approximate probability that the word wd,n is the opinion target can be calculated by equation (13), the approximate probability that the word wd,n is the positive opinion can be calculated by equation (14), and the approximate probability that the word wd,n is the negative opinion can be calculated by equation (15).
(13)φwd,nt,A=nwd,nt,A+βwd,nt,A∑vVnvt,A+βvt,A,(14)φwd,nt,P=nwd,nt,P+βwd,nt,P∑vVnvt,P+βvt,P,(15)φwd,nt,N=nwd,nt,N+βwd,nt,N∑vVnvt,N+βvt,N.(4) Gibbs Sampling Implementation of the SLDA Model. The Gibbs sampling process of SLDA mainly has the following steps:
(1)
Random initialization: randomly assign a topic number z to all sentences in the corpus, the topic numbers of all words in the sentence are also set to z, then the values of the indicator variables, y and u, are randomly set for all words in the sentence, where y∈A,O, u∈P,N.(2)
Traverse the corpus again, resample the topics of all words according to equation (9), and update the relevant values. Then, resample the indicator variables, y and u, according to equation (10) and equation (11), and update their values(3)
Repeat step 2 until the Gibbs sampling results converge(4)
Process the final results to improve the readability and save the resultsIn the initialization phase of the Gibbs sampling, a topic number is randomly assigned to each sentence in the corpus. For each word in the sentence, three categories are randomly assigned, which are determined by the indicator variables,y and u. Then, add 1 to the corresponding statistics. A part of the pseudocode in the initialization phase is shown in Algorithm 2.Algorithm 2. The initialization of Gibbs sampling.
1:ford=1 to Ddo2://Randomly assign topic t to sentences in Corpus3:t = randomint from (1, T);4:documentTopics [d] = t;5:documentTopicsCount [d][t]++;6:forw=1 to Ndo7://Randomly assign value to y8:y = randomint from (0,1);9:Ydw=y;10://Randomly assign value to u11:u = randomint from (0,1);12:Udw=u;13:ify ==0 then14://statistic of aspect targets with topic t plus 115:    aspectWordCount [w][t]++;16:end if17:ify ==1 and u ==0 then18://statistic of positive opinion with topic t plus 119:    positiveWordCount [w][t]++;20:end if21:ify ==1 and u ==1 then22://statistic of negative opinion with topic t plus 123:    negativeWordCount [w][t]++;24:end if25:end for26:end forIn the repeated iteration phase of Gibbs sampling, the topic of each sentence in the corpus and the category of each word in the sentence are resampled. Then, update the relevant statistics after each sampling. Repeat the process until the end of the iteration.In the result processing stage, the relevant valueφ can be calculated by equation (13), equation (14), and equation (15). The calculated φ value is a digital, and the variables required in the calculation are the topic number, the category to which the word belongs, and the number of the word in the vocabulary. The relevant sentence information is missing; accordingly, it is necessary to effectively organize various types of information for the user to view. We use result to represent the final result. The result.topic is the topic information, that is, the seed word set by ourselves, which also can be regarded as the comment category word. The result.word saves the original word. The result.wordType is the category of the current word (aspect target words, positive and negative opinion words). The result.sentences is the sentence to which the word belongs. The result.prob is the probability that the word becomes a member of the category which the comment belongs to. The first m results are generated for each category under all the topics, and the relevant pseudocode is shown in Algorithm 3.Algorithm 3. The result processing of the Gibbs sampling.
1://Create a 3d array to store results2: init results[][][]3:for topic in TopicList do4:for wordType in {A, P, N} do5:for v in V do6:result = new result7:result.topic = topic8:result.word = v9:result.wordType = wordType10:   Calculateφ according to Equation. (13), Equation. (14), Equation. (15)11:result.prob =φ12://Add result to the array13:results[topic][wordType][v] = result14:end for15:end for16:end for17://Next, sort the results18:for topic in TopicList do19:/Store sentences contain results20: finalResults[][][]21:for wordType in {A, P, N} do22://Sort byφvalue and get the results of the top m23:finalResults[topic][wordType] = getTopKByPhi(results[topic][wordType], m)24:fork=1 to m do25:   Query the corresponding sentence ofresult.v from the inverted index described in Section 3.1.1 and add it to the result26:end for27:end for28:end for29:return finalResultsAfter getting the finalResults, we can query the results according to both the topic and the wordType.


### 2.2. Optimized Model, HME-LDA, Based on MAXENT-LDA

Because the WordNet and SentiWordNet only support English, SLDA has no linguistic adaptation. Therefore, we propose an optimized model in this chapter, namely, the HME-LDA model that has the linguistic adaptation. The optimized model, HME-LDA, is proposed by combining with the MaxEnt-LDA [22] model and the SLDA model. And the HME-LDA uses an unsupervised hierarchical clustering method to generate the annotated data set required by the maximum entropy model. Based on these, the new model can be used for comment opinion mining in many other languages.


#### 2.2.1. Implementation Process

The overall process of the scheme that conducts aspect-based opinion mining based on the HME-LDA and the inverted index is shown in Figure4:
(1)
This step is the same as the implementation of step 1 in Section2.1.1(2)
Data preprocessing, whose main task is to remove stop words. The text in the corpus contains many useless stop words, such as “is” and “a,” which should be removed before further processing to avoid interference with the results(3)
Automatically generate annotated data sets by hierarchical clustering and train maximum entropy models for classification of the opinion target words and sentiment opinion words(4)
Enter the data into the HME-LDA model and perform the training to get the results(5)
Process the results for better readability. The results in the topic model are probability matrixes, whose readability is poor. To solve this problem, it is necessary to find the word which corresponds to the result that has a higher probability and find its original sentence by the inverted indexFigure 4
The flow chart of the scheme that is based on the HME-LDA and inverted index.As for Step 3 above, there are a lot of word order information in the corpus; thus, when the category of a word in the corpus can be determined, the featurewi−2,wi−1,wi,wi+1,wi+2 can be extracted. In HME-LDA, seed words are divided into topic seed words of aspect categories, seed words expressing positive sentiment, and seed words expressing negative sentiment. By scanning corpus, annotated feature sets, wi−2,wi−1,wiu,wi+1,wi+2, can be obtained, where u∈A,O. When u=A, wiA is the topic seed words, and when u=O, wiO is the sentiment seed words, and u is regarded as the label. However, the number of seed words is limited, so the training set obtained by scanning corpus may be insufficient in size. Therefore, the words in the corpus are considered to be clustered, and all the words in the category of the seed words are considered to have the same category as the seed words, and the word wiu in the scanned annotated feature set wi−2,wi−1,wiu,wi+1,wi+2 is replaced with the word in the category of seed words to get the new annotated data.


#### 2.2.2. MaxEnt-LDA Model

In order to realize the aspect-based opinion mining of reviews, Zhao et al. [22] proposed a MaxEnt-LDA model based on LDA. Figure 5 is the probability model diagram of MaxEnt-LDA [22].Figure 5
The probability diagram of MaxEnt-LDA.Adopting the second thought of optimizing LDA in Section2.1.2, MaxEnt-LDA further divides topics and increases the topic number by introducing another two indicator variables, yd,s,n and ud,s,n, that are similar to Zm,n. Three new topic-word distributions are generated by the Dirichlet distribution with the parameter β. Meanwhile, the original topic is further divided into categories A (aspect-term) and O (opinion word).In the MaxEnt-LDA model, the indicator variableyd,s,n is generated by the sampling of the maximum entropy model. In this model, the selected features of the maximum entropy model include the word and the part of speech, while there are three labels which are background word B, opinion word O, and the opinion target word A. The annotated training set is partially extracted from the SemEval data set and partially annotated by manual. The indicator variable yd,s,n is the same as Zm,n, both of which are sampled from the multinomial distribution generated by the Dirichlet distribution.MaxEnt-LDA increases both the type and number of classification and introduces the maximum entropy model. However, the MaxEnt-LDA model increases the dependence on the annotated data simultaneously. In order to pursue the unsupervised features of the model, there are two ways to improve the MaxEnt-LDA: one is to replace the maximum entropy model with other unsupervised classification models; the other is to use unsupervised methods to automatically label data sets to avoid the dependence on annotated data. Meanwhile, the parametersα and β can be considered for bias.


#### 2.2.3. The Description of the HME-LDA Model

In the HME-LDA model, seed words are directly set as an aspect category of reviews. Therefore, it only needs to classify the opinion targets of reviews and the review opinions by introducing the maximum entropy model as a classifier. The Bernoulli distribution with the parameterπd,n, where d indicates the d-th sentence and n indicates the n-th word, is jointly determined by the weight λd,n and the eigenvector fd,n of the maximum entropy model. A beta distribution with a parameter δd will be introduced as a priori to generate a Bernoulli distribution with a parameter Ωd for the classification of positive and negative sentiment opinion words. Figure 6 is the PGM.Figure 6
The probabilistic graphical model of HME-LDA.The generation process of the HME-LDA model is similar to that of the SLDA model in Section2.1. The difference is that yd,n is determined by the maximum entropy model rather than by the WordNet and SentiWordNet. And vd,n is determined by the parameter δd rather than by the calculation of the WordNet and SentiWordNet. The generation process of the HME-LDA model can refer to Section 2.1.4.In addition, because there are no methods to calculate the sentiment polarity in HME-LDA, it is necessary to set a sentiment opinion word whose sentiment polarity is positive or negative for each comment category.


#### 2.2.4. The Automatic Data Annotation Method Based on Hierarchical Clustering

The MaxEnt-LDA [22] model uses the word feature with a window whose size is 3 to extract the features from the annotated words. The selected features include the word and the word order wi−1,wi,wi+1, where wi is the current word. The selected features also include the features of grammatical rules of words POSi−1,POSi,POSi+1, where POSi indicates the part of speech of the current word (adjectives, nouns, verbs, etc.). The part-of-speech tagging requires the use of additional tools, while it is different for the accuracy of part-of-speech tagging due to different kinds of languages, and there is a possibility that the tagging tool is lacking. Therefore, the features selected in this chapter are only the words themselves.(1) The Process of Automatically Annotating Data. After identifying the feature information obtained from the training, the next step is to consider how to automatically label it. In order to extract the feature wi−2,wi−1,wi,wi+1,wi+2, the only thing that needs to do here is to determine the category of the word wi in the corpus with information of word order. The seed-word setting of HEM-LDA is explained in Section 2.2.3. In the HME-LDA, seed words can be divided into three categories: topic seed words of a review category, seed words expressing positive sentiments, and seed words expressing negative sentiments. And the opinion target words belong to the corresponding category of comments, both of which are the same kind of topic. Therefore, seed words can be divided into two categories: opinion target words and sentiment opinion words. The categories of these seed words are able to be determined. The annotated feature sets wi−2,wi−1,wiu,wi+1,wi+2 can be obtained by scanning the corpus, where u∈A,O. When u=A, wiA is the topic seed word. When u=O, wiO is the sentiment seed word, and u is the label.However, the number of seed words is limited, and the size of the training set obtained by scanning corpus may be insufficient. Therefore, we attempt to cluster the words in the corpus and treat that all words in the category of seed words have the same category as the seed words. Besides, the word in this category is used to replace the wordwiu in the scanned annotated feature set wi−2,wi−1,wiu,wi+1,wi+2, so as to obtain the new annotated data. The pseudocode is shown in Algorithm 4.Algorithm 4. The process of automatically labelling data.
1:fort in Topics do  //Process each seed word2:wordList = getWrodListFromCorpus(t) //Find the location where t appears from the corpus and get the corresponding word order3:wordCluster = getWordCluster(t) //Get all the words of the category t from the clustering results4:trainSet = new Set //Used to save labelled training samples5:for wOrder in wordList do6:for w in wordCluster do7:   replaceWord(wOrder, t, w)//Replace the word t in the word order with w8:   trainSet.add(wOrder, t.Type)//Add the label to which wOrder and t belong to the training set9:end for10:end for11:end for(2) The Selection of Clustering Method. In this section, in order to improve the domain adaptability of the model and avoid the different effects of the same value of K when using the data of different fields, and to avoid more parameter adjustments as well, we select the hierarchical clustering method which has no need for the number of clusters. The result of hierarchical clustering is shown in Figure 7.Figure 7
The results presentation of hierarchical clustering methods.In the tree structure generated by the hierarchical clustering results, the leaf nodes of the tree are words in the corpus. When looking for words of the same category, the intermediate node of the upper layer can be found by the current word. All the leaf nodes of the subtree with this intermediate node as the root node belong to the same category.


#### 2.2.5. The Inference of the HEM-LDA Model

The reasoning of the HME-LDA model mainly includes two parts. The first part is the inference of the maximum entropy model for the classification of the opinion target words and sentiment opinion words. And the second part is the inference of the optimized LDA model.(1) The Maximum Entropy Model. The maximum entropy model solves the classification problem actually. When the input of the model is x, the probability py∣x of the category y can be calculated by equation (16).
(16)py∣x=e∑i=1nλifix,y∑ye∑i=1nλifix,y.In equation (16), λi represents the weight vector, fix,y is the eigenfunction, and n is the number of categories. When using the maximum entropy model for the classification of the opinion target words and sentiment opinion words, it is necessary to select appropriate features. The MaxEnt-LDA [22] model uses two features, i.e., word order and part of speech. The part of speech tagging relies on some other tools that is different with different languages. In order to avoid using tools that rely on languages, this chapter chooses the word order as a feature. Section 2.2.4 gives the method of automatically annotating the training set.By training the maximum entropy model, the weightλu of the feature set fd,n can be obtained, and πd,nu can be obtained by equation (17), where d represents the d-th sentence, n is the n-th word in the sentence, u∈0,1 is the label collection, 0 represents the opinion target whose type is A, and 1 represents the sentiment opinion whose type is O.
(17)pyd,n∣fd,n=πd,nu=eλu×fd,n∑i=01eλi×fd,n.(2) The Model Inference of HME-LDA. In SLDA, the parameters, α and β, are offset by calculating the similarity between the input corpus and the seed words. The fixed parameters are denoted as αbase and βbase, and the offset parameters are denoted as αd, βt,A, βt,P, and βt,N. In Chapter 2, the above parameters are offset by using the semantic similarity calculation of the WordNet and the sentiment polarity calculation of the SentiWordNet. In this section, both the Word2Vec model and the cosine distance are used to calculate the similarity between words. The training corpus of the Word2Vec model is the all content in the corpus. The vector of the current word in the Word2Vec is represented by vw, and the vector of the seed word wt whose topic is t in Word2Vec is represented by vwt. The similarity between the word w and the topic word wt can be calculated by equation (18).
(18)simw,wt=cosθ=vw⋅vwtvw×vwt.Similar to the SLDA model, the HME-LDA will set the parameterαd separately for each document (each sentence in the SLDA model) based on the similarity between the vocabulary and seed words. The biased parameters can be calculated by equation (19).
(19)αd=∑i=1Ndsimwd,i,wt∑t′T∑iNdsimwd,i,wt′×αbase.In equation (19), Nd is the number of all words in the current sentence, T is the number of topics, wd,i is the i-th word in the current sentence, and t is the seed word.Based on the similarity between the vocabulary and seed words, the SLDA model will set parameters separately for each topic. These biased parameters can be calculated by equation (20), equation (21), and equation (22).
(20)βt,A=simw,wt×βbase,(21)βt,P=simw,wt,P×βbase,(22)βt,N=simw,wt,N×βbase.Different from SLDA, HME-LDA introduces the parameterδ to control the sentiment polarity of each sentence. The parameter δd,q can be calculated by equation (23).
(23)δd,q=∑iNdsimwd,i,wt,q∑q′∈P,N∑iNdsimwd,i,wt,q′×δbase.Similar to SLDA, HME-LDA uses the method of Gibbs sampling for solving. And the variables involved in the solution process have the same meanings as those in Tables1 and 2.Then, we use equation (24) to sample the topic of each sentence.
(24)pzd,n=t∣z¬d,n,y¬d,n,v¬d,n,⋅∝nwd,nt,A+βwd,nt,A∑vVnvt,A+βvt,A×nwd,nt,P+βwd,nt,P∑vVnvt,P+βvt,P×nwd,nt,N+βwd,nt,N∑vVnvt,N+βvt,N×nd,t+αd,t.Equation (25) and equation (26) are used to sample yd,n and vd,n,
(25)pyd,n=u∣zd,n=t,⋅∝nwd,nt,u+βwd,nt,u∑vVnvt,u+βvt,u×eλu×fd,n∑u′∈A,Oeλu′×fd,n,u∈A,O,(26)pvd,n=q∣zd,n=t,⋅∝nwd,nt,q+βwd,nt,q∑vVnvt,q+βvt,q×nd,q+δd,q.In the corpus, the approximate probability of the topict in sentence d can be calculated by equation (27),
(27)θdt=nd,t+αd,tnd+∑t′Tαd,t′.Witht as the topic, the approximate probability that the word wd,n is the opinion target can be calculated by equation (28),
(28)φwd,nt,A=nwd,nt,A+βwd,nt,A∑vVnvt,A+βvt,A.Witht as the topic, the approximate probability that the word wd,n is the positive opinion word can be calculated by equation (29),
(29)φwd,nt,P=nwd,nt,P+βwd,nt,P∑vVnvt,P+βvt,P.Witht as the topic, the approximate probability that the word wd,n is the negative opinion word can be calculated by equation (30),
(30)φwd,nt,N=nwd,nt,N+βwd,nt,N∑vVnvt,N+βvt,N.


## 2.1. Optimized Model, SLDA, Based on SentiWordNet and WordNet

Different from the text messages such as documents, blogs, and news, network comments tend to be shorter and often appear in the form of sentences. Please consider the following restaurant comments:(1)
“We, there were four of us, arrived at noon - the place was empty and the staff acted like we were imposing on them and they were very rude.”(2)
“Everything is always cooked to perfection, the service is excellent, the decor cool and understated.”In nonaspect opinion mining, the only thing you need to do is to analyse the sentiment polarity of sentences. For example, the word “rude” in the first sentence is negative, so the sentiment polarity of the first sentence is negative. In the second one, the sentiment polarity of the word “perfection” is positive, so the sentiment polarity of the second sentence is positive. This way that only judges the sentiment polarity of sentences does not apply to network comments. More meaningful information should be specific to the word pairs of <Aspect Opinion Target-Opinion> such as <staff-rude>, <cook-perfect>, and <service-excellent>, while the algorithm based on the topic model lacks readability and appointed key words, where the relevant original content fails to be directly found by the final result.The SLDA is an optimized LDA model based on SentiWordNet (a sentiment dictionary based on WordNet) and WordNet (a large database of English words). Since it is impossible for the LDA itself to separate opinions from opinion targets, this chapter adds a classification layer of opinion words and opinion target words based on the LDA to realize the separation of opinions and aspect opinion targets. The similarity between the word and the seed word in the text, which is reflected on LDA parameters, is calculated by setting the seed word and using the calculation tools of the vocabulary similarity in WordNet. Meanwhile, the opinion target is separated from the sentiment opinion words using tools that calculate the vocabulary sentiment in SentiWordNet. Aiming at the lack of readability of LDA results, we establish a belonging relationship among the clustering results, seed words, and original texts. Also, the SLDA model needs to set seed words, but has no need for the additional annotated data sets.In order to achieve the goal that users can quickly find what they want from massive comments by inquiring the index rather than by reading, there are three steps to do:(1)
Construct an inverted index to number sentences and words(2)
Determine the aspect category of comments, separate the aspect-based attribute words from the sentiment opinion words, and determine the aspect category to which they belong(3)
Enhance the readability of results. Users can see an overview of Domain-Aspect-Opinion and find the specific sentence by the inverted index of a wordIt is worth noting that the premise of our study is that the aspect category of the original corpus is known. The aspect category, which has the belonging relationship with the aspect opinion target, is usually given in advance by the original corpus, which can be learned by the user guide of the corpus. For example, the aspect opinion target “steak” belongs to the aspect category “Food.” In addition, both words and sentences can belong to an aspect category. Thus, what only need to do is to label aspect opinion target words and sentences with a specific aspect category.


### 2.1.1. Implementation Process

Figure2 is an overall flow chart of the scheme that conducts aspect-based opinion mining based on SLDA and the inverted index.
(1)
Construct an inverted index. The words in the corpus are numbered in the form of binary group<a, b>, where a is the serial number of the sentence and b is the serial number of the words in the sentence. In addition, the generation of the inverted list requires the removal of duplicate word and the recording of their numbers. The inverted list reserves the sentence number that contains the word and the position information of the word in the sentence, making it easier to retrieve with context information from the original corpus later(2)
Data preprocessing, whose main tasks are to extract the data required and remove stop words from the original corpus for making a training set. The formats of the original data sets are XML and CSV. It is necessary to extract comment statements by the corresponding labels and fields. And the text in the corpus contains many useless stop words, such as “is” and “a,” which should be removed before further processing to avoid interference with the training of the SLDA model(3)
Introduce preprocessed data into SLDA for model training and get clustering results. The setup of seed words, as well as the assist of WordNet and SentiWordNet, is required in this process(4)
Process the clustering results for better readability. The results in the topic model are probability matrix, whose readability is poor. To solve this problem, it is necessary to find the word corresponding to the result with higher probability and find its original sentence by the inverted indexFigure 2
The overall flow chart of the scheme that is based on SLDA and inverted index.In short, the SLDA model, trained with the preprocessed data of step 2 above, queries the original sentences that contain keywords from the original corpus by the inverted index of step 1 above.


### 2.1.2. The Optimized Direction of LDA

The expectation of every random variableμi of the Dirichlet distribution can be expressed as Eμi. The value of Eμi can be calculated by equation (1), where α represents the parameter of the Dirichlet distribution and K represents the number of topics:
(1)Eμi=αi∑i=1Kαi,whereα is a fixed value and the Eμi of each topic is same. Based on this, a biasing method of α can be explored to make the expectations of the corresponding probabilities of each topic different, so as to generate the topic bias. More visually, when α is fixed, it means that we fail to know which topic word to use before generating the document. When α is biased, the ideal topic is determined before the document is generated.Actually, the topic numberZm,n can be used as an indicator variable in the LDA model to control the selection of topic-word distribution. Based on this, more indicator variables can be introduced to refine the topic and obtain more topic-word distributions.From the above, there are two aspect opinion targets that the LDA model can optimize: the first is to bias the parameters,α and β, which can generate topic biases to improve the classification effect; the second is to introduce more indicator variables like Zm,n, which can generate more detailed topic classifications.


### 2.1.3. The Description of the SLDA Model

The standard LDA uses the document as the unit of topic allocation, while in the aspect-based opinion mining of the network comments, the sentence is often used as the unit of topic allocation, because there is no document with large contents in network comments and the topic allocation of vocabulary in network comments is actually more meaningful. In the aspect-based opinion mining of network comments, it is necessary to extract the aspect category of the comment, the opinion target, and the comment opinion (sentiment polarity) from the text. For example, in restaurant comments, “food,” “service,” and “ambiance” are the aspect categories of comments. In the “food” category, “steak” is the opinion target, and the evaluation of “good” for “steak” is the opinion of the comments (sentiment polarity).In the SLDA model, seed words are directly used as aspect categories of comments, while the opinion targets of comments and the comment opinions are separated by the introduced sentiment layer, and the positive and negative polarities of comments are classified as well. The PGM (Probabilistic Graphical Model) of SLDA is shown in Figure3.Figure 3
The probabilistic graphical model of SLDA.


### 2.1.4. The Generation Process of the SLDA Model

Only the parametersα and β, which, respectively, belong to the document-topic Dirichlet distribution and the topic-word Dirichlet distribution, are introduced into the standard LDA. In the SLDA model, seed words have been identified as the aspect category of comments. The variable yd∈A,O is introduced to represent the separation of the opinion target and the comment opinion. When yd=A, the current word is the opinion target of comments. When yd=O, the current word is the comment opinion. Meanwhile, the variable vd∈P,N is introduced into SLDA. When vd=P, the sentiment polarity of the current vocabulary is positive, and when vd=N, the sentiment polarity of the current vocabulary is negative. Both yd and vd are determined by the corresponding algorithms based on the WordNet and SentiWordNet. In the standard LDA model, the values of parameters, α and β, are fixed. In the SLDA model, different αd will be set for each sentence, and the parameters, βt,A, βt,P, and βt,N, will be set for the opinion target, the positive comments and the negative comments, respectively.In SLDA, the first step is to generate the sentence-topic distribution and determine a topic for each sentence by the multinomial distribution. Then, two influence factors,yd and vd, are determined by WordNet and SentiWordNet to indicate the aspect categories of words. By the SLDA generation method above, we finally select a topic-word distribution and determine the final word.The concrete generation process of SLDA is as follows:Firstly, the distributions of the opinion targetφt,A, the positive opinion word φt,P, and the negative opinion word φt,N are, respectively, extracted from the parameters, βt,A, βt,P, βt,N, where φt,A~Dirβt,A, φt,P~Dirβt,P, and φt,N~Dirβt,N.Similar to the standard LDA, for each sentence (document in the standard LDA), a topic distribution,θd~Dirαd, is extracted from the Dirichlet distribution with the parameter αd.Then, the wordwd,n in the sentence d extracts a topic number t, i.e., extracts zd,n~Multiθd.After determining the topic number of the wordwd,n, the classification of it remains to be determined. After determining the topic t in SLDA, in order to further classify the word as the opinion target word or the opinion of comments, the variables, yd,n∈A,O and vd,n∈P,N, are introduced into the SLDA model. The yd,n and vd,n point to the n-th word wd,n in the sentence d jointly. The yd,n is used to indicate that the word wd,n is the opinion target of comments or the comment opinion, which is extracted from the Bernoulli distribution on 0,1 with the parameter πd,n. The vd,n is used to indicate that the word wd,n is a positive or a negative comment opinion, which is extracted from the Bernoulli distribution on 0,1 with the parameter Ωd,n. The above two Bernoulli distributions are calculated by the WordNet and SentiWordNet. Finally, the word wd,n can be extracted according to equation (2):
(2)Wd,n~Multiφt,A,ifyd,n=A,Multiφt,A,ifyd,n=O,andvd,n=P,Multiφt,A,ifyd,n=O,andvd,n=N.Table1 gives the description of the related symbols in the SLDA model, which is useful for readers in reading.Table 1
The description of related symbols in the SLDA model.
SymbolDescriptionDThe total number of comments in the corpus. The unit of corpus is the sentenceTThe number of topicsVThe number of words in the corpuswd,nThen-th word in the d-th comment in the corpuszd,nThe topic ofd-th comment. The value is 1,⋯,Tyd,nAn indicator variable. The value isA,O. It is used to indicate the opinion target words and the sentiment opinion words.vd,nAn indicator variable. The value isP,N. It is used to indicate positive and negative sentiment opinion words.A,O,P,NThe opinion target, the sentiment opinion word, the positive sentiment word, the negative sentiment wordφt,AThe distribution of the opinion target generated by a priori Dirichlet distribution with the parameterβt,Aφt,PThe distribution of positive comment words generated by a priori Dirichlet distribution with parametersβt,Pφt,NThe distribution of negative comment words generated by a priori Dirichlet distribution with parametersβt,NθdThe distribution of sentence topic terms generated by a priori Dirichlet distribution with parameterαd


### 2.1.5. The Inference Process of the SLDA Model

The SLDA model consists of two major parts. One is the classification of the opinion target words and sentiment opinion words composed of the WordNet and SentiWordNet as well as the classification of positive and negative sentiment opinion words. The other is the LDA topic model. Besides, the seed words should be set before the inference of SLDA. Next, several modules will be introduced in turn.(1) The Setting of Seed Words. In SLDA, seed words are set as aspect categories of comments. For example, in the classic English comment set of the Restaurant, the aspect categories are food, service, ambiance, etc. If the corpus is the Restaurant English comment set, the seed words can be directly set as food, service, and ambience. The seed word is recorded as wt, where t∈1,⋯,T, that is to say, the number of seed words determines the number of topics in the SLDA model.(2) The Inference of the Word Classification Model Based on the WordNet and SentiWordNet. In the SLDA model, the Bernoulli distribution with parameter πd,n and Bernoulli distribution with parameter Ωd,n, which are, respectively, used to separate the opinion target words from sentiment opinion words and separate positive sentiment opinion words from negative sentiment opinion words, are related to the WordNet and SentiWordNet. The calculation of πd,n depends on the seed word wt.The words in WordNet have the feature of polysemy. To calculate the similarity between words, it is necessary to determine the exact meaning of a word. Therefore, before the model inference, it is necessary to determine the semantic interpretationst,k0 of the seed word wt in the WordNet. When the current word is wd,n, its semantic interpretation in the WordNet is sd,n,k, where k∈1,⋯,K and K is the number of semantic interpretations. After determining the semantics of the seed words, we can regard the seed word as the topic and the aspect category of the final opinion target. All nonsentiment opinion words will be grouped into a certain aspect category of comments. Therefore, the similarity between the semantics of the current word and the semantics of the seed word can be calculated, and the semantics with the greatest similarity can be determined as the meaning expressed by the word in the sentence finally. The degree of semantic similarity between different words in the WordNet is recorded as Sims1,s2; then, the semantic similarity, Simsd,n,k,st,k0, between the sd,n,k of wd,n and the st,k0 of each seed word wt can be calculated, where k∈1,⋯,K. Besides, K is the number of semantic interpretations, t∈1,⋯,T, and T is the number of seed words. In all calculation results, we choose the semantics with the max value of the similarity result and determine the largest result k′ as the semantics sd,n,k′ to which the current word wd,n belongs.After determining the semanticsd,n,k′ of the word wd,n, the sentiment polarity of sd,n,k′ can be queried in the SentiWordNet. In the SentiWordNet, the semantics of a word has three sentiment propensity probabilities: ρo indicates the probability that the semantics is objective (excluding sentiment polarity), pP indicates the probability that the semantics is positive, and ρN indicates the probability that the semantics is negative. Besides, po+pP+pN=1. It is believed that if the semantics is objective, the word is the opinion target, whose corresponding probability is po; otherwise, it is a sentiment opinion word. The sentiment scores of the semantics sd,n,k′ of the word wd,n are recorded as pd,n,ko, pd,n,kP, and pd,n,kN. In Section 2.1.4, the Bernoulli distributions with parameter πd,n and parameter Ωd,n can be determined by equation (3) and equation (4):
(3)πd,n=pd,n,ko,(4)Ωd,n=pd,n,kPpd,n,kP+pd,n,kN.So far, the model inference based on the WordNet and SentiWordNet has been completed. Algorithm1 is the pseudocode for calculating πd,n, Ωd,n.Algorithm 1. The calculation ofπd,n, Ωd,n.
1: Query the semantic listSlist of wd,n in WordNet2://Record the semantic value of the max similarity3:sw=0;4://Record the maximum similarity5:simmax=0;6://Each semanticsofwd,n7:fors∈slistdo8:forst∈stlistdo9://Sim() is a function provided by WordNet10:ifsimmax<Sims,stthen11:sw=s;12:sim_max=Sims,st;13;end if14:end for15:end for16: Use SentiWordNet to query the sentiment polarity of semanticSw17: Calculateπd,n using equation (3)18: CalculateΩd,n using equation (4)19:returnsw20:returnπd,n21:returnΩd,n(3) The Inference of the SLDA Model. In the standard LDA, the parameters, α and β, of the Dirichlet distribution are fixed. While in the SLDA model, the parameters, α and β, are biased by calculating the similarity between the input corpus and seed words. The fixed parameters are recorded as αbase and βbase, and the biased parameters are recorded as αd, βt,A, βt,P, and βt,N. In this paragraph, the semantic similarity between the word w and the seed word t is recorded as simw,t. The probability that w is a positive word is recorded as simw,P. The probability that w is a negative word is recorded as simw,N. The w is recorded as simw,A if it belongs to the opinion target word. The w is recorded as simw,O if it belongs to the sentiment opinion word.In the standard LDA, the parameterα is used to control the topic distribution probability of the document. For all documents in the corpus, the values of α are the same, leading to determine which topic word will be the document topic before generating the document hardly. However, the parameter αd in SLDA, which can be calculated by equation (5), will be set separately for each document (for each sentence in SLDA) based on the similarity between vocabulary and seed words, leading to determine a more ideal topic in advance before generating the document.
(5)αd=∑iNdsimwd,i,t∑t′T∑iNdsimwd,i,t′×αbase.In equation (5), Nd is the number of all words in the current sentence, T is the number of topics, wd,i is the i-th word in the current sentence, and t is the seed word.In the standard LDA, the parameterβ is used to control the word distribution of each topic, and the value of β is the same for each topic. In SLDA, parameters which can be calculated by equation (6), equation (7), and equation (8) will be set separately for each topic based on the similarity between vocabulary and seed words.
(6)βt,A=simw,A×βbase,(7)βt,P=simw,P×βbase,(8)βt,N=simw,N×βbase.Similar to the standard LDA, SLDA is solved by the Gibbs sampling method. The variables involved in the solution process are explained in Table2.Table 2
The partial symbolic description of SLDA model inference.
SymbolDescriptionnvt,AThe number of wordsv in topic t and category And,tThe number of words with the topict in the d-th sentenceβvt,uThe number of wordsv in topic t and category AEquation (9) is used to sample the topic of each sentence.
(9)pZd,n=tz¬d,n,y¬d,n,v¬d,n,⋅∝nwd,nt,A+βwd,nt,A∑vVnvt,A+βvt,A×nwd,nt,P+βwd,nt,P∑vVnvt,P+βvt,P×nwd,nt,N+βwd,nt,N∑vVnvt,N+βvt,N×nd,t+αd,t.Equation (10) and equation (11) are used to sample yd,n and vd,n.
(10)pyd,n=uzd,n=t,⋅αnwd,nt,u+βwd,nt,u×simwd,n,u∑vVnvt,u+βvt,u,u∈A,O,(11)pvd,n=qzd,n=t,⋅∝nwd,nt,q+βwd,nt,q×simwd,n,q∑vVnvt,q+βvt,q,q∈P,N.In the corpus, the approximate probability of the topict and the sentence d can be calculated by equation (12).
(12)θd=nd,t+αd,tnd+∑t′Tαd,t′.Witht as the topic, the approximate probability that the word wd,n is the opinion target can be calculated by equation (13), the approximate probability that the word wd,n is the positive opinion can be calculated by equation (14), and the approximate probability that the word wd,n is the negative opinion can be calculated by equation (15).
(13)φwd,nt,A=nwd,nt,A+βwd,nt,A∑vVnvt,A+βvt,A,(14)φwd,nt,P=nwd,nt,P+βwd,nt,P∑vVnvt,P+βvt,P,(15)φwd,nt,N=nwd,nt,N+βwd,nt,N∑vVnvt,N+βvt,N.(4) Gibbs Sampling Implementation of the SLDA Model. The Gibbs sampling process of SLDA mainly has the following steps:
(1)
Random initialization: randomly assign a topic number z to all sentences in the corpus, the topic numbers of all words in the sentence are also set to z, then the values of the indicator variables, y and u, are randomly set for all words in the sentence, where y∈A,O, u∈P,N.(2)
Traverse the corpus again, resample the topics of all words according to equation (9), and update the relevant values. Then, resample the indicator variables, y and u, according to equation (10) and equation (11), and update their values(3)
Repeat step 2 until the Gibbs sampling results converge(4)
Process the final results to improve the readability and save the resultsIn the initialization phase of the Gibbs sampling, a topic number is randomly assigned to each sentence in the corpus. For each word in the sentence, three categories are randomly assigned, which are determined by the indicator variables,y and u. Then, add 1 to the corresponding statistics. A part of the pseudocode in the initialization phase is shown in Algorithm 2.Algorithm 2. The initialization of Gibbs sampling.
1:ford=1 to Ddo2://Randomly assign topic t to sentences in Corpus3:t = randomint from (1, T);4:documentTopics [d] = t;5:documentTopicsCount [d][t]++;6:forw=1 to Ndo7://Randomly assign value to y8:y = randomint from (0,1);9:Ydw=y;10://Randomly assign value to u11:u = randomint from (0,1);12:Udw=u;13:ify ==0 then14://statistic of aspect targets with topic t plus 115:    aspectWordCount [w][t]++;16:end if17:ify ==1 and u ==0 then18://statistic of positive opinion with topic t plus 119:    positiveWordCount [w][t]++;20:end if21:ify ==1 and u ==1 then22://statistic of negative opinion with topic t plus 123:    negativeWordCount [w][t]++;24:end if25:end for26:end forIn the repeated iteration phase of Gibbs sampling, the topic of each sentence in the corpus and the category of each word in the sentence are resampled. Then, update the relevant statistics after each sampling. Repeat the process until the end of the iteration.In the result processing stage, the relevant valueφ can be calculated by equation (13), equation (14), and equation (15). The calculated φ value is a digital, and the variables required in the calculation are the topic number, the category to which the word belongs, and the number of the word in the vocabulary. The relevant sentence information is missing; accordingly, it is necessary to effectively organize various types of information for the user to view. We use result to represent the final result. The result.topic is the topic information, that is, the seed word set by ourselves, which also can be regarded as the comment category word. The result.word saves the original word. The result.wordType is the category of the current word (aspect target words, positive and negative opinion words). The result.sentences is the sentence to which the word belongs. The result.prob is the probability that the word becomes a member of the category which the comment belongs to. The first m results are generated for each category under all the topics, and the relevant pseudocode is shown in Algorithm 3.Algorithm 3. The result processing of the Gibbs sampling.
1://Create a 3d array to store results2: init results[][][]3:for topic in TopicList do4:for wordType in {A, P, N} do5:for v in V do6:result = new result7:result.topic = topic8:result.word = v9:result.wordType = wordType10:   Calculateφ according to Equation. (13), Equation. (14), Equation. (15)11:result.prob =φ12://Add result to the array13:results[topic][wordType][v] = result14:end for15:end for16:end for17://Next, sort the results18:for topic in TopicList do19:/Store sentences contain results20: finalResults[][][]21:for wordType in {A, P, N} do22://Sort byφvalue and get the results of the top m23:finalResults[topic][wordType] = getTopKByPhi(results[topic][wordType], m)24:fork=1 to m do25:   Query the corresponding sentence ofresult.v from the inverted index described in Section 3.1.1 and add it to the result26:end for27:end for28:end for29:return finalResultsAfter getting the finalResults, we can query the results according to both the topic and the wordType.


## 2.1.1. Implementation Process

Figure2 is an overall flow chart of the scheme that conducts aspect-based opinion mining based on SLDA and the inverted index.
(1)
Construct an inverted index. The words in the corpus are numbered in the form of binary group<a, b>, where a is the serial number of the sentence and b is the serial number of the words in the sentence. In addition, the generation of the inverted list requires the removal of duplicate word and the recording of their numbers. The inverted list reserves the sentence number that contains the word and the position information of the word in the sentence, making it easier to retrieve with context information from the original corpus later(2)
Data preprocessing, whose main tasks are to extract the data required and remove stop words from the original corpus for making a training set. The formats of the original data sets are XML and CSV. It is necessary to extract comment statements by the corresponding labels and fields. And the text in the corpus contains many useless stop words, such as “is” and “a,” which should be removed before further processing to avoid interference with the training of the SLDA model(3)
Introduce preprocessed data into SLDA for model training and get clustering results. The setup of seed words, as well as the assist of WordNet and SentiWordNet, is required in this process(4)
Process the clustering results for better readability. The results in the topic model are probability matrix, whose readability is poor. To solve this problem, it is necessary to find the word corresponding to the result with higher probability and find its original sentence by the inverted indexFigure 2
The overall flow chart of the scheme that is based on SLDA and inverted index.In short, the SLDA model, trained with the preprocessed data of step 2 above, queries the original sentences that contain keywords from the original corpus by the inverted index of step 1 above.


## 2.1.2. The Optimized Direction of LDA

The expectation of every random variableμi of the Dirichlet distribution can be expressed as Eμi. The value of Eμi can be calculated by equation (1), where α represents the parameter of the Dirichlet distribution and K represents the number of topics:
(1)Eμi=αi∑i=1Kαi,whereα is a fixed value and the Eμi of each topic is same. Based on this, a biasing method of α can be explored to make the expectations of the corresponding probabilities of each topic different, so as to generate the topic bias. More visually, when α is fixed, it means that we fail to know which topic word to use before generating the document. When α is biased, the ideal topic is determined before the document is generated.Actually, the topic numberZm,n can be used as an indicator variable in the LDA model to control the selection of topic-word distribution. Based on this, more indicator variables can be introduced to refine the topic and obtain more topic-word distributions.From the above, there are two aspect opinion targets that the LDA model can optimize: the first is to bias the parameters,α and β, which can generate topic biases to improve the classification effect; the second is to introduce more indicator variables like Zm,n, which can generate more detailed topic classifications.


## 2.1.3. The Description of the SLDA Model

The standard LDA uses the document as the unit of topic allocation, while in the aspect-based opinion mining of the network comments, the sentence is often used as the unit of topic allocation, because there is no document with large contents in network comments and the topic allocation of vocabulary in network comments is actually more meaningful. In the aspect-based opinion mining of network comments, it is necessary to extract the aspect category of the comment, the opinion target, and the comment opinion (sentiment polarity) from the text. For example, in restaurant comments, “food,” “service,” and “ambiance” are the aspect categories of comments. In the “food” category, “steak” is the opinion target, and the evaluation of “good” for “steak” is the opinion of the comments (sentiment polarity).In the SLDA model, seed words are directly used as aspect categories of comments, while the opinion targets of comments and the comment opinions are separated by the introduced sentiment layer, and the positive and negative polarities of comments are classified as well. The PGM (Probabilistic Graphical Model) of SLDA is shown in Figure3.Figure 3
The probabilistic graphical model of SLDA.


## 2.1.4. The Generation Process of the SLDA Model

Only the parametersα and β, which, respectively, belong to the document-topic Dirichlet distribution and the topic-word Dirichlet distribution, are introduced into the standard LDA. In the SLDA model, seed words have been identified as the aspect category of comments. The variable yd∈A,O is introduced to represent the separation of the opinion target and the comment opinion. When yd=A, the current word is the opinion target of comments. When yd=O, the current word is the comment opinion. Meanwhile, the variable vd∈P,N is introduced into SLDA. When vd=P, the sentiment polarity of the current vocabulary is positive, and when vd=N, the sentiment polarity of the current vocabulary is negative. Both yd and vd are determined by the corresponding algorithms based on the WordNet and SentiWordNet. In the standard LDA model, the values of parameters, α and β, are fixed. In the SLDA model, different αd will be set for each sentence, and the parameters, βt,A, βt,P, and βt,N, will be set for the opinion target, the positive comments and the negative comments, respectively.In SLDA, the first step is to generate the sentence-topic distribution and determine a topic for each sentence by the multinomial distribution. Then, two influence factors,yd and vd, are determined by WordNet and SentiWordNet to indicate the aspect categories of words. By the SLDA generation method above, we finally select a topic-word distribution and determine the final word.The concrete generation process of SLDA is as follows:Firstly, the distributions of the opinion targetφt,A, the positive opinion word φt,P, and the negative opinion word φt,N are, respectively, extracted from the parameters, βt,A, βt,P, βt,N, where φt,A~Dirβt,A, φt,P~Dirβt,P, and φt,N~Dirβt,N.Similar to the standard LDA, for each sentence (document in the standard LDA), a topic distribution,θd~Dirαd, is extracted from the Dirichlet distribution with the parameter αd.Then, the wordwd,n in the sentence d extracts a topic number t, i.e., extracts zd,n~Multiθd.After determining the topic number of the wordwd,n, the classification of it remains to be determined. After determining the topic t in SLDA, in order to further classify the word as the opinion target word or the opinion of comments, the variables, yd,n∈A,O and vd,n∈P,N, are introduced into the SLDA model. The yd,n and vd,n point to the n-th word wd,n in the sentence d jointly. The yd,n is used to indicate that the word wd,n is the opinion target of comments or the comment opinion, which is extracted from the Bernoulli distribution on 0,1 with the parameter πd,n. The vd,n is used to indicate that the word wd,n is a positive or a negative comment opinion, which is extracted from the Bernoulli distribution on 0,1 with the parameter Ωd,n. The above two Bernoulli distributions are calculated by the WordNet and SentiWordNet. Finally, the word wd,n can be extracted according to equation (2):
(2)Wd,n~Multiφt,A,ifyd,n=A,Multiφt,A,ifyd,n=O,andvd,n=P,Multiφt,A,ifyd,n=O,andvd,n=N.Table1 gives the description of the related symbols in the SLDA model, which is useful for readers in reading.Table 1
The description of related symbols in the SLDA model.
SymbolDescriptionDThe total number of comments in the corpus. The unit of corpus is the sentenceTThe number of topicsVThe number of words in the corpuswd,nThen-th word in the d-th comment in the corpuszd,nThe topic ofd-th comment. The value is 1,⋯,Tyd,nAn indicator variable. The value isA,O. It is used to indicate the opinion target words and the sentiment opinion words.vd,nAn indicator variable. The value isP,N. It is used to indicate positive and negative sentiment opinion words.A,O,P,NThe opinion target, the sentiment opinion word, the positive sentiment word, the negative sentiment wordφt,AThe distribution of the opinion target generated by a priori Dirichlet distribution with the parameterβt,Aφt,PThe distribution of positive comment words generated by a priori Dirichlet distribution with parametersβt,Pφt,NThe distribution of negative comment words generated by a priori Dirichlet distribution with parametersβt,NθdThe distribution of sentence topic terms generated by a priori Dirichlet distribution with parameterαd


## 2.1.5. The Inference Process of the SLDA Model

The SLDA model consists of two major parts. One is the classification of the opinion target words and sentiment opinion words composed of the WordNet and SentiWordNet as well as the classification of positive and negative sentiment opinion words. The other is the LDA topic model. Besides, the seed words should be set before the inference of SLDA. Next, several modules will be introduced in turn.(1) The Setting of Seed Words. In SLDA, seed words are set as aspect categories of comments. For example, in the classic English comment set of the Restaurant, the aspect categories are food, service, ambiance, etc. If the corpus is the Restaurant English comment set, the seed words can be directly set as food, service, and ambience. The seed word is recorded as wt, where t∈1,⋯,T, that is to say, the number of seed words determines the number of topics in the SLDA model.(2) The Inference of the Word Classification Model Based on the WordNet and SentiWordNet. In the SLDA model, the Bernoulli distribution with parameter πd,n and Bernoulli distribution with parameter Ωd,n, which are, respectively, used to separate the opinion target words from sentiment opinion words and separate positive sentiment opinion words from negative sentiment opinion words, are related to the WordNet and SentiWordNet. The calculation of πd,n depends on the seed word wt.The words in WordNet have the feature of polysemy. To calculate the similarity between words, it is necessary to determine the exact meaning of a word. Therefore, before the model inference, it is necessary to determine the semantic interpretationst,k0 of the seed word wt in the WordNet. When the current word is wd,n, its semantic interpretation in the WordNet is sd,n,k, where k∈1,⋯,K and K is the number of semantic interpretations. After determining the semantics of the seed words, we can regard the seed word as the topic and the aspect category of the final opinion target. All nonsentiment opinion words will be grouped into a certain aspect category of comments. Therefore, the similarity between the semantics of the current word and the semantics of the seed word can be calculated, and the semantics with the greatest similarity can be determined as the meaning expressed by the word in the sentence finally. The degree of semantic similarity between different words in the WordNet is recorded as Sims1,s2; then, the semantic similarity, Simsd,n,k,st,k0, between the sd,n,k of wd,n and the st,k0 of each seed word wt can be calculated, where k∈1,⋯,K. Besides, K is the number of semantic interpretations, t∈1,⋯,T, and T is the number of seed words. In all calculation results, we choose the semantics with the max value of the similarity result and determine the largest result k′ as the semantics sd,n,k′ to which the current word wd,n belongs.After determining the semanticsd,n,k′ of the word wd,n, the sentiment polarity of sd,n,k′ can be queried in the SentiWordNet. In the SentiWordNet, the semantics of a word has three sentiment propensity probabilities: ρo indicates the probability that the semantics is objective (excluding sentiment polarity), pP indicates the probability that the semantics is positive, and ρN indicates the probability that the semantics is negative. Besides, po+pP+pN=1. It is believed that if the semantics is objective, the word is the opinion target, whose corresponding probability is po; otherwise, it is a sentiment opinion word. The sentiment scores of the semantics sd,n,k′ of the word wd,n are recorded as pd,n,ko, pd,n,kP, and pd,n,kN. In Section 2.1.4, the Bernoulli distributions with parameter πd,n and parameter Ωd,n can be determined by equation (3) and equation (4):
(3)πd,n=pd,n,ko,(4)Ωd,n=pd,n,kPpd,n,kP+pd,n,kN.So far, the model inference based on the WordNet and SentiWordNet has been completed. Algorithm1 is the pseudocode for calculating πd,n, Ωd,n.Algorithm 1. The calculation ofπd,n, Ωd,n.
1: Query the semantic listSlist of wd,n in WordNet2://Record the semantic value of the max similarity3:sw=0;4://Record the maximum similarity5:simmax=0;6://Each semanticsofwd,n7:fors∈slistdo8:forst∈stlistdo9://Sim() is a function provided by WordNet10:ifsimmax<Sims,stthen11:sw=s;12:sim_max=Sims,st;13;end if14:end for15:end for16: Use SentiWordNet to query the sentiment polarity of semanticSw17: Calculateπd,n using equation (3)18: CalculateΩd,n using equation (4)19:returnsw20:returnπd,n21:returnΩd,n(3) The Inference of the SLDA Model. In the standard LDA, the parameters, α and β, of the Dirichlet distribution are fixed. While in the SLDA model, the parameters, α and β, are biased by calculating the similarity between the input corpus and seed words. The fixed parameters are recorded as αbase and βbase, and the biased parameters are recorded as αd, βt,A, βt,P, and βt,N. In this paragraph, the semantic similarity between the word w and the seed word t is recorded as simw,t. The probability that w is a positive word is recorded as simw,P. The probability that w is a negative word is recorded as simw,N. The w is recorded as simw,A if it belongs to the opinion target word. The w is recorded as simw,O if it belongs to the sentiment opinion word.In the standard LDA, the parameterα is used to control the topic distribution probability of the document. For all documents in the corpus, the values of α are the same, leading to determine which topic word will be the document topic before generating the document hardly. However, the parameter αd in SLDA, which can be calculated by equation (5), will be set separately for each document (for each sentence in SLDA) based on the similarity between vocabulary and seed words, leading to determine a more ideal topic in advance before generating the document.
(5)αd=∑iNdsimwd,i,t∑t′T∑iNdsimwd,i,t′×αbase.In equation (5), Nd is the number of all words in the current sentence, T is the number of topics, wd,i is the i-th word in the current sentence, and t is the seed word.In the standard LDA, the parameterβ is used to control the word distribution of each topic, and the value of β is the same for each topic. In SLDA, parameters which can be calculated by equation (6), equation (7), and equation (8) will be set separately for each topic based on the similarity between vocabulary and seed words.
(6)βt,A=simw,A×βbase,(7)βt,P=simw,P×βbase,(8)βt,N=simw,N×βbase.Similar to the standard LDA, SLDA is solved by the Gibbs sampling method. The variables involved in the solution process are explained in Table2.Table 2
The partial symbolic description of SLDA model inference.
SymbolDescriptionnvt,AThe number of wordsv in topic t and category And,tThe number of words with the topict in the d-th sentenceβvt,uThe number of wordsv in topic t and category AEquation (9) is used to sample the topic of each sentence.
(9)pZd,n=tz¬d,n,y¬d,n,v¬d,n,⋅∝nwd,nt,A+βwd,nt,A∑vVnvt,A+βvt,A×nwd,nt,P+βwd,nt,P∑vVnvt,P+βvt,P×nwd,nt,N+βwd,nt,N∑vVnvt,N+βvt,N×nd,t+αd,t.Equation (10) and equation (11) are used to sample yd,n and vd,n.
(10)pyd,n=uzd,n=t,⋅αnwd,nt,u+βwd,nt,u×simwd,n,u∑vVnvt,u+βvt,u,u∈A,O,(11)pvd,n=qzd,n=t,⋅∝nwd,nt,q+βwd,nt,q×simwd,n,q∑vVnvt,q+βvt,q,q∈P,N.In the corpus, the approximate probability of the topict and the sentence d can be calculated by equation (12).
(12)θd=nd,t+αd,tnd+∑t′Tαd,t′.Witht as the topic, the approximate probability that the word wd,n is the opinion target can be calculated by equation (13), the approximate probability that the word wd,n is the positive opinion can be calculated by equation (14), and the approximate probability that the word wd,n is the negative opinion can be calculated by equation (15).
(13)φwd,nt,A=nwd,nt,A+βwd,nt,A∑vVnvt,A+βvt,A,(14)φwd,nt,P=nwd,nt,P+βwd,nt,P∑vVnvt,P+βvt,P,(15)φwd,nt,N=nwd,nt,N+βwd,nt,N∑vVnvt,N+βvt,N.(4) Gibbs Sampling Implementation of the SLDA Model. The Gibbs sampling process of SLDA mainly has the following steps:
(1)
Random initialization: randomly assign a topic number z to all sentences in the corpus, the topic numbers of all words in the sentence are also set to z, then the values of the indicator variables, y and u, are randomly set for all words in the sentence, where y∈A,O, u∈P,N.(2)
Traverse the corpus again, resample the topics of all words according to equation (9), and update the relevant values. Then, resample the indicator variables, y and u, according to equation (10) and equation (11), and update their values(3)
Repeat step 2 until the Gibbs sampling results converge(4)
Process the final results to improve the readability and save the resultsIn the initialization phase of the Gibbs sampling, a topic number is randomly assigned to each sentence in the corpus. For each word in the sentence, three categories are randomly assigned, which are determined by the indicator variables,y and u. Then, add 1 to the corresponding statistics. A part of the pseudocode in the initialization phase is shown in Algorithm 2.Algorithm 2. The initialization of Gibbs sampling.
1:ford=1 to Ddo2://Randomly assign topic t to sentences in Corpus3:t = randomint from (1, T);4:documentTopics [d] = t;5:documentTopicsCount [d][t]++;6:forw=1 to Ndo7://Randomly assign value to y8:y = randomint from (0,1);9:Ydw=y;10://Randomly assign value to u11:u = randomint from (0,1);12:Udw=u;13:ify ==0 then14://statistic of aspect targets with topic t plus 115:    aspectWordCount [w][t]++;16:end if17:ify ==1 and u ==0 then18://statistic of positive opinion with topic t plus 119:    positiveWordCount [w][t]++;20:end if21:ify ==1 and u ==1 then22://statistic of negative opinion with topic t plus 123:    negativeWordCount [w][t]++;24:end if25:end for26:end forIn the repeated iteration phase of Gibbs sampling, the topic of each sentence in the corpus and the category of each word in the sentence are resampled. Then, update the relevant statistics after each sampling. Repeat the process until the end of the iteration.In the result processing stage, the relevant valueφ can be calculated by equation (13), equation (14), and equation (15). The calculated φ value is a digital, and the variables required in the calculation are the topic number, the category to which the word belongs, and the number of the word in the vocabulary. The relevant sentence information is missing; accordingly, it is necessary to effectively organize various types of information for the user to view. We use result to represent the final result. The result.topic is the topic information, that is, the seed word set by ourselves, which also can be regarded as the comment category word. The result.word saves the original word. The result.wordType is the category of the current word (aspect target words, positive and negative opinion words). The result.sentences is the sentence to which the word belongs. The result.prob is the probability that the word becomes a member of the category which the comment belongs to. The first m results are generated for each category under all the topics, and the relevant pseudocode is shown in Algorithm 3.Algorithm 3. The result processing of the Gibbs sampling.
1://Create a 3d array to store results2: init results[][][]3:for topic in TopicList do4:for wordType in {A, P, N} do5:for v in V do6:result = new result7:result.topic = topic8:result.word = v9:result.wordType = wordType10:   Calculateφ according to Equation. (13), Equation. (14), Equation. (15)11:result.prob =φ12://Add result to the array13:results[topic][wordType][v] = result14:end for15:end for16:end for17://Next, sort the results18:for topic in TopicList do19:/Store sentences contain results20: finalResults[][][]21:for wordType in {A, P, N} do22://Sort byφvalue and get the results of the top m23:finalResults[topic][wordType] = getTopKByPhi(results[topic][wordType], m)24:fork=1 to m do25:   Query the corresponding sentence ofresult.v from the inverted index described in Section 3.1.1 and add it to the result26:end for27:end for28:end for29:return finalResultsAfter getting the finalResults, we can query the results according to both the topic and the wordType.


## 2.2. Optimized Model, HME-LDA, Based on MAXENT-LDA

Because the WordNet and SentiWordNet only support English, SLDA has no linguistic adaptation. Therefore, we propose an optimized model in this chapter, namely, the HME-LDA model that has the linguistic adaptation. The optimized model, HME-LDA, is proposed by combining with the MaxEnt-LDA [22] model and the SLDA model. And the HME-LDA uses an unsupervised hierarchical clustering method to generate the annotated data set required by the maximum entropy model. Based on these, the new model can be used for comment opinion mining in many other languages.


### 2.2.1. Implementation Process

The overall process of the scheme that conducts aspect-based opinion mining based on the HME-LDA and the inverted index is shown in Figure4:
(1)
This step is the same as the implementation of step 1 in Section2.1.1(2)
Data preprocessing, whose main task is to remove stop words. The text in the corpus contains many useless stop words, such as “is” and “a,” which should be removed before further processing to avoid interference with the results(3)
Automatically generate annotated data sets by hierarchical clustering and train maximum entropy models for classification of the opinion target words and sentiment opinion words(4)
Enter the data into the HME-LDA model and perform the training to get the results(5)
Process the results for better readability. The results in the topic model are probability matrixes, whose readability is poor. To solve this problem, it is necessary to find the word which corresponds to the result that has a higher probability and find its original sentence by the inverted indexFigure 4
The flow chart of the scheme that is based on the HME-LDA and inverted index.As for Step 3 above, there are a lot of word order information in the corpus; thus, when the category of a word in the corpus can be determined, the featurewi−2,wi−1,wi,wi+1,wi+2 can be extracted. In HME-LDA, seed words are divided into topic seed words of aspect categories, seed words expressing positive sentiment, and seed words expressing negative sentiment. By scanning corpus, annotated feature sets, wi−2,wi−1,wiu,wi+1,wi+2, can be obtained, where u∈A,O. When u=A, wiA is the topic seed words, and when u=O, wiO is the sentiment seed words, and u is regarded as the label. However, the number of seed words is limited, so the training set obtained by scanning corpus may be insufficient in size. Therefore, the words in the corpus are considered to be clustered, and all the words in the category of the seed words are considered to have the same category as the seed words, and the word wiu in the scanned annotated feature set wi−2,wi−1,wiu,wi+1,wi+2 is replaced with the word in the category of seed words to get the new annotated data.


### 2.2.2. MaxEnt-LDA Model

In order to realize the aspect-based opinion mining of reviews, Zhao et al. [22] proposed a MaxEnt-LDA model based on LDA. Figure 5 is the probability model diagram of MaxEnt-LDA [22].Figure 5
The probability diagram of MaxEnt-LDA.Adopting the second thought of optimizing LDA in Section2.1.2, MaxEnt-LDA further divides topics and increases the topic number by introducing another two indicator variables, yd,s,n and ud,s,n, that are similar to Zm,n. Three new topic-word distributions are generated by the Dirichlet distribution with the parameter β. Meanwhile, the original topic is further divided into categories A (aspect-term) and O (opinion word).In the MaxEnt-LDA model, the indicator variableyd,s,n is generated by the sampling of the maximum entropy model. In this model, the selected features of the maximum entropy model include the word and the part of speech, while there are three labels which are background word B, opinion word O, and the opinion target word A. The annotated training set is partially extracted from the SemEval data set and partially annotated by manual. The indicator variable yd,s,n is the same as Zm,n, both of which are sampled from the multinomial distribution generated by the Dirichlet distribution.MaxEnt-LDA increases both the type and number of classification and introduces the maximum entropy model. However, the MaxEnt-LDA model increases the dependence on the annotated data simultaneously. In order to pursue the unsupervised features of the model, there are two ways to improve the MaxEnt-LDA: one is to replace the maximum entropy model with other unsupervised classification models; the other is to use unsupervised methods to automatically label data sets to avoid the dependence on annotated data. Meanwhile, the parametersα and β can be considered for bias.


### 2.2.3. The Description of the HME-LDA Model

In the HME-LDA model, seed words are directly set as an aspect category of reviews. Therefore, it only needs to classify the opinion targets of reviews and the review opinions by introducing the maximum entropy model as a classifier. The Bernoulli distribution with the parameterπd,n, where d indicates the d-th sentence and n indicates the n-th word, is jointly determined by the weight λd,n and the eigenvector fd,n of the maximum entropy model. A beta distribution with a parameter δd will be introduced as a priori to generate a Bernoulli distribution with a parameter Ωd for the classification of positive and negative sentiment opinion words. Figure 6 is the PGM.Figure 6
The probabilistic graphical model of HME-LDA.The generation process of the HME-LDA model is similar to that of the SLDA model in Section2.1. The difference is that yd,n is determined by the maximum entropy model rather than by the WordNet and SentiWordNet. And vd,n is determined by the parameter δd rather than by the calculation of the WordNet and SentiWordNet. The generation process of the HME-LDA model can refer to Section 2.1.4.In addition, because there are no methods to calculate the sentiment polarity in HME-LDA, it is necessary to set a sentiment opinion word whose sentiment polarity is positive or negative for each comment category.


### 2.2.4. The Automatic Data Annotation Method Based on Hierarchical Clustering

The MaxEnt-LDA [22] model uses the word feature with a window whose size is 3 to extract the features from the annotated words. The selected features include the word and the word order wi−1,wi,wi+1, where wi is the current word. The selected features also include the features of grammatical rules of words POSi−1,POSi,POSi+1, where POSi indicates the part of speech of the current word (adjectives, nouns, verbs, etc.). The part-of-speech tagging requires the use of additional tools, while it is different for the accuracy of part-of-speech tagging due to different kinds of languages, and there is a possibility that the tagging tool is lacking. Therefore, the features selected in this chapter are only the words themselves.(1) The Process of Automatically Annotating Data. After identifying the feature information obtained from the training, the next step is to consider how to automatically label it. In order to extract the feature wi−2,wi−1,wi,wi+1,wi+2, the only thing that needs to do here is to determine the category of the word wi in the corpus with information of word order. The seed-word setting of HEM-LDA is explained in Section 2.2.3. In the HME-LDA, seed words can be divided into three categories: topic seed words of a review category, seed words expressing positive sentiments, and seed words expressing negative sentiments. And the opinion target words belong to the corresponding category of comments, both of which are the same kind of topic. Therefore, seed words can be divided into two categories: opinion target words and sentiment opinion words. The categories of these seed words are able to be determined. The annotated feature sets wi−2,wi−1,wiu,wi+1,wi+2 can be obtained by scanning the corpus, where u∈A,O. When u=A, wiA is the topic seed word. When u=O, wiO is the sentiment seed word, and u is the label.However, the number of seed words is limited, and the size of the training set obtained by scanning corpus may be insufficient. Therefore, we attempt to cluster the words in the corpus and treat that all words in the category of seed words have the same category as the seed words. Besides, the word in this category is used to replace the wordwiu in the scanned annotated feature set wi−2,wi−1,wiu,wi+1,wi+2, so as to obtain the new annotated data. The pseudocode is shown in Algorithm 4.Algorithm 4. The process of automatically labelling data.
1:fort in Topics do  //Process each seed word2:wordList = getWrodListFromCorpus(t) //Find the location where t appears from the corpus and get the corresponding word order3:wordCluster = getWordCluster(t) //Get all the words of the category t from the clustering results4:trainSet = new Set //Used to save labelled training samples5:for wOrder in wordList do6:for w in wordCluster do7:   replaceWord(wOrder, t, w)//Replace the word t in the word order with w8:   trainSet.add(wOrder, t.Type)//Add the label to which wOrder and t belong to the training set9:end for10:end for11:end for(2) The Selection of Clustering Method. In this section, in order to improve the domain adaptability of the model and avoid the different effects of the same value of K when using the data of different fields, and to avoid more parameter adjustments as well, we select the hierarchical clustering method which has no need for the number of clusters. The result of hierarchical clustering is shown in Figure 7.Figure 7
The results presentation of hierarchical clustering methods.In the tree structure generated by the hierarchical clustering results, the leaf nodes of the tree are words in the corpus. When looking for words of the same category, the intermediate node of the upper layer can be found by the current word. All the leaf nodes of the subtree with this intermediate node as the root node belong to the same category.


### 2.2.5. The Inference of the HEM-LDA Model

The reasoning of the HME-LDA model mainly includes two parts. The first part is the inference of the maximum entropy model for the classification of the opinion target words and sentiment opinion words. And the second part is the inference of the optimized LDA model.(1) The Maximum Entropy Model. The maximum entropy model solves the classification problem actually. When the input of the model is x, the probability py∣x of the category y can be calculated by equation (16).
(16)py∣x=e∑i=1nλifix,y∑ye∑i=1nλifix,y.In equation (16), λi represents the weight vector, fix,y is the eigenfunction, and n is the number of categories. When using the maximum entropy model for the classification of the opinion target words and sentiment opinion words, it is necessary to select appropriate features. The MaxEnt-LDA [22] model uses two features, i.e., word order and part of speech. The part of speech tagging relies on some other tools that is different with different languages. In order to avoid using tools that rely on languages, this chapter chooses the word order as a feature. Section 2.2.4 gives the method of automatically annotating the training set.By training the maximum entropy model, the weightλu of the feature set fd,n can be obtained, and πd,nu can be obtained by equation (17), where d represents the d-th sentence, n is the n-th word in the sentence, u∈0,1 is the label collection, 0 represents the opinion target whose type is A, and 1 represents the sentiment opinion whose type is O.
(17)pyd,n∣fd,n=πd,nu=eλu×fd,n∑i=01eλi×fd,n.(2) The Model Inference of HME-LDA. In SLDA, the parameters, α and β, are offset by calculating the similarity between the input corpus and the seed words. The fixed parameters are denoted as αbase and βbase, and the offset parameters are denoted as αd, βt,A, βt,P, and βt,N. In Chapter 2, the above parameters are offset by using the semantic similarity calculation of the WordNet and the sentiment polarity calculation of the SentiWordNet. In this section, both the Word2Vec model and the cosine distance are used to calculate the similarity between words. The training corpus of the Word2Vec model is the all content in the corpus. The vector of the current word in the Word2Vec is represented by vw, and the vector of the seed word wt whose topic is t in Word2Vec is represented by vwt. The similarity between the word w and the topic word wt can be calculated by equation (18).
(18)simw,wt=cosθ=vw⋅vwtvw×vwt.Similar to the SLDA model, the HME-LDA will set the parameterαd separately for each document (each sentence in the SLDA model) based on the similarity between the vocabulary and seed words. The biased parameters can be calculated by equation (19).
(19)αd=∑i=1Ndsimwd,i,wt∑t′T∑iNdsimwd,i,wt′×αbase.In equation (19), Nd is the number of all words in the current sentence, T is the number of topics, wd,i is the i-th word in the current sentence, and t is the seed word.Based on the similarity between the vocabulary and seed words, the SLDA model will set parameters separately for each topic. These biased parameters can be calculated by equation (20), equation (21), and equation (22).
(20)βt,A=simw,wt×βbase,(21)βt,P=simw,wt,P×βbase,(22)βt,N=simw,wt,N×βbase.Different from SLDA, HME-LDA introduces the parameterδ to control the sentiment polarity of each sentence. The parameter δd,q can be calculated by equation (23).
(23)δd,q=∑iNdsimwd,i,wt,q∑q′∈P,N∑iNdsimwd,i,wt,q′×δbase.Similar to SLDA, HME-LDA uses the method of Gibbs sampling for solving. And the variables involved in the solution process have the same meanings as those in Tables1 and 2.Then, we use equation (24) to sample the topic of each sentence.
(24)pzd,n=t∣z¬d,n,y¬d,n,v¬d,n,⋅∝nwd,nt,A+βwd,nt,A∑vVnvt,A+βvt,A×nwd,nt,P+βwd,nt,P∑vVnvt,P+βvt,P×nwd,nt,N+βwd,nt,N∑vVnvt,N+βvt,N×nd,t+αd,t.Equation (25) and equation (26) are used to sample yd,n and vd,n,
(25)pyd,n=u∣zd,n=t,⋅∝nwd,nt,u+βwd,nt,u∑vVnvt,u+βvt,u×eλu×fd,n∑u′∈A,Oeλu′×fd,n,u∈A,O,(26)pvd,n=q∣zd,n=t,⋅∝nwd,nt,q+βwd,nt,q∑vVnvt,q+βvt,q×nd,q+δd,q.In the corpus, the approximate probability of the topict in sentence d can be calculated by equation (27),
(27)θdt=nd,t+αd,tnd+∑t′Tαd,t′.Witht as the topic, the approximate probability that the word wd,n is the opinion target can be calculated by equation (28),
(28)φwd,nt,A=nwd,nt,A+βwd,nt,A∑vVnvt,A+βvt,A.Witht as the topic, the approximate probability that the word wd,n is the positive opinion word can be calculated by equation (29),
(29)φwd,nt,P=nwd,nt,P+βwd,nt,P∑vVnvt,P+βvt,P.Witht as the topic, the approximate probability that the word wd,n is the negative opinion word can be calculated by equation (30),
(30)φwd,nt,N=nwd,nt,N+βwd,nt,N∑vVnvt,N+βvt,N.


## 2.2.1. Implementation Process

The overall process of the scheme that conducts aspect-based opinion mining based on the HME-LDA and the inverted index is shown in Figure4:
(1)
This step is the same as the implementation of step 1 in Section2.1.1(2)
Data preprocessing, whose main task is to remove stop words. The text in the corpus contains many useless stop words, such as “is” and “a,” which should be removed before further processing to avoid interference with the results(3)
Automatically generate annotated data sets by hierarchical clustering and train maximum entropy models for classification of the opinion target words and sentiment opinion words(4)
Enter the data into the HME-LDA model and perform the training to get the results(5)
Process the results for better readability. The results in the topic model are probability matrixes, whose readability is poor. To solve this problem, it is necessary to find the word which corresponds to the result that has a higher probability and find its original sentence by the inverted indexFigure 4
The flow chart of the scheme that is based on the HME-LDA and inverted index.As for Step 3 above, there are a lot of word order information in the corpus; thus, when the category of a word in the corpus can be determined, the featurewi−2,wi−1,wi,wi+1,wi+2 can be extracted. In HME-LDA, seed words are divided into topic seed words of aspect categories, seed words expressing positive sentiment, and seed words expressing negative sentiment. By scanning corpus, annotated feature sets, wi−2,wi−1,wiu,wi+1,wi+2, can be obtained, where u∈A,O. When u=A, wiA is the topic seed words, and when u=O, wiO is the sentiment seed words, and u is regarded as the label. However, the number of seed words is limited, so the training set obtained by scanning corpus may be insufficient in size. Therefore, the words in the corpus are considered to be clustered, and all the words in the category of the seed words are considered to have the same category as the seed words, and the word wiu in the scanned annotated feature set wi−2,wi−1,wiu,wi+1,wi+2 is replaced with the word in the category of seed words to get the new annotated data.


## 2.2.2. MaxEnt-LDA Model

In order to realize the aspect-based opinion mining of reviews, Zhao et al. [22] proposed a MaxEnt-LDA model based on LDA. Figure 5 is the probability model diagram of MaxEnt-LDA [22].Figure 5
The probability diagram of MaxEnt-LDA.Adopting the second thought of optimizing LDA in Section2.1.2, MaxEnt-LDA further divides topics and increases the topic number by introducing another two indicator variables, yd,s,n and ud,s,n, that are similar to Zm,n. Three new topic-word distributions are generated by the Dirichlet distribution with the parameter β. Meanwhile, the original topic is further divided into categories A (aspect-term) and O (opinion word).In the MaxEnt-LDA model, the indicator variableyd,s,n is generated by the sampling of the maximum entropy model. In this model, the selected features of the maximum entropy model include the word and the part of speech, while there are three labels which are background word B, opinion word O, and the opinion target word A. The annotated training set is partially extracted from the SemEval data set and partially annotated by manual. The indicator variable yd,s,n is the same as Zm,n, both of which are sampled from the multinomial distribution generated by the Dirichlet distribution.MaxEnt-LDA increases both the type and number of classification and introduces the maximum entropy model. However, the MaxEnt-LDA model increases the dependence on the annotated data simultaneously. In order to pursue the unsupervised features of the model, there are two ways to improve the MaxEnt-LDA: one is to replace the maximum entropy model with other unsupervised classification models; the other is to use unsupervised methods to automatically label data sets to avoid the dependence on annotated data. Meanwhile, the parametersα and β can be considered for bias.


## 2.2.3. The Description of the HME-LDA Model

In the HME-LDA model, seed words are directly set as an aspect category of reviews. Therefore, it only needs to classify the opinion targets of reviews and the review opinions by introducing the maximum entropy model as a classifier. The Bernoulli distribution with the parameterπd,n, where d indicates the d-th sentence and n indicates the n-th word, is jointly determined by the weight λd,n and the eigenvector fd,n of the maximum entropy model. A beta distribution with a parameter δd will be introduced as a priori to generate a Bernoulli distribution with a parameter Ωd for the classification of positive and negative sentiment opinion words. Figure 6 is the PGM.Figure 6
The probabilistic graphical model of HME-LDA.The generation process of the HME-LDA model is similar to that of the SLDA model in Section2.1. The difference is that yd,n is determined by the maximum entropy model rather than by the WordNet and SentiWordNet. And vd,n is determined by the parameter δd rather than by the calculation of the WordNet and SentiWordNet. The generation process of the HME-LDA model can refer to Section 2.1.4.In addition, because there are no methods to calculate the sentiment polarity in HME-LDA, it is necessary to set a sentiment opinion word whose sentiment polarity is positive or negative for each comment category.


## 2.2.4. The Automatic Data Annotation Method Based on Hierarchical Clustering

The MaxEnt-LDA [22] model uses the word feature with a window whose size is 3 to extract the features from the annotated words. The selected features include the word and the word order wi−1,wi,wi+1, where wi is the current word. The selected features also include the features of grammatical rules of words POSi−1,POSi,POSi+1, where POSi indicates the part of speech of the current word (adjectives, nouns, verbs, etc.). The part-of-speech tagging requires the use of additional tools, while it is different for the accuracy of part-of-speech tagging due to different kinds of languages, and there is a possibility that the tagging tool is lacking. Therefore, the features selected in this chapter are only the words themselves.(1) The Process of Automatically Annotating Data. After identifying the feature information obtained from the training, the next step is to consider how to automatically label it. In order to extract the feature wi−2,wi−1,wi,wi+1,wi+2, the only thing that needs to do here is to determine the category of the word wi in the corpus with information of word order. The seed-word setting of HEM-LDA is explained in Section 2.2.3. In the HME-LDA, seed words can be divided into three categories: topic seed words of a review category, seed words expressing positive sentiments, and seed words expressing negative sentiments. And the opinion target words belong to the corresponding category of comments, both of which are the same kind of topic. Therefore, seed words can be divided into two categories: opinion target words and sentiment opinion words. The categories of these seed words are able to be determined. The annotated feature sets wi−2,wi−1,wiu,wi+1,wi+2 can be obtained by scanning the corpus, where u∈A,O. When u=A, wiA is the topic seed word. When u=O, wiO is the sentiment seed word, and u is the label.However, the number of seed words is limited, and the size of the training set obtained by scanning corpus may be insufficient. Therefore, we attempt to cluster the words in the corpus and treat that all words in the category of seed words have the same category as the seed words. Besides, the word in this category is used to replace the wordwiu in the scanned annotated feature set wi−2,wi−1,wiu,wi+1,wi+2, so as to obtain the new annotated data. The pseudocode is shown in Algorithm 4.Algorithm 4. The process of automatically labelling data.
1:fort in Topics do  //Process each seed word2:wordList = getWrodListFromCorpus(t) //Find the location where t appears from the corpus and get the corresponding word order3:wordCluster = getWordCluster(t) //Get all the words of the category t from the clustering results4:trainSet = new Set //Used to save labelled training samples5:for wOrder in wordList do6:for w in wordCluster do7:   replaceWord(wOrder, t, w)//Replace the word t in the word order with w8:   trainSet.add(wOrder, t.Type)//Add the label to which wOrder and t belong to the training set9:end for10:end for11:end for(2) The Selection of Clustering Method. In this section, in order to improve the domain adaptability of the model and avoid the different effects of the same value of K when using the data of different fields, and to avoid more parameter adjustments as well, we select the hierarchical clustering method which has no need for the number of clusters. The result of hierarchical clustering is shown in Figure 7.Figure 7
The results presentation of hierarchical clustering methods.In the tree structure generated by the hierarchical clustering results, the leaf nodes of the tree are words in the corpus. When looking for words of the same category, the intermediate node of the upper layer can be found by the current word. All the leaf nodes of the subtree with this intermediate node as the root node belong to the same category.


## 2.2.5. The Inference of the HEM-LDA Model

The reasoning of the HME-LDA model mainly includes two parts. The first part is the inference of the maximum entropy model for the classification of the opinion target words and sentiment opinion words. And the second part is the inference of the optimized LDA model.(1) The Maximum Entropy Model. The maximum entropy model solves the classification problem actually. When the input of the model is x, the probability py∣x of the category y can be calculated by equation (16).
(16)py∣x=e∑i=1nλifix,y∑ye∑i=1nλifix,y.In equation (16), λi represents the weight vector, fix,y is the eigenfunction, and n is the number of categories. When using the maximum entropy model for the classification of the opinion target words and sentiment opinion words, it is necessary to select appropriate features. The MaxEnt-LDA [22] model uses two features, i.e., word order and part of speech. The part of speech tagging relies on some other tools that is different with different languages. In order to avoid using tools that rely on languages, this chapter chooses the word order as a feature. Section 2.2.4 gives the method of automatically annotating the training set.By training the maximum entropy model, the weightλu of the feature set fd,n can be obtained, and πd,nu can be obtained by equation (17), where d represents the d-th sentence, n is the n-th word in the sentence, u∈0,1 is the label collection, 0 represents the opinion target whose type is A, and 1 represents the sentiment opinion whose type is O.
(17)pyd,n∣fd,n=πd,nu=eλu×fd,n∑i=01eλi×fd,n.(2) The Model Inference of HME-LDA. In SLDA, the parameters, α and β, are offset by calculating the similarity between the input corpus and the seed words. The fixed parameters are denoted as αbase and βbase, and the offset parameters are denoted as αd, βt,A, βt,P, and βt,N. In Chapter 2, the above parameters are offset by using the semantic similarity calculation of the WordNet and the sentiment polarity calculation of the SentiWordNet. In this section, both the Word2Vec model and the cosine distance are used to calculate the similarity between words. The training corpus of the Word2Vec model is the all content in the corpus. The vector of the current word in the Word2Vec is represented by vw, and the vector of the seed word wt whose topic is t in Word2Vec is represented by vwt. The similarity between the word w and the topic word wt can be calculated by equation (18).
(18)simw,wt=cosθ=vw⋅vwtvw×vwt.Similar to the SLDA model, the HME-LDA will set the parameterαd separately for each document (each sentence in the SLDA model) based on the similarity between the vocabulary and seed words. The biased parameters can be calculated by equation (19).
(19)αd=∑i=1Ndsimwd,i,wt∑t′T∑iNdsimwd,i,wt′×αbase.In equation (19), Nd is the number of all words in the current sentence, T is the number of topics, wd,i is the i-th word in the current sentence, and t is the seed word.Based on the similarity between the vocabulary and seed words, the SLDA model will set parameters separately for each topic. These biased parameters can be calculated by equation (20), equation (21), and equation (22).
(20)βt,A=simw,wt×βbase,(21)βt,P=simw,wt,P×βbase,(22)βt,N=simw,wt,N×βbase.Different from SLDA, HME-LDA introduces the parameterδ to control the sentiment polarity of each sentence. The parameter δd,q can be calculated by equation (23).
(23)δd,q=∑iNdsimwd,i,wt,q∑q′∈P,N∑iNdsimwd,i,wt,q′×δbase.Similar to SLDA, HME-LDA uses the method of Gibbs sampling for solving. And the variables involved in the solution process have the same meanings as those in Tables1 and 2.Then, we use equation (24) to sample the topic of each sentence.
(24)pzd,n=t∣z¬d,n,y¬d,n,v¬d,n,⋅∝nwd,nt,A+βwd,nt,A∑vVnvt,A+βvt,A×nwd,nt,P+βwd,nt,P∑vVnvt,P+βvt,P×nwd,nt,N+βwd,nt,N∑vVnvt,N+βvt,N×nd,t+αd,t.Equation (25) and equation (26) are used to sample yd,n and vd,n,
(25)pyd,n=u∣zd,n=t,⋅∝nwd,nt,u+βwd,nt,u∑vVnvt,u+βvt,u×eλu×fd,n∑u′∈A,Oeλu′×fd,n,u∈A,O,(26)pvd,n=q∣zd,n=t,⋅∝nwd,nt,q+βwd,nt,q∑vVnvt,q+βvt,q×nd,q+δd,q.In the corpus, the approximate probability of the topict in sentence d can be calculated by equation (27),
(27)θdt=nd,t+αd,tnd+∑t′Tαd,t′.Witht as the topic, the approximate probability that the word wd,n is the opinion target can be calculated by equation (28),
(28)φwd,nt,A=nwd,nt,A+βwd,nt,A∑vVnvt,A+βvt,A.Witht as the topic, the approximate probability that the word wd,n is the positive opinion word can be calculated by equation (29),
(29)φwd,nt,P=nwd,nt,P+βwd,nt,P∑vVnvt,P+βvt,P.Witht as the topic, the approximate probability that the word wd,n is the negative opinion word can be calculated by equation (30),
(30)φwd,nt,N=nwd,nt,N+βwd,nt,N∑vVnvt,N+βvt,N.


## 3. Results and Analysis

This chapter mainly trains the SLDA model in Section2.1 and the HME-LDA model in Section 2.2 and, respectively, uses the above two models to extract the opinion targets and opinion review words on the Restaurant English data set, and then, this chapter will verify the feasibility of the models and analyse the experimental results.


### 3.1. Experimental Data Set

The data set of the experiment is from SemEval2016ABSA and Yelp. The original data of reviews on Yelp includes restaurants and hotels, so it needs to be screened. The test data is from the task B of SemEval2016ABSA. The test part of this experiment only pays attention to the classification of aspect-based opinion words, so the original test data need to be processed.


#### 3.1.1. Original Data Set

SemEvel provides the training set and test set of the Restaurant reviews in XML format, where the size of the training set is 737 KB and the size of test set is 264 KB. The label structure of a sentence is shown in Figure8.Figure 8
The format of the SemEval data set.The content in the label text is the original sentence, the attribute target in the label opinion is the opinion target, the category is the aspect category of reviews, and the polarity describes the sentiment polarity of the target. The model input proposed in this paper is the plain text without annotated information. The task of this paper is to extract the opinion target words and opinion sentiment opinion words and judge the polarity of the sentiment word. Here, the content in the text label needs to be extracted and added to the corpus, and the content of target and category is extracted as test data. When verifying the experimental results based on the evaluation indicators, we only take the opinion target into account.The size of the original data in Yelp is 231.2 MB, which contains lots of useless fields and covers a wide range of fields, includingRestaurant, Hotel, and Wine Bar. There is a field, business_categories, which represents the realm of the review, in the Yelp data set. Therefore, the data can be filtered through the field above. The Yelp data is used to provide additional training sets and can be used to train Word2Vec and hierarchical clustering models to provide automatic annotated training sets for the maximum entropy model in HME-LDA.


#### 3.1.2. Data Preprocessing

The Restaurant data set in SemEval is in XML format, while the format of Yelp data is CSV. Therefore, it is necessary to extract the data needed for this experiment from the files with two formats and carry out unified numbering. In theRestaurant data set, all sentences are annotated with text, so the sentences can be directly obtained from the labels. In the Yelp data set, the field, business_categories, stores the category of the review, and if the review contains the restaurant word, it is a restaurant-related review. The review of Yelp contains multiple sentences, so it can be split into multiple sentences by punctuations. Finally, the two txt files are used to store the extracted results, and one line in the file is a sentence. The xml.dom package and the csv package in python are used in the process. The amount of data finally extracted is shown in Table 3.Table 3
The information of the experimental data set.
IndexSemEvalYelpThe number of reviews200085000The number of words337367066The average length of sentence1313Repetitive words are removed from the statistics of the number of words, and no repetition words are removed in the statistics of the average length of sentences. When words are extracted from the sentences, they are separated by spaces and punctuation marks. Due to the need for additional tools for word type reduction, the word type reduction is not carried out here. Therefore, when counting the number of words, both the different tenses and the singular and plural forms of the same word are taken into account, resulting in the final number of words is larger than the fact.There is additional annotated information in the SemEval corpus. The opinion target word and the aspect category of the word in theRestaurant review field can be extracted from the annotated information. The opinion label in the original xml file is extracted by python’s xml.dom package, then the attributes, target, and category are extracted from the opinion label and make statistics. The statistical results are shown in Table 4.Table 4
The information of the annotating data.
The aspect category of reviewsThe number of sentencesThe number of words before repetitionThe number of words after repetitionFood952952420Restaurant25825890Service32432457Drinks969651Ambience22822894Location22229Figure9 generated by statistics in the training corpus of SimEval shows that the review category, food, accounts for a large proportion, while the review categories, location and drinks, account for a small proportion. The review category, restaurant, which contains a lot of semantics, is not usually the object of extraction. In this chapter, we only take the review categories, food, service, and ambience into account.Figure 9
The proportion of review categories in the SemEval corpus.


#### 3.1.3. The Construction of the Experimental Data Set

The SemEval2016Restaurant review set is used in both the experimental data set and the test set. In the HME-LDA model, additional Yelp data sets are needed to train the hierarchical classification model and the Word2Vec model, and the amount of additional data provided will affect the final results as well. Therefore, the Yelp data set is divided into four training sets according to the number of sentences, which are shown in Table5.Table 5
The information of the Yelp training set.
The name of the training setsThe number of sentencesThe number of sentencesYelp2K20003809Yelp4K40005553Yelp10K100008948Yelp20K2000012490


### 3.2. The Main Evaluation Indicators

#### 3.2.1. The Evaluating Indicators of the Aspect Review Category of Sentences

Referring to the evaluation methods of Zhao et al. [22], this paper chooses accuracy P, recall R, and their harmonic value F1 as verification indicators, which are shown in equation (31), equation (32), and equation (33). In the MaxEnt-LDA [22] model, the topic of a sentence is undefined. According to the distribution of the topic words, the topics should be set to food, service, and ambience manually. Then, the sentence should be set to the corresponding topic according to the probability of the words appearing in each topic. The SLDA model proposed in this paper, as well as the HME-LDA model that has a fewer dependence on the language than SLDA, treats the seed word as the topic word and the aspect category of reviews. Therefore, the topic of a sentence can be determined directly by the distribution in the model.
(31)P=ThenumberofcorrectpredictionsThetotalnumberofprojections,(32)R=ThenumberofcorrectpredictionsThetotalcorrectnumber,(33)F1=2×P×RP+R.


#### 3.2.2. The Evaluation Indicators of the Opinion Target

In the models proposed in this paper, the distribution information of the opinion target based on the specific topic is stored inφt,A. The φt,A preserves the word probability based on the features of a topic model, and the same word may exist in different topics, so it is impossible to calculate the extracted accuracy and recall rate directly with φt,A. Referring to the scheme of Zhao et al. [22], n words with the highest probability of each topic in φt,A are extracted and are treated as representatives, and then, their accuracy is calculated. In Section 1, the annotated data set contains information about the opinion targets and the categories of reviews they belong to. From this, the n words with the highest frequency of occurrence are selected as references for each review category. If n is 5, 10, 20, respectively, the accuracy rate Pt@n can be calculated according to equation (31), where t is the topic number and n is the number of words taken. The average accuracy of extraction is expressed by Pt@n, which is calculated by equation (34).
(34)P@n=∑t=0TPt@nT.


### 3.3. The Experimental Results and Analysis

#### 3.3.1. The Setting of Experimental Parameters

In this experiment, the parameters related to the topic model are set asαbase=50/T, βbase=0.01, and δbase=T, where T is the number of topics, and T=3 in the subsequent experiment. In addition, the other one we need to set is the seed word and the cluster number of the hierarchical classification model. The seed word set is divided into two groups for comparison. One is the seed word set A {food, service, ambience}, and the other is seed word set B {chicken, staff, atmosphere}. The cluster number of the hierarchical classification model does not directly determine the number of categories. The cluster number is set to 200 according to experience.The parameter setting of the comparison model MaxEnt-LDA [22] is the same as the original text. When training maximum entropy, the features of the model can be divided into three categories: word, part of speech, and part of speech plus word. The HME-LDA model proposed in this paper only uses the feature word. In the comparative experiment, the feature of the maximum entropy of the MaxEnt-LDA [22] model is chosen as the word.


#### 3.3.2. The Influence of Seed Word Set on SLDA and HME-LDA

Figures10 and 11 show the impact of seed words on the SLDA model and the HME-LDA model when they are, respectively, set to seed word set A {food, service, ambience} and seed word set B {chicken, staff, atmosphere}. When using the seed word set A, the model is recorded as SLDA-A, HME-LDA-A. When using the seed word set B, the model is recorded as SLDA-B, HME-LDA-B. Besides, the HME-LDA model additionally uses Yelp10K as the training set of the automatic annotated method.Figure 10
The influence of seed word sets on the extraction of sentence review categories: (a) evaluation index value of review category Food; (b) evaluation index value of review category Service; (c) evaluation index value of review category Ambience.
(a)(b)(c)Figure 11
The influence of the seed word sets on P@N.It can be seen from the experimental results in Figure10 that the selection of seed word sets has a greater impact on the SLDA model, while the impact on the HME-LDA model is not significant. Under the topics of food and ambience, the accuracy, recall rate and F1 value of the SLDA-A model are higher than those of SLDA-B, and when under the topic of service, the evaluation indexes above of SLDA-A model are lower than those of the SLDA-B model. This is related to the features of WordNet and SentiWordNet. In WordNet, the similarity between words is mainly calculated by the path between words. When the seed word food is used as the initial position, all branches only need to trace up to food. When a word in a branch of food is used as the seed word, the words of the other branches need to trace back to food first, and then searched down. This way makes the number of paths increase, which reduces the similarity of the words, thus affecting the results, while WordNet mainly preserves the concept of words and fails to perform knowledge reasoning, that is, it fails to infer the relationship between waiter and service to improve the similarity between them. When the seed word is replaced by staff, the explicit conceptual noun will make it easier to find noun words such as waiter, thus improving the effect of the SLDA. The eigenvector of the maximum entropy classifier in the HME-LDA model is taken as word order information, ignoring the conceptual problem of the word itself, so the choices of seed words have little effect on the final results. The evaluation indicators of SLDA-A and HME-LDA-A models are both slightly better than MaxEnt-LDA.The results of Figure11 show that the smaller the value of n, the higher the accuracy of the opinion target extraction in each model. And all models, namely, MaxEnt-LDA, SLDA, and HME-LDA, are suitable for extracting the opinion target words with the highest correlation, which is consistent with the comment habits of people. Most people only focus on a few aspect opinion targets of the comment objects, and the information people want to get from the comments is also based on their most concerned aspect. Similar to the results in Figure 11, when the seed word set is changed from A to B, the accuracy of the SLDA model decreases a lot, while the accuracy of the HME-LDA model decreases little, which has a great relationship with the word classification method of the SLDA. Meanwhile, it can be seen that SLDA-A and HME-LDA-A are slightly higher in the accuracy of the opinion target extraction than that of the MaxEnt-LDA.


#### 3.3.3. The Influence of the Size of Training Set on HME-LDA

In the HME-LDA model, the effect of the maximum entropy model is related to the accuracy of automatically annotated data sets. The size of the training set that is used for training the hierarchical classification model will affect the accuracy of automatic data annotation. Since it is difficult to calculate the accuracy of the automatic annotated data set, the impact of the training set size on the HME-LDA model is indicated by the evaluation indexes F1 and P@5 of the final HME-LDA model. Here,F1 takes the mean value of F1 under the three topics, and P@5 indicates the mean value of the accuracy of the opinion target extraction in each review category when taking the top five values. In the experiment, the seed word is set to the seed word set A {food, service, ambience}. The experimental results are shown in Figures 12 and 13.Figure 12
TheF1 value of HME-LDA in training sets of different sizes.Figure 13
The P@5 value of HME-LDA in training sets of different sizes.The value of the MaxEnt-LDA model is taken as a reference in the above figures. The Yelp data set is not used in the MaxEnt-LDA model, so the evaluation indicators of the MaxEnt-LDA model in these figures remain unchanged. With the increase of training set size, the effect of HEM-LDA is on the rise. When the training set reaches 10K, the effect of the HME-LDA model tends to be stable and slightly better than that of the MaxEnt-LDA model. The reason why the effect of the model is related to the size of the training set is because the larger the training set is, the more effective information it contains. Meanwhile, since the number of clusters in the hierarchical classification model is constant, when the amount of data is small, the number of words in each cluster is relatively small, and the accuracy of classification is relatively low. When the amount of data increases, the accuracy of classification will be improved relatively, which will affect the final effect of the model.


## 3.1. Experimental Data Set

The data set of the experiment is from SemEval2016ABSA and Yelp. The original data of reviews on Yelp includes restaurants and hotels, so it needs to be screened. The test data is from the task B of SemEval2016ABSA. The test part of this experiment only pays attention to the classification of aspect-based opinion words, so the original test data need to be processed.


### 3.1.1. Original Data Set

SemEvel provides the training set and test set of the Restaurant reviews in XML format, where the size of the training set is 737 KB and the size of test set is 264 KB. The label structure of a sentence is shown in Figure8.Figure 8
The format of the SemEval data set.The content in the label text is the original sentence, the attribute target in the label opinion is the opinion target, the category is the aspect category of reviews, and the polarity describes the sentiment polarity of the target. The model input proposed in this paper is the plain text without annotated information. The task of this paper is to extract the opinion target words and opinion sentiment opinion words and judge the polarity of the sentiment word. Here, the content in the text label needs to be extracted and added to the corpus, and the content of target and category is extracted as test data. When verifying the experimental results based on the evaluation indicators, we only take the opinion target into account.The size of the original data in Yelp is 231.2 MB, which contains lots of useless fields and covers a wide range of fields, includingRestaurant, Hotel, and Wine Bar. There is a field, business_categories, which represents the realm of the review, in the Yelp data set. Therefore, the data can be filtered through the field above. The Yelp data is used to provide additional training sets and can be used to train Word2Vec and hierarchical clustering models to provide automatic annotated training sets for the maximum entropy model in HME-LDA.


### 3.1.2. Data Preprocessing

The Restaurant data set in SemEval is in XML format, while the format of Yelp data is CSV. Therefore, it is necessary to extract the data needed for this experiment from the files with two formats and carry out unified numbering. In theRestaurant data set, all sentences are annotated with text, so the sentences can be directly obtained from the labels. In the Yelp data set, the field, business_categories, stores the category of the review, and if the review contains the restaurant word, it is a restaurant-related review. The review of Yelp contains multiple sentences, so it can be split into multiple sentences by punctuations. Finally, the two txt files are used to store the extracted results, and one line in the file is a sentence. The xml.dom package and the csv package in python are used in the process. The amount of data finally extracted is shown in Table 3.Table 3
The information of the experimental data set.
IndexSemEvalYelpThe number of reviews200085000The number of words337367066The average length of sentence1313Repetitive words are removed from the statistics of the number of words, and no repetition words are removed in the statistics of the average length of sentences. When words are extracted from the sentences, they are separated by spaces and punctuation marks. Due to the need for additional tools for word type reduction, the word type reduction is not carried out here. Therefore, when counting the number of words, both the different tenses and the singular and plural forms of the same word are taken into account, resulting in the final number of words is larger than the fact.There is additional annotated information in the SemEval corpus. The opinion target word and the aspect category of the word in theRestaurant review field can be extracted from the annotated information. The opinion label in the original xml file is extracted by python’s xml.dom package, then the attributes, target, and category are extracted from the opinion label and make statistics. The statistical results are shown in Table 4.Table 4
The information of the annotating data.
The aspect category of reviewsThe number of sentencesThe number of words before repetitionThe number of words after repetitionFood952952420Restaurant25825890Service32432457Drinks969651Ambience22822894Location22229Figure9 generated by statistics in the training corpus of SimEval shows that the review category, food, accounts for a large proportion, while the review categories, location and drinks, account for a small proportion. The review category, restaurant, which contains a lot of semantics, is not usually the object of extraction. In this chapter, we only take the review categories, food, service, and ambience into account.Figure 9
The proportion of review categories in the SemEval corpus.


### 3.1.3. The Construction of the Experimental Data Set

The SemEval2016Restaurant review set is used in both the experimental data set and the test set. In the HME-LDA model, additional Yelp data sets are needed to train the hierarchical classification model and the Word2Vec model, and the amount of additional data provided will affect the final results as well. Therefore, the Yelp data set is divided into four training sets according to the number of sentences, which are shown in Table5.Table 5
The information of the Yelp training set.
The name of the training setsThe number of sentencesThe number of sentencesYelp2K20003809Yelp4K40005553Yelp10K100008948Yelp20K2000012490


## 3.1.1. Original Data Set

SemEvel provides the training set and test set of the Restaurant reviews in XML format, where the size of the training set is 737 KB and the size of test set is 264 KB. The label structure of a sentence is shown in Figure8.Figure 8
The format of the SemEval data set.The content in the label text is the original sentence, the attribute target in the label opinion is the opinion target, the category is the aspect category of reviews, and the polarity describes the sentiment polarity of the target. The model input proposed in this paper is the plain text without annotated information. The task of this paper is to extract the opinion target words and opinion sentiment opinion words and judge the polarity of the sentiment word. Here, the content in the text label needs to be extracted and added to the corpus, and the content of target and category is extracted as test data. When verifying the experimental results based on the evaluation indicators, we only take the opinion target into account.The size of the original data in Yelp is 231.2 MB, which contains lots of useless fields and covers a wide range of fields, includingRestaurant, Hotel, and Wine Bar. There is a field, business_categories, which represents the realm of the review, in the Yelp data set. Therefore, the data can be filtered through the field above. The Yelp data is used to provide additional training sets and can be used to train Word2Vec and hierarchical clustering models to provide automatic annotated training sets for the maximum entropy model in HME-LDA.


## 3.1.2. Data Preprocessing

The Restaurant data set in SemEval is in XML format, while the format of Yelp data is CSV. Therefore, it is necessary to extract the data needed for this experiment from the files with two formats and carry out unified numbering. In theRestaurant data set, all sentences are annotated with text, so the sentences can be directly obtained from the labels. In the Yelp data set, the field, business_categories, stores the category of the review, and if the review contains the restaurant word, it is a restaurant-related review. The review of Yelp contains multiple sentences, so it can be split into multiple sentences by punctuations. Finally, the two txt files are used to store the extracted results, and one line in the file is a sentence. The xml.dom package and the csv package in python are used in the process. The amount of data finally extracted is shown in Table 3.Table 3
The information of the experimental data set.
IndexSemEvalYelpThe number of reviews200085000The number of words337367066The average length of sentence1313Repetitive words are removed from the statistics of the number of words, and no repetition words are removed in the statistics of the average length of sentences. When words are extracted from the sentences, they are separated by spaces and punctuation marks. Due to the need for additional tools for word type reduction, the word type reduction is not carried out here. Therefore, when counting the number of words, both the different tenses and the singular and plural forms of the same word are taken into account, resulting in the final number of words is larger than the fact.There is additional annotated information in the SemEval corpus. The opinion target word and the aspect category of the word in theRestaurant review field can be extracted from the annotated information. The opinion label in the original xml file is extracted by python’s xml.dom package, then the attributes, target, and category are extracted from the opinion label and make statistics. The statistical results are shown in Table 4.Table 4
The information of the annotating data.
The aspect category of reviewsThe number of sentencesThe number of words before repetitionThe number of words after repetitionFood952952420Restaurant25825890Service32432457Drinks969651Ambience22822894Location22229Figure9 generated by statistics in the training corpus of SimEval shows that the review category, food, accounts for a large proportion, while the review categories, location and drinks, account for a small proportion. The review category, restaurant, which contains a lot of semantics, is not usually the object of extraction. In this chapter, we only take the review categories, food, service, and ambience into account.Figure 9
The proportion of review categories in the SemEval corpus.


## 3.1.3. The Construction of the Experimental Data Set

The SemEval2016Restaurant review set is used in both the experimental data set and the test set. In the HME-LDA model, additional Yelp data sets are needed to train the hierarchical classification model and the Word2Vec model, and the amount of additional data provided will affect the final results as well. Therefore, the Yelp data set is divided into four training sets according to the number of sentences, which are shown in Table5.Table 5
The information of the Yelp training set.
The name of the training setsThe number of sentencesThe number of sentencesYelp2K20003809Yelp4K40005553Yelp10K100008948Yelp20K2000012490


## 3.2. The Main Evaluation Indicators

### 3.2.1. The Evaluating Indicators of the Aspect Review Category of Sentences

Referring to the evaluation methods of Zhao et al. [22], this paper chooses accuracy P, recall R, and their harmonic value F1 as verification indicators, which are shown in equation (31), equation (32), and equation (33). In the MaxEnt-LDA [22] model, the topic of a sentence is undefined. According to the distribution of the topic words, the topics should be set to food, service, and ambience manually. Then, the sentence should be set to the corresponding topic according to the probability of the words appearing in each topic. The SLDA model proposed in this paper, as well as the HME-LDA model that has a fewer dependence on the language than SLDA, treats the seed word as the topic word and the aspect category of reviews. Therefore, the topic of a sentence can be determined directly by the distribution in the model.
(31)P=ThenumberofcorrectpredictionsThetotalnumberofprojections,(32)R=ThenumberofcorrectpredictionsThetotalcorrectnumber,(33)F1=2×P×RP+R.


### 3.2.2. The Evaluation Indicators of the Opinion Target

In the models proposed in this paper, the distribution information of the opinion target based on the specific topic is stored inφt,A. The φt,A preserves the word probability based on the features of a topic model, and the same word may exist in different topics, so it is impossible to calculate the extracted accuracy and recall rate directly with φt,A. Referring to the scheme of Zhao et al. [22], n words with the highest probability of each topic in φt,A are extracted and are treated as representatives, and then, their accuracy is calculated. In Section 1, the annotated data set contains information about the opinion targets and the categories of reviews they belong to. From this, the n words with the highest frequency of occurrence are selected as references for each review category. If n is 5, 10, 20, respectively, the accuracy rate Pt@n can be calculated according to equation (31), where t is the topic number and n is the number of words taken. The average accuracy of extraction is expressed by Pt@n, which is calculated by equation (34).
(34)P@n=∑t=0TPt@nT.


## 3.2.1. The Evaluating Indicators of the Aspect Review Category of Sentences

Referring to the evaluation methods of Zhao et al. [22], this paper chooses accuracy P, recall R, and their harmonic value F1 as verification indicators, which are shown in equation (31), equation (32), and equation (33). In the MaxEnt-LDA [22] model, the topic of a sentence is undefined. According to the distribution of the topic words, the topics should be set to food, service, and ambience manually. Then, the sentence should be set to the corresponding topic according to the probability of the words appearing in each topic. The SLDA model proposed in this paper, as well as the HME-LDA model that has a fewer dependence on the language than SLDA, treats the seed word as the topic word and the aspect category of reviews. Therefore, the topic of a sentence can be determined directly by the distribution in the model.
(31)P=ThenumberofcorrectpredictionsThetotalnumberofprojections,(32)R=ThenumberofcorrectpredictionsThetotalcorrectnumber,(33)F1=2×P×RP+R.


## 3.2.2. The Evaluation Indicators of the Opinion Target

In the models proposed in this paper, the distribution information of the opinion target based on the specific topic is stored inφt,A. The φt,A preserves the word probability based on the features of a topic model, and the same word may exist in different topics, so it is impossible to calculate the extracted accuracy and recall rate directly with φt,A. Referring to the scheme of Zhao et al. [22], n words with the highest probability of each topic in φt,A are extracted and are treated as representatives, and then, their accuracy is calculated. In Section 1, the annotated data set contains information about the opinion targets and the categories of reviews they belong to. From this, the n words with the highest frequency of occurrence are selected as references for each review category. If n is 5, 10, 20, respectively, the accuracy rate Pt@n can be calculated according to equation (31), where t is the topic number and n is the number of words taken. The average accuracy of extraction is expressed by Pt@n, which is calculated by equation (34).
(34)P@n=∑t=0TPt@nT.


## 3.3. The Experimental Results and Analysis

### 3.3.1. The Setting of Experimental Parameters

In this experiment, the parameters related to the topic model are set asαbase=50/T, βbase=0.01, and δbase=T, where T is the number of topics, and T=3 in the subsequent experiment. In addition, the other one we need to set is the seed word and the cluster number of the hierarchical classification model. The seed word set is divided into two groups for comparison. One is the seed word set A {food, service, ambience}, and the other is seed word set B {chicken, staff, atmosphere}. The cluster number of the hierarchical classification model does not directly determine the number of categories. The cluster number is set to 200 according to experience.The parameter setting of the comparison model MaxEnt-LDA [22] is the same as the original text. When training maximum entropy, the features of the model can be divided into three categories: word, part of speech, and part of speech plus word. The HME-LDA model proposed in this paper only uses the feature word. In the comparative experiment, the feature of the maximum entropy of the MaxEnt-LDA [22] model is chosen as the word.


### 3.3.2. The Influence of Seed Word Set on SLDA and HME-LDA

Figures10 and 11 show the impact of seed words on the SLDA model and the HME-LDA model when they are, respectively, set to seed word set A {food, service, ambience} and seed word set B {chicken, staff, atmosphere}. When using the seed word set A, the model is recorded as SLDA-A, HME-LDA-A. When using the seed word set B, the model is recorded as SLDA-B, HME-LDA-B. Besides, the HME-LDA model additionally uses Yelp10K as the training set of the automatic annotated method.Figure 10
The influence of seed word sets on the extraction of sentence review categories: (a) evaluation index value of review category Food; (b) evaluation index value of review category Service; (c) evaluation index value of review category Ambience.
(a)(b)(c)Figure 11
The influence of the seed word sets on P@N.It can be seen from the experimental results in Figure10 that the selection of seed word sets has a greater impact on the SLDA model, while the impact on the HME-LDA model is not significant. Under the topics of food and ambience, the accuracy, recall rate and F1 value of the SLDA-A model are higher than those of SLDA-B, and when under the topic of service, the evaluation indexes above of SLDA-A model are lower than those of the SLDA-B model. This is related to the features of WordNet and SentiWordNet. In WordNet, the similarity between words is mainly calculated by the path between words. When the seed word food is used as the initial position, all branches only need to trace up to food. When a word in a branch of food is used as the seed word, the words of the other branches need to trace back to food first, and then searched down. This way makes the number of paths increase, which reduces the similarity of the words, thus affecting the results, while WordNet mainly preserves the concept of words and fails to perform knowledge reasoning, that is, it fails to infer the relationship between waiter and service to improve the similarity between them. When the seed word is replaced by staff, the explicit conceptual noun will make it easier to find noun words such as waiter, thus improving the effect of the SLDA. The eigenvector of the maximum entropy classifier in the HME-LDA model is taken as word order information, ignoring the conceptual problem of the word itself, so the choices of seed words have little effect on the final results. The evaluation indicators of SLDA-A and HME-LDA-A models are both slightly better than MaxEnt-LDA.The results of Figure11 show that the smaller the value of n, the higher the accuracy of the opinion target extraction in each model. And all models, namely, MaxEnt-LDA, SLDA, and HME-LDA, are suitable for extracting the opinion target words with the highest correlation, which is consistent with the comment habits of people. Most people only focus on a few aspect opinion targets of the comment objects, and the information people want to get from the comments is also based on their most concerned aspect. Similar to the results in Figure 11, when the seed word set is changed from A to B, the accuracy of the SLDA model decreases a lot, while the accuracy of the HME-LDA model decreases little, which has a great relationship with the word classification method of the SLDA. Meanwhile, it can be seen that SLDA-A and HME-LDA-A are slightly higher in the accuracy of the opinion target extraction than that of the MaxEnt-LDA.


### 3.3.3. The Influence of the Size of Training Set on HME-LDA

In the HME-LDA model, the effect of the maximum entropy model is related to the accuracy of automatically annotated data sets. The size of the training set that is used for training the hierarchical classification model will affect the accuracy of automatic data annotation. Since it is difficult to calculate the accuracy of the automatic annotated data set, the impact of the training set size on the HME-LDA model is indicated by the evaluation indexes F1 and P@5 of the final HME-LDA model. Here,F1 takes the mean value of F1 under the three topics, and P@5 indicates the mean value of the accuracy of the opinion target extraction in each review category when taking the top five values. In the experiment, the seed word is set to the seed word set A {food, service, ambience}. The experimental results are shown in Figures 12 and 13.Figure 12
TheF1 value of HME-LDA in training sets of different sizes.Figure 13
The P@5 value of HME-LDA in training sets of different sizes.The value of the MaxEnt-LDA model is taken as a reference in the above figures. The Yelp data set is not used in the MaxEnt-LDA model, so the evaluation indicators of the MaxEnt-LDA model in these figures remain unchanged. With the increase of training set size, the effect of HEM-LDA is on the rise. When the training set reaches 10K, the effect of the HME-LDA model tends to be stable and slightly better than that of the MaxEnt-LDA model. The reason why the effect of the model is related to the size of the training set is because the larger the training set is, the more effective information it contains. Meanwhile, since the number of clusters in the hierarchical classification model is constant, when the amount of data is small, the number of words in each cluster is relatively small, and the accuracy of classification is relatively low. When the amount of data increases, the accuracy of classification will be improved relatively, which will affect the final effect of the model.


## 3.3.1. The Setting of Experimental Parameters

In this experiment, the parameters related to the topic model are set asαbase=50/T, βbase=0.01, and δbase=T, where T is the number of topics, and T=3 in the subsequent experiment. In addition, the other one we need to set is the seed word and the cluster number of the hierarchical classification model. The seed word set is divided into two groups for comparison. One is the seed word set A {food, service, ambience}, and the other is seed word set B {chicken, staff, atmosphere}. The cluster number of the hierarchical classification model does not directly determine the number of categories. The cluster number is set to 200 according to experience.The parameter setting of the comparison model MaxEnt-LDA [22] is the same as the original text. When training maximum entropy, the features of the model can be divided into three categories: word, part of speech, and part of speech plus word. The HME-LDA model proposed in this paper only uses the feature word. In the comparative experiment, the feature of the maximum entropy of the MaxEnt-LDA [22] model is chosen as the word.


## 3.3.2. The Influence of Seed Word Set on SLDA and HME-LDA

Figures10 and 11 show the impact of seed words on the SLDA model and the HME-LDA model when they are, respectively, set to seed word set A {food, service, ambience} and seed word set B {chicken, staff, atmosphere}. When using the seed word set A, the model is recorded as SLDA-A, HME-LDA-A. When using the seed word set B, the model is recorded as SLDA-B, HME-LDA-B. Besides, the HME-LDA model additionally uses Yelp10K as the training set of the automatic annotated method.Figure 10
The influence of seed word sets on the extraction of sentence review categories: (a) evaluation index value of review category Food; (b) evaluation index value of review category Service; (c) evaluation index value of review category Ambience.
(a)(b)(c)Figure 11
The influence of the seed word sets on P@N.It can be seen from the experimental results in Figure10 that the selection of seed word sets has a greater impact on the SLDA model, while the impact on the HME-LDA model is not significant. Under the topics of food and ambience, the accuracy, recall rate and F1 value of the SLDA-A model are higher than those of SLDA-B, and when under the topic of service, the evaluation indexes above of SLDA-A model are lower than those of the SLDA-B model. This is related to the features of WordNet and SentiWordNet. In WordNet, the similarity between words is mainly calculated by the path between words. When the seed word food is used as the initial position, all branches only need to trace up to food. When a word in a branch of food is used as the seed word, the words of the other branches need to trace back to food first, and then searched down. This way makes the number of paths increase, which reduces the similarity of the words, thus affecting the results, while WordNet mainly preserves the concept of words and fails to perform knowledge reasoning, that is, it fails to infer the relationship between waiter and service to improve the similarity between them. When the seed word is replaced by staff, the explicit conceptual noun will make it easier to find noun words such as waiter, thus improving the effect of the SLDA. The eigenvector of the maximum entropy classifier in the HME-LDA model is taken as word order information, ignoring the conceptual problem of the word itself, so the choices of seed words have little effect on the final results. The evaluation indicators of SLDA-A and HME-LDA-A models are both slightly better than MaxEnt-LDA.The results of Figure11 show that the smaller the value of n, the higher the accuracy of the opinion target extraction in each model. And all models, namely, MaxEnt-LDA, SLDA, and HME-LDA, are suitable for extracting the opinion target words with the highest correlation, which is consistent with the comment habits of people. Most people only focus on a few aspect opinion targets of the comment objects, and the information people want to get from the comments is also based on their most concerned aspect. Similar to the results in Figure 11, when the seed word set is changed from A to B, the accuracy of the SLDA model decreases a lot, while the accuracy of the HME-LDA model decreases little, which has a great relationship with the word classification method of the SLDA. Meanwhile, it can be seen that SLDA-A and HME-LDA-A are slightly higher in the accuracy of the opinion target extraction than that of the MaxEnt-LDA.


## 3.3.3. The Influence of the Size of Training Set on HME-LDA

In the HME-LDA model, the effect of the maximum entropy model is related to the accuracy of automatically annotated data sets. The size of the training set that is used for training the hierarchical classification model will affect the accuracy of automatic data annotation. Since it is difficult to calculate the accuracy of the automatic annotated data set, the impact of the training set size on the HME-LDA model is indicated by the evaluation indexes F1 and P@5 of the final HME-LDA model. Here,F1 takes the mean value of F1 under the three topics, and P@5 indicates the mean value of the accuracy of the opinion target extraction in each review category when taking the top five values. In the experiment, the seed word is set to the seed word set A {food, service, ambience}. The experimental results are shown in Figures 12 and 13.Figure 12
TheF1 value of HME-LDA in training sets of different sizes.Figure 13
The P@5 value of HME-LDA in training sets of different sizes.The value of the MaxEnt-LDA model is taken as a reference in the above figures. The Yelp data set is not used in the MaxEnt-LDA model, so the evaluation indicators of the MaxEnt-LDA model in these figures remain unchanged. With the increase of training set size, the effect of HEM-LDA is on the rise. When the training set reaches 10K, the effect of the HME-LDA model tends to be stable and slightly better than that of the MaxEnt-LDA model. The reason why the effect of the model is related to the size of the training set is because the larger the training set is, the more effective information it contains. Meanwhile, since the number of clusters in the hierarchical classification model is constant, when the amount of data is small, the number of words in each cluster is relatively small, and the accuracy of classification is relatively low. When the amount of data increases, the accuracy of classification will be improved relatively, which will affect the final effect of the model.


## 4. Conclusions

This paper mainly studies the methods to reduce the dependence of models on annotated data by focusing on the topic of aspect-based opinion mining. The unsupervised LDA topic model has good expansibility. Based on the LDA topic model, this paper introduces the two types of dictionary tools, WordNet and SentiWordNet, to propose the SLDA model. In order to further reduce the dependence on language tools, the maximum entropy model and the method of automatically annotating data are introduced to propose an optimized model HME-LDA. The experiments show that both the SLDA model and the HME-LDA model have good results on accuracy and recall rate without relying on the annotated data. Therefore, the two optimized models will give more detailed and accurate information for cryptocurrency investors in the blockchain to assist them in better decision-making.



---

*Source: 8869385-2020-11-19.xml*