A novel Bayesian estimation-based word embedding model for sentiment analysis

1 Introduction . Word embeddings provide continuous low-dimensional vector representations of words from documents ( Li et al. , 2017 ) . Aiming to capture semantic and syntactic contextual information from large datasets , the word embedding models are extensively employed to represent words in natural language processing tasks ( Levy and Goldberg , 2014 ) . For this reason , many modelling methods are proposed to generate dense representations of words ( Rath , 2017 ) . Seeing the flourish of word embeddings , Word2vec ( Mikolov et al. , 2013 ) and GloVe ( Pennington et al. , 2014 ) are considered as the edge-cutting approaches to deal with the word contexts . C & W is another most widespread method due to the progress in neural networks ( Collobert and Weston , 2008 ) . Besides , other algorithms are integrated into the existing models . For instance , Jameel and Schockaert proposes D-GloVe by combing GloVe and Dirichlet-Multinomial language modeling ( Jameel and Schockaert , 2016 ) . More recently , the contextualized word embedding models , which improve the accuracy to a large extent , are put forward . As such , the traditional approaches are concluded as pre-trained word embeddings . Whereas , the newly proposed methods , such as ELMo by Peters M. E. et al . ( Peters et al. , 2018 ) , BERT by Devlin J. et al . ( Devlin et al. , 2018 ) and XLNet by Yang Z et al . ( Yang et al. , 2019 ) , cost large amount of computing resource for training whilst obtain a better working performance in downstream tasks . In this way , the pre-trained word embeddings still hold a great promise in handling complicated natural language processing tasks . The aforementioned models are effective in dealing with semantic-oriented tasks . Likewise , in sentiment analysis , research is still ongoing to capture sufficient sentiment information while the sentiment embeddings typically depend on the sentiment polarity labels provided by labeled corpora to guide learning processes via objective functions ( Yu et al. , 2017 ) . Tang et al . propose a method for learning sentiment embeddings by regulating the C & W model , which encodes sentiment information in the continuous word representations ( Tang et al. , 2015 ) . By exploiting the prior knowledge , Li et al . incorporate the sentiment information to analyze the sentiment label of each word in target and contexts ( Li et al. , 2017 ) . Maas et al . apply a semi-supervised method to get sentiment information and carry out the maximum likelihood estimation for parameter determination ( Maas et al. , 2011 ) . Notwithstanding , the pre-trained word embeddings still have challenges in tackling sentiment analysis tasks , which are concluded as the following two aspects . On the one hand , the semantically similar word may have opposite sentiment polarities . Thus the sentiment polarity identification process has to be dedicatedly designed ( Tang et al. , 2015 ) ( Li et al. , 2017 ) ( Shi et al. , 2018 ) . On the other hand , the capturing of sentiment information from low-frequency words is most pronounced . Typically , the low-frequency words can be regarded as the derivation of entity nouns , new terms and some deformation high-frequency words , which also contain significant semantic information . Nevertheless , due to the low frequency , current models are absent of processing their sentiment . The objective of this work is to devise a sentiment word embedding model . Specifically , the issue of parameter setting is deeply studied . Methods for effectively estimating the involving parameters based on Bayesian estimation and maximum likelihood estimation are proposed . For low-frequency word analysis , the Bayesian estimation is applied to determine the co-occurrence probabilities and the sentiment probabilities . This work describes current parameter estimation approaches and the model of GloVe in Section 2 , illustrates our sentiment word embedding model in Section 3 , shows the experiments in Section 4 , and presents the research findings in Section 5 . 2 Preliminary . This section introduces the basic theory related to parameter estimating algorithms and the GloVe model , so as to facilitate the description of subsequent model architecture . 2.1 Parameter Estimating Algorithms . Typically , word vectors are taken as learning variables in the word embeddings , which results in the use of parameter estimating algorithms . The way of establishing objective function is therefore be employed . According to ( Tang et al. , 2014 ) , the objective function of the Skip-Gram model is to maximize the average log probability , which is expressed as : J = 1 T T∑ i=1 ∑ −c⩽j⩽c , j ̸=0 ln p ( wi+j | ei ) ( 1 ) where T is the number of words in the corpus and c indicates the size of window . We take ei as the embedding of target word wi and wi+j as the context of wi . The outcome p ( wi+j |ei ) is obtained via the hierarchical softmax . Similarly , the objective of GloVe refers to the maximum of likelihood probability and is defined as ( Jameel et al. , 2019 ) : J = ∏ i , j N ( lnxij ; wi · w̃j + bi + b̃j , σ2 ) ( 2 ) where N ( . ; µ , σ2 ) represents the normal distribution with the mean µ and the variance σ2 . In GloVe , the variance is determined by each word couple ( i , j ) . In addition to objective function constructing , the estimation algorithms are applied to compute other parameters within word embeddings . In ( Maas et al. , 2011 ) , the maximum posterior probability estimation identifies the parameter to weigh the semantic information ( Maas et al. , 2011 ) . In D-GloVe , Jameel and Schockaert also use the semantic information weighing parameter , whose value corresponds to the Bayesian estimating outcome ( Jameel and Schockaert , 2016 ) . 2.2 The GloVe model . Basically , the GloVe model is a word-embedding method that combines evidence from the local context and the global counts . Typically , three distinguished words are used in this model , which are Wi , Wj and Wk . Both Wi and Wj are target words while Wk is the context . Let x be the matrix representing the word-word co-occurrence counts . We define the element xik as the times for word Wk appearing in the context of Wi . Correspondingly , xi = ∑ k xik indicates the frequency of each word occurs in the context of Wi . The cooccurrence probability of Wk being the context word of Wi is given as Pik = P ( Wk|Wi ) = xik/xi ( 3 ) The parameter Pik/Pjk is taken to determine the relation of Wi to Wk and Wj to Wk . For Wk has a similar relation to Wi and Wj , i.e . both relevant or irrelevant , the ratio approaches 1 . The information in the ratio of co-occurrence probabilities is : F ( wTi w̃k − wTj w̃k ) = Pik/Pjk ( 4 ) where wϵRn refers to the target word vector and w̃ϵRn to the context vector . Commonly , GloVe extracts the semantic relation between different words by using the ratio of cooccurrence probabilities while the semantic information are identified via the maximum likelihood estimation ( Maas et al. , 2011 ) . 3 Methodology . This section depicts the architecture of the sentiment word embedding , working principle of the parameter estimating process using two different estimation algorithms . 3.1 Sentiment Word Embedding Model Architecture . In sentiment analysis tasks , the sentiment information is captured during processing . Aiming to identify the sentiment polarities of different words , a word embedding model , incorporating the sentiment information , is established . Typically , we tend to characterize the proposed model by the loss function . To compute the sentiment embeddings , we define the probability of Wi being positive as Bi and negative as 1−Bi . Assuming that Wi = good and Wj = bad , the value of Bi/Bj is larger than 1 , which indicates good is more positive than bad . In turn , the value ( 1−Bi ) / ( 1−Bj ) is less than 1 since bad shows a negative polarity . In this way , the relations of the word sentiment are expressed by the ratio of sentiment probabilities . For Bi+ ( 1−Bi ) = 1 , Bi/Bj and ( 1−Bi ) / ( 1−Bj ) make the same sense in conveying the sentiment , we take Bi/Bj to construct the sentiment relation of Wi and Wj . More details of the words ’ relation and the ratios are presented in Appendix 1 . Considering the bias vector corresponds to positive sentiment polarity , we take sϵRn to indicate the bias vector to match the size of word vector . By transforming Wi and Wj into word vectors wi and wj , the difference established upon si and sj is written as : F ( wTi si − wTj sj ) = Bi/Bj ( 5 ) Assuming that F is confirming to the homomorphisms between groups ( R , + ) and ( R > 0 , × ) , the semantic and sentiment information is combined to get : F ( ( wTi w̃k − wTj w̃k ) + ( wTi si − wTj sj ) ) =F ( wTi w̃k − wTj w̃k ) · F ( wTi si − wTj sj ) = Pik Pjk · Bi Bj ( 6 ) Due to properties of group homomorphism , eqn.6 is transformed into F ( wTi w̃k − wTj w̃k + wTi si − wTj sj ) =F ( ( wTi w̃k + w T i si ) − ( wTj w̃k + w T j sj ) ) = F ( wTi w̃k + w T i si ) F ( wTj w̃k + w T j sj ) = Pik Pjk · Bi Bj ( 7 ) in line with F ( wTi w̃k + w T i si ) = Pik ·Bi ( 8 ) According to eqn.7 , the basic objective function F is in the form of exponential , that is F ( x ) = exp ( x ) . Thus , we apply the logarithm operation to each side and have : wTi w̃k + w T i si = ln ( Pik ·Bi ) = lnPik + lnBi ( 9 ) By incorporating the sentiment information , the loss function of the word embedding model is defined as loss ( wi , w̃k , si ) = V∑ i , k=1 [ wTi w̃k + w T i si − lnPik − lnBi ] 2 ( 10 ) where V indicates the size of the vocabulary . The parameters wTi , w̃k and si are computed via gradient descent algorithms . 3.2 Incorporating Sentiment Information . As pointed out in the Introduction , current models use the maximum likelihood estimating algorithm for parameter determination . In this part , we preliminarily carry out the parameter estimation based on the maximum likelihood principle . For each target word Wi , xi times Bernoulli experiments are conducted to extract the context independently with V different outcomes in each experiment ( Djuric and Huang , 2000 ) . The occurrence number of the kth outcome and its probability are represented by xik and Pik . If the random variable Xi = ( Xi1 , Xi2 , · · · , XiV ) stands for the occurrence times of all the possibilities , i.e . Xik is the number for the kth one , the parameter Xi obeys the Multinomial distribution , i.e . Xi ∼ Multinomial ( −→xi , −→Pi ) with −→Pi = ( Pi1 , Pi2 , · · · , PiV ) and −→xi = ( xi1 , xi2 , · · · , xiV ) . Hence , a log-likelihood function is constructed : max Pi1 , Pi2 , ··· , Pik , ··· , PiV lnL ( Pi1 , Pi2 , · · · , Pik , · · · , PiV ) = max Pi1 , Pi2 , ··· , Pik , ··· , PiV ln [ ( Pi1 ) xi1 · ( Pi2 ) xi2 · · · ( Pik ) xik · · · ( PiV ) xiV ] = max Pi1 , Pi2 , ··· , Pik , ··· , PiV V∑ k=1 xik · lnPik s.t . V∑ k=1 Pik = 1 ( 11 ) According to eqn.11 , the objective function can be resolved as an optimal problem that equality constraints . Thus , the corresponding Lagrangian function is formulated as J ( Pi1 , Pi2 , · · · , PiV , λ ) = V∑ k=1 xik · logPik + λ ( 1− V∑ k=1 Pik ) ( 12 ) where we have Pik = xikλ determined by ∂J ( Pi1 , Pi2 , ··· , PiV , λ ) ∂Pik = xikPik − λ = 0 . Likewise , λ = ∑V k=1 xik = xi is calculated with respect to ∑V k=1 Pik = ∑V k=1 ( xik λ ) = ∑V k=1 xik/λ = 1 . Thus , the estimation of Pik is written as P̂ik = xik/xi ( 13 ) Notably , the obtained Pik is the same with that from GloVe according to eqn.3 , which demonstrates the feasibility for parameter estimation in our model . In this way , the outcome of parameter sentiment probability can also be computed by using the maximum likelihood estimator . As such , a maximum likelihood estimation-based sentiment word embedding , namely MLESWE , is put forward . In this case , the Bernoulli experiments are applied to pick up the sentiment polarity of the target word Wi and the outcome can be either positive or negative . Since Bi is the probability of Wi being positive , we designate the distribution of Wi obeys−→ ti = ( ti0 , ti1 ) where ti0 is the number of negative texts and ti1 indicates that of the positive ones . Thus , the total number of texts including Wi is expressed as ti = ti0 + ti1 . Support a random variable Ti = ( Ti0 , Ti1 ) denotes the times of all the possibilities of outcomes and Ti conforms to the binomial distribution , i.e . Ti ∼ Binomial ( −→ ti , −→ Bi ) where −→Bi = ( Bi , 1−Bi ) . The log-likelihood function of sentiment probabilities is delivered as : max Bi lnL ( Bi ) = max Bi ln [ ( Bi ) ti1 · ( 1−Bi ) ti0 ] = max Bi [ ti1 · lnBi + ti0 · ln ( 1−Bi ) ] ( 14 ) Similarly , B̂i = ti1ti is obtained based on ∂ ( ln L ) ∂Bi = ti1Bi − ti0 1−Bi = 0 . Combining the semantic and sentiment information , the final loss function based on maximum likelihood principle is loss ( wi , w̃k , si ) = V∑ i , k=1 [ wTi w̃k + w T i si − ln xik xi − ln ti1 ti ] 2 ( 15 )

Summary: The paper aims at extending GloVe word embedding model so that the resulting embeddings should capture sentiments (e.g. "good" is positive while "bad" is negative). The key idea is to employ an extension term to deal with the fact that some words appearing in text with sentiment information. Furthermore, to deal with the fact that many words an infrequent, besides maximum likelihood estimation, the paper proposes to use bayesian estimation. In the experiments, Stanford sentiment tree (SST) corpus is used. The word embeddings from the two models (each trained on different estimation methods) show their capability of expressing sentiments, compared with popular methods like Glove, word2vec.