A novel Bayesian estimation-based word embedding model for sentiment analysis
Authors: Jingyao Tang
Yun Xue
Ziwen Wang
Haoliang Zhao
Venue:  (2020)
Keywords: sentiment analysis
sentiment word embeddings
maximum likelihood estimation
Bayesian estimation

Abstract: The word embedding models have achieved state-of-the-art results in a variety of natural language processing tasks. Whereas, current word embedding models mainly focus on the rich semantic meanings while are challenged by capturing the sentiment information. For this reason, we propose a novel sentiment word embedding model. In line with the working principle, the parameter estimating method is highlighted. On the task of semantic and sentiment embeddings, the parameters in the proposed model are determined by using both the maximum likelihood estimation and the Bayesian estimation. Experimental results show the proposed model significantly outperforms the baseline methods in sentiment analysis for low-frequency words and sentences. Besides, it is also effective in conventional semantic and sentiment analysis tasks.

Under review as a conference paper at ICLR 2020
A NOVEL BAYESIAN ESTIMATION-BASED WORD EMBEDDING MODEL FOR SENTIMENT ANALYSIS
Anonymous authors Paper under double-blind review
ABSTRACT
The word embedding models have achieved seate-of-the-art results in vari- ety of natural language processing tasks. Whereas, current word embedding models mainly focus on the rich semantic meanings while are challenged by capturing the sentiment information. For this reason, we propose novel sentiment word embedding model. In line with the working principle, the parameter estimating method is highlighted. On the task of semantic and sentiment embeddings, the parameters in the proposed model are deter- mined by using both the maximum likelihood estimation and the Bayesian estimation. Experimental results show the proposed model significantly perforrrs the baseline moth in sentiment analysis for low-frequency words and sentences. Besides, it also effective in conventional semantic and sentiment analysis tasks.
INTRODUCTION
Word embeddings provide continuous low-dimensional vector representations words from documents (Li et al., 2017). Aiming to capture semantic and syntactic contextual informa- tion from large datasets, the word embedding models are areeextennsiielleeepplloo employed to represent words in natural language processing tasks (Levy and Goldberg, 2014). For this reason, many modelling methods are proposed to generate dense representations of words (Rath. 2017).
empploy
Seeing the flourish of word embeddings, Word2vec (Mikolov et al, 2013) and GloVe (Pen- nington et al. 2014) are considered as the edgo-cutting approaches deal with the word contexts C&W is another most widespread method due to the progress in neural networks (Collobert and Weston, 2008). Besides, other algorithms are integrated into the existing models. For instance, Jameel and proposss D-GloVe by combiing GloVe and Dirichlet-Multinomme language modeling (Jameel and Schockaerl, 2016). More recently, the contextualized word embedding models, which improve the accuracy to large extent, are put forward. As such, the traditional approaches are concluded as pre-trained word embeddings W Whereas, the newly proposed methods_such as ELMo by Peters M. F. et al. (Peters et al 201$)_BER* BERT by Devlin J. et al.(Devlin et al., 2018) and XLNet by Yang et al. YYang al. 2019) cost large amount of computing resource for training whilst ob- tain a better working performance in downstream tasks. In this way, the pre-trained word embeddings still hold great promise in handling complicated natural language processing tasks.
The aforementioned models are effective dealing with cemantic-orientee tasks. Likewise, in sentiment analysis, research is still ongoing to capture sufficient sentiment information while the sentiment embeddings typically depend on the sentiment polarity labels lahels provided by labeled corpora to guide learning processes via objective functions (Yu et al., 2017). Tang et al. propose a method for learning sentiment embeddings by regulating the C&WW C&-Wmodel, model, which encodes sentiment information in the continuous word representations (Tang et al, 2015). By exploiting the prior knowledge, Li et al. incorporate the.sentiment information to analyze the sentiment label of each word in target and contexts (Li et al., 2017). Maas et al. apply a semi-supervised method to get sentiment information and carry out the maximum likelihood estimation for parameter determination (Maas et al.. 2011).

---

Under review as a conference paper at ICLR 2020
Notwithstanding, the pre-trained word embeddings still have challenges in tackling senti- ment analysis tasks, which are concluded as the following two aspects. On the one hand, the semantically similar word may have opposite sentiment polarities. Thus the sentiment polarity identification process has to be dedicatedly designed (Tang et al 2015) (Li et al 2017) (Shi et alf 2018). On the other hand, the capturing of sentiment information from low-frequenc, words is most pronounced. Typically, the now-frequency, words can be re- garded the derivation of entity nouns, new terms and some deformation high-frequency words, which also contain significant semantic information. Nevertheless, duc to the low frequency, current models are absent of processing their sentiment.
The objective of this work is to devise sentiment word embedding model. Specifically, the issue of parameter setting deeply studied. Methods for effectively estimating the involving parameters based on Bayesian estimation and maximum likelihood estimation are proposed. For low-frequency word analysis, the Bayesian estimation applied to determine the co-occurrence probabilities and the sentiment probabilities.
This work describes current parameter estimation approaches and the model of GloVe in Section 2, illustrates our senliment word embedding model in Section 3, shows the experi- ments in Section 4, and presents the research findings Section
PRELIMINARY
This section introduces the basic theory related to parameter estimating algorithms and the GloVe Ve model, so as to facilitate the description of subsequent model architecture.
2.1 PARAMETER ESTIMATING ALGORITHMS
Typically, word vectors are taken as learning variables in the word embeddings, which results in the use of parameter estimating algorithms. The way establishing objective function is therefore be employed. According to (Tang et al. 2014), the objective function of the Skip-Gram model to maximize the average log probability, which expressed as:
T J
Σ
In (Wi+sled
(1)
T i=1 E
where is the number of words the corpus and c indicates the size of window. We take e; the embedding of target word w; and WHy the context of w,, The outcome p (wazlee) is obtained via the hierarchical softmax. Similarly the objective of GloVe refers to the maximum of likelihood probability and is defined as (Jameel al, 2019):
II N In (nn wy + b, + by 0²)
J=
(2)
where N (i, o²) represents the normal distribution with the mean μ and the variance o². In GloVe, the variance is determined by each word couple (i.j).
In addition to objective function constructing, the estimation algorithms are applied to compute other parameters within word embeddings. In (Maas et al., 2011), the maximum pusterior-probabilll probahility estimation identifies the parameter to weigh the semantic information posterior
(Maas al,, 2011). In D-GloVe, Jameel and Schockaert also use the semantic information seigghing-parmmeter, parameter_ value corresponds to the Bayesian estimating outcome (Jameel and Schockaert, 2016).
2.2 THE GLOVE MODEL
Basically, the GloVe model is a word-embedding method that combines evidence from the local context and the global counts. Typically, three distinguished words used in this
2

---

Under review as a conference paper at ICLR 2020
model, which are W,W,, and We. Both W; and W, are target words while Wa is the context. Let x be the matrix representing the word-word co-occureence counts. We define the element Tik the times for word We appearing in the context of W;. Correspondingly, 3; = E. Tat indicates the frequency of each word occurs in the context of W,. The co- occurrence probability of W being the context word of W; is given
(3) The parameter Pas/ Pi taken to determine the relation of W; to W; and W, to W For We has similar relation to W; and Wj, i.c. both relevant irrelevant, the ratio approaches 1. The information in the ratio of co-occurrence probabilities is:
Ps P (W,) = 22k/24
F [wf Wk w i0k) = Pik! Pjk
(4)
where welR" refers to the target word vector and welR" to the context vector. Commonly, GloVe extracts the semantic relation between different words by using the ratio of co- occurrence probabilities while the semantic information are identified via the maximum likelihood cstimation (Maas et al. 2011).
METHODOLOGG
This section depicts the architecture of the sentiment word embedding, working principle of the parameter estimating process using two different estimation algorithms.
3.1 SENTIMENT WORD EMBEDDING MODEL ARCHITECTURE
In sentiment analysis tasks, the sentiment information is captured during processing. Aim- ing to identify the sentiment polarities of different words, word embedding model, incor- porating the sentiment information, is established. Typically, we tend to characterize the proposed model by the loss function.
To compute the sentiment embeddings, we define the probability W, being positive as B and negative 1- B;. Assuning that W, good and W; bad, the value of B,/ B, is larger than 1, which indicates good is more positive than bad. In turn, the value (1 - B;) (1 B,) is less than 1 since bad shows negative polarity. In this way, the relations of the word sentiment are expressed by the ratio of sentiment probabilitics. For B (1- Bj) make the same sense in conveying the sentiment, we take B B, vo
1. B,/ Bj
and (1 B,) (1-B;) construct the sentiment relation of W, and Wj. More details of the words' relation and the ratios are presented Appendix 1.
Considering the bias vector corresponds to positive sentiment polarity, we take sel? to indicate the bias vector to match the size of word vector. By transforming W; and W, into word vectors w; and Wj, the difference established upon and sj is written as:
F(W 8; w5.s) B./B,
(5)
Assuming that F is confirming to the homomorphisms between groups (IR, +) and (R x), the semantic and sentiment information is combined to get:
F (wf Wh - wi wx) + (w s; wif s)))
=F F (w) wb - wx) F (w. 9t w/ 83)
Pa B,
Pzk B,
3

---

Under review as a conference paper at ICLR 2020
Due to properties of group homomorphism, eqn is transformed into
F (w Wk - w wk + wi 8, w 85)
F (w Wk + w s:)
(wf Üx + w 85))
(7)
F (wt wi + wp 8;)
PoR B.
F (wf we w/ 85)
Pjk. Bj
in line with
F (w) Wi wF+ss) 81) Pis
(8)
According equ 7te basic objective function in the form exponential, that F (xx) = exp (x)) Thus, we apply the logarithm operation to each side and have:
wi U6 + w s, In (Pix B,) = In P.x + In B;
(9)
By incorporating the sentiment information, the loss function of the word embedding model defined as
V
loss (w,, Wk. si)
=
Σ
i Wk + w 8: In Pix - In B.]
(10)
1,k=1
where V indicates the size of the vocabulary. The parameters wf, Wy and s, are computed via gradient descent algorithms.
3.2 INCORPORATING SENTIMENT INFORMATION
As pointed out in the Introduction, current models use the maximum likelihood estimat- ing algorithm for parameter determination. In this part, we preliminarily carry out the parameter estimation based on the maximum likelihood principle.
For each target word W;, Hú times Bernoulli experiments are conducted extract the context independently with V different outcomes in each experiment (Djuric and Huang 2000). The occurrence number the 4th outcome and its probability are represented by Ta and Pit. If the random variable X; = (Xa,, X12, ",,,..,,, ..,,. stands for the occurrence times of all the possibilities, iee. X is the number for the kth one, the parameter X, obeys
the Multinomial distribution, ie. X; Multinomial (z) P. with P. (i,,, Piv)
and b: (Ia1 Fa, .* F.). Hence, log-likelihood function is constructed:
max Pa. Pa.Par Pur... Piv
In L(P1, Pai.... Pik. .P..
In [PPa) (122)a ... (Pix) ... .......PP.. (Piv)
mar
Pu.Par P.e.
Put... P.v
mar Put... Piv k=1
In Ps
(11)
Pa.Pas Pue
11
According to 11 the objective function can resolved as an optimal problem that equality constraints. Thus, the corresponding Lagrangian function is formulated as
4

---

Under review as a conference paper at ICLR 2020
(Pa. Pia. PPv,
Σ Fik log Pak:
11-E Pit
1
Pit
(12)
km
where we have Pa ft determined by BJIP PJ.P...PP...... .Pv.dd) A Ex= Thus, the estimation of P.x. is written as
0.
Likewise,
is calculated with respect to > k= Pik
-K==1
-
EK- Likk 1.
Pix 2///24
(13)
Notably, the obtained Pa is the same with that from GloVe according eqn. which demonstrates the feasibility for parameter estimation in our model. In this way, the outcome of parameter sentiment probability can also be computed by using the maximum likelihood estimator. As such, maximum likelihood estimation-based sentiment word embedding, namely MLESWE, put forward. In this case, the Bernoulli experiments are applied to pick up the sentiment polarity of the target word W; and the outcome can be either positive or negative.
Since B; is the probability of W; being positive, designate the distribution of W; obeys T, = (l.,, 41) where 4a is the number of negative texts and la indicates that of the positive ones. Thus, the total number of texts including W; is expressed as 4 = 4.o + tn- Support a random variable T; (To, Ta) denotes the times of all the possibilities of outcomes and T; conforms to the binomial distribution, i.. T. Binomial (1, B.) where B, (B,, 1 B.).
-
The log-likelihood function of sentiment probabilities delivered as:
maxln (B,) = maa In (B.) (1 (1- -B) = max 1111 In B, + 400 In (1 B,)] (14) B.
B,
Similarly, B,
is obtained based on 801 L)
0B, B. 1-B, 0. Combining the semantic
and sentiment information, the final loss function based on maximum likelihood principle is
loss (wi, Wk, si)
w Wx + w/ st
In
Fikk I;
In
4.1 4;
(15)
3.3 PARAMETER ESTIMATING USING BAYESIAN ESTIMATION
The Bayesian estimating method is highlighted due to its not sensitive to initialization via proper prior distributions to parameters (Ma et al., 2018). By using the prior knowledge, the deficiency of lacking information of small datasets can be resolved, which leads to the converge to the actual value (Phoong and Ismai) 2015). Accordingly, the generalization ability of the model can be improved (Wu et al., 2018). The Bayesian approach, this is able to present an elegant solution for automatically determining the parameters (Ferguson, 1973). We thus employ the Bayesian estimation for the parameter estimating of the proposed model. The Bayesian estimation-based sentiment word embedding, namely BESWE, is performed.
way.
In accordance to the assumption of maximum likelihood principle mentioned before, the prior distribution P P is assumed to obey the Dirichlet distribution of रे (as, Oi, ,av).
The prior distribution is converted to P P Dir (α) r(>ian) (as) Il pa pas-1 with the
identical likelihood function:

---

Under review as a conference paper at ICLR 2020
V
(Z.(P.) Mult P....)
(16)
Il (maa!)
Considering the Dirichlet-Multinomiie conjugate structure, the posterior distribution is
P ((P...) = Dir (a + 2,) = r a&k + xxk) II I' (as I(k)
P II k
+
(17)
where 0g A1 mx is the total number of occurrences of word Wi in the corpus and A1 is determined by tuning data. By satisfying
GA E ((..
IIn Piul
(18)
compute the Bayesian estimating outcome of In Pa in the loss function provided by equ equ which also the inenn yalue of posterior probability line with Psk- As stated
in Jameel and Schockaert, 2016), the computation of E p (P...))) [In Psk] is facilitated via
Taylor expansion:
Var PP..13,) [Pak]
P( [In Pak) se In En (F..iz.) [Px]
(19)
2 E² P P (P.IZ.) [Pa)
ort (1- Ei (oxta.)))
where we have Var p PFF.Iz.) (P( [Px) whose form is unlike that of equ 13 Comparing to the maximum likelihood estimation,
[Pa]]
(ogtoa)t and
actra tan))
Note that in Pa is estimated via Bayesian principle in equ 18
direct outcome is obtained without using Laplace smoothing in experiment. Comparatively,
P B is designed to obey Beta distribution with the parameter 3 = (5o, 31), along with
the prior distribution given as
P
B:
= Beta 3
(1 00- 1-9-1 r (3)) (8))
I' (Bo + 31).
(20)
from which the log-like lieeliodl function is
P ((..B.) = (T. C/u (1- Bi) to B
(21)
and the posterior distribution subject to the conjugate st ructure of Reta-Binomial is
P B.lt.) Beta (3+7.)
[3, + to0) I (B, + tn) (1 B,) Bot B,9
1 to + 31 + ta)
(22)

---

Under review as a conference paper at ICLR 2020
where my stands for the texts of the sentiment label k, A2 0 is parameter dependdinn
mh on tuning data and 3x = X2 mm Thereupon, to determine the B, in equ.ll we take
the Bayesian estimation approach. The solution to the posterior probability expectation of In B, which involved with B, is characterized as
e: E P(B.7..) [In B,]
(23)
Furthermore, the Taylor expansion is employed to update the equation:
Var r 7..
[Bi]
E (B.I7.) [In B.] In E
p [B,]
(24)
2 E² P ?.) [B:]
where have Var p (B..lT.) [B,]
414 (oe++ti
B Butti (1-
Ex(ett)r)))
Σ. (b+t))+++ and
0tt E (....7.) [B:] = (+++)e))
Hence, the final loss function of BESWE can be obtained:
loss (w,, Wk 81)
Σ 1=
Wf Wi wt 8 Cik. ei]
(25)
EXPERIMENTS
In this section, the working performance of BESWE and MLESWE are evaluated. The task of word similarity analysis carried out. To deliver the sentiment embeddings, both word- and sentence-level sentiment analysis using different models is conducted.
4.1 EXPERIMENT SETTINGS
Datasets. The dataset SST (Stanford Sentiment Tree) is employed for the mode training. There are five classes annotations within SST, which very negative, negative, neutral, positive and very positive. Typically, we assign the value 3 and 4 to represent the positive polarity, 0 and 1 to negative and 2 to else. For our models, the word representation dimension is 50, the learning rate is 0.05 and the iteration number is 50. Besides, the loss function is optimized with the deployment. of AdaGrad.
We evaluate the proposed model in comparison to other state-of-the models. The models of word embeddings, such C&W, word2vec and GloVe, together with models of sentiment embeddings, including SE-HyRank and DLJT2, are implemented. For the bascline methods, we use dcfault scttings the provided implementations or de- scribed as their papers and the word representation dimension is 50.
Baseline Methods.
Word Similarity. Computing word similarity (Levy al, 2015) aims to capture the general meanings of words. In this research, the word similarity tasks are conducted on the dataset. EN- EN-WS-35--LL,, FN-WS-333-STM and SCWS, which are detailed illustrated in (Jameel et al. 2019).
Word-level Sentiment Analysis. We conduct word-level sentiment analysis on two sentiment lexicons, namely MPQA and NRC. The number of positive and negative items for MPQA is 2301 and 4151 while for NRC 2231 and 3324. The N-fold cross validation with N=5 and N=10 is performed. An SVM classifier is trained whose average accuracy is the evaluation metric. Specifically, the words from SST corpus are extracted and converted into word embeddings, which taken the features of SVM.

---

Under review as a conference paper at ICLR 2020
As Bayesian estimating principle is capable of tackling low-frequency words, distinctively pick up the words with a frequency less than 5 for analysis. Statistically, the SST corpus contains 9984 low-frequency words.
Sentence-level Sentiment Analysis. Considering the sentiment analysis for sentence, the movie review polarity datasets MovicReview is employed (Pang and Loc, 2005), which contains 10622 samples with the proportion of cach polarity 1:1. We use a convolutional neural network (CNN) model, namely Text-CNN, with its online implementation (Kim, 2014). Likewise, the imputs of Text-CNN are word embeddings as well. The training episode set as 200 epochs using the default settings.
Similarly, we also pick the low-frequency words with the occupation over 10% as the low- frequency sentences for testing. There are totally 1258 sentences cater the demands and are all sent to Text-CNN classifier for processing.
4.2 EXPERIMENTAL RESULTS
Word Similarity. On the task of working performance evaluation, we first present the results for of word similarity analysis (Fig. D) It can be observed BESWE outperforms other algorithms all datasets, indicating that our model is capable to capture sufficient semantic information. Distinctively, the implementation of MLESWE, although not as good as BESWE, still achieves a better result on the average accuracy (i.e. Ave Ave_Ace Ace in Fig. !]]
than the original GloVe. Yet. the maximum likelihood estimating algorithm can also be applied to parameter determination of the word embeddings.
#150R00SIOLL RISANSMNNN INEVS BAr
00.1
4525
3123
20
123
CAA
C301
02222
RUT2
VLISAE
Figure Word similarity results in Spearman rank correlation
Word-level Sentiment Analysis Results. The word-level sentiment analysis lask conducted on the dataset of single-word entries. The DLJJ2 model outperforms other word embedding models by incorporating sentiment information into the learning processes, as shown in 2. Compared to the state-of- teffthhe-rtss model fails to exceed the outcome of the best method on average accuracy. Encouragingly, the BESWE model shows an even better performance in tackling the low-frequency words.
Frecol (mpor))
KXPVI
11813N Jeporoo
(1:
113
K255
.441
IEN
-112
41:
911
[1911
114
C2F
0:4%
1072
M.DAT
1354:
08V
C278
06Ac
[L112
VUSV::
B0KC
Figure Word-level sentiment analysis results
8

---

Under review as a conference paper at ICLR 2020
Sentence-level Sentiment Analysis Results. The working performance of the proposed model is further evaluated on the sentence-level sentiment analysis task. From Fig we see that SE-HyRank has better performance than any other algorithms in average accuracy. Clearly, the outcome of BESWE anyhow decent which is comparable with that of SE- HyRank. Regarding low-frequency sentences, the minimum performance gap of over 9% against SE-HyRank is reported. Consequently, for the sentiment analysis of low-frequency words www-frequency sentences, BESWE always obtain the best and most consistent results in the identification of sentiment polarity.
Average Sccurncy
lloe-frspurcc--eerd accurscy
71,44
71.11
67:41
385584444
60
55
50
Figure Sentence-level sentiment analysis results
Effects of X and λ2. The hyperparameters in BESWE, i.e. regulatory factors λι and X2, are used to represent the semantic and the sent iment information. The settings of the involving parameter can be therefore determined. The values of λ₁ and d are varied within the collection of {1, 0.75, 0.5, 0.25, 0.1, 0.05, 0.02, 0.013. Firstly, the value of a is set as {1, 0.75, 0.5, 0.25, 0.1, 0.05, 0.02, 0.01). When X2 1, we name BESWE as BESWE#1 and A2 = 0.75 as BESWE#2, and so on so forth. Correspondingly, the value of x2 is also picked from the same set and named from BESWE#9 to BESWE#116 in the same order. Totally, get 16 different models.
The results on the sentence-level seutiment analysis against, different hyperparameter set- tings are shown in Fig, Ka) and (cc. Likewise, the results for low-frequency sentences are in Fig (b) and Fig (d).. We take LowFreSentence/#4 to nominate the outcomes from low-frequency sentences.
372
3.70
2.00
ECSATA)
8.00
366
0.30
0.43
0.2
0.58
100
0.90
0.23
1.4
0.00
0,66
3459239
34592313
30593311
32592312
32592313
MNS Inderanned
325923114
32592511
6.50
8.50
32593315
029
Figure Sensitivity of A and A2 on BESWE with Sentence-level Sentiment Analysis
The sentence-level sentiment analysis reaches the highest accuracy 71.64% at the point \ = 0.05 and X₂ = 0.02. For the analysis low-frequency sentences, the optimal values of Å1 and Xa 0.1 and 0.01, which results in accuracy of 79.92%.

---

Under review as a conference paper at ICLR 2020
The experimental results verify the effectiveness of the proposed sentiment word embedding. The BESWE model outperforms other state-of-the-ars in word similarity identification. In the sentiment analysis of both word level and sentence level, our method still presents com- parable outcomes. Specifically, by integrating the prior knowledge into sentiment probabili- ties estimating, the BESWE model is better alternative for low-frequeney-wort sentiment capturing. It is reasonable to expect better performance in sentiment analysis tasks, as it the case.
CONCLUSION
In this work, the designing and deploying of the sentiment word embeddings is deeply studicd. On the foundation of current word embedding models, the estimation principle of the objective function, together with other parameters, are investigated. Motivated by the significance of sentiment information, novel word embedding model for sentiment analysis is established.
Within the proposed model, both semantic and sentiment information integrated into the word vectors. Aiming to construct the objective function, the group homomorphism theory applied. As for the parameter determination, the maximum likelihood estimator and the Bayesian estimator employed. Experiments the working performance. In comparison to the bascline models, our model is capable of tackling word similarity tasks. For the purpose of sentiment embeddings representation, the proposed model is elfective in word-level and sentence-level sentiment analysis. Specifically, it outperforms other methods on low-frequency words and sentences sentiment polarity identification to demonstrate its efficacy.
conducted various tasks to evaluate
REFERENCES
Ronan Collobert and Jason Weston. unified architecture for natural language processing: Deep neural networks with iulti learning. In Proceedings of the 25th international conference on Machine learning, pages 160 167. ACM, 2008
Jacob Devlin, Ming- Wei Chang, Kenton Lec, and Kristina Toutanova. Bert: Pro- training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.m 04805, 2018
Petar M Djuric and Yufei Huang. Estimation of a bernoulli parameter p from imperfect trials. IEEE Signal Processing Letters, 7(6):160 163, 2000.
Thomas S Ferguson. A bayesian analysis of some nonparametric problems. The annals of statislics, pages 209 230, 1973.
Shoaib Jameel and Steven Schockaert. D-glove: A feasible least squares model for estimating word embedding densities. 2016.
Shoaib Jameel, Zihao Fu, Bei Shi, Wai Lam, and Steven Schockaert. Word embedding as maximum posteriori estimation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 6562 6569, 2019.
Yoon Kim. Convolutional neural networks for sentence classification. ar preprint arXiv: arXin:1408.50 2014.
Omer Levy and Yoav Goldberg, Dependency-based word embeddings. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 302-308, 2014.
Omer Levy, Yoav Goldberg, and Ido Dagan. Improving distributional similarity with lessons learned from word embeddings. Transactions of the Association for Computational Lin- guistics, 3:211 225, 2015.
Yang Li, Quan Pan, Tao Yang, Suhang Wang, Jiliang Tang, and Erik Cambria. Learning word representations for sentiment analysis. Cognitive Computation, 9(6):833 851, 2017.
10

---

Under review as a conference paper at ICLR 2020
Zhanyu Ma, Yuping Lai, W Bastiaan Kleijn, Yi-Zhe Song, Liang Wang, and Jun Guo. Vari- ational bayesian learning for dirichlet process mixture of inverted dirichlet distributions in non-gaussian image feature modeling. IEEE transactions neural networks and learning systems, 30(2):449-405 2018.
Andrew L Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, and Christo- pher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies- volume 1, pages 142-150. Association for Computational Linguistics, 2011.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeflrey Dean. Efficient estimation of word representations in vector space. arXiv preprint arXi::1301.378. 2013.
Bo Pang and Lillian Lec. Secing stars:
Exploiting class relationships for sentiment cate- gorization with respect to rating scales. In Proceedings of the 43rd anmual meeting on association for compulational linguistics, pages 115 124. Association for Computational Linguistics, 2005.
Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural longuage processing (EMNLP), pages 1532 1543, 2014.
Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lec, and Luke Zettlemoyer. Deep contextualized word represontations. arXiv preprint arXiv: 000655 2018.
Seuk- Yen Phoong and Mohd Tahir Ismail. A comparison between bayesian and maxi- mum likelihood estimations in estimating finite mixture model for financial data. Sains Malaysiana, 44(7):1033-1039, 2015.
Trideep Rath. Word and Relation Embedding for Sentence Representation. PhD thesis, Arizona State University, 2017.
Bei Shi, Zihao Fu, Lidong Bing, and Wai Lam. Learning domain-sensitive and sentiment- aware word embeddings. arXiv preprint ar Xrxi::1800 180.003801, 2018.
Duyu Tang, Furu Wei, Bing Qin, Ming Zhou, and Ting Liu. Building large-scale twitter- specific sentiment lexicon: A representation learning approach. In Proceedings of coling 2014, the 25th international conference on computational linguistics: Technical papers, pages 172 172-182, 2014.
Duyu Tang, Furu Wei, Bing Qin, Nan Yang, Ting Liu, and Ming Zhou. Sentiment em- beddings with applications to sentiment analysis. IEEE Transactions Knowledge and Dala Engineering, 28(2):496-50) 509, 2015.
Ançi Wu, Sebastian Nowozin, Edward Meeds, Richard E Turner, José Miguel Hernández- Lobato, and Alexander Gaunt. Deterministic variational inference for robust bayesian neural networks. 2018.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoe V Le. Xlnet: Generalized autoregressive pretraining for language understanding. arXiv preprint arxiv:1906 1906..08377 2019.
Liang- Chih Yu, Jin Wang, K Robert Lai, and Xuejic Zhang. Refining word embeddings using intensity scores for sentiment analysis. IEEE/ACM Trunsactions on Audio, Speech, and Language Processing, 26(3):671 681, 2017.
11

---

Under review as a conference paper at ICLR 2020
A APPENDIX
For W; = =good and W = bad,
have B,/B, B, 1,
/1 (1-B.)/C
-
B,) <
ie. W; good is more positive than W, bad.
For W; good and W = great,
have B:/ B3 1,
(1 B.) 11,
i.c. W; = good and W great are of positive polarities.
For W; then and W, home,
we have B:/B, B3 ~ 1,
(1 (L-B,)
Bj) 11
ie. W = then and W = home are of neutral polarities.
The specific sentiment probabilities calculated by maximum likelihood estimation are pre- sented in Table
Table 1: Sentiment probabilities of different words using maximum likelihood estimation
W; " "good"
W, "good"
W; "then"
Probability and Ratio
W, "bad"
W, = "great"
W = "home"
B
0.5879
0.5879
0.3855
Bj
0.0905
0.7063
0.4909
BB/B, (1 - (1
6.4944
0.8323
0.7854
0.1540
1.2015
1.2733
Likewise, the outcomes based on Bayesian estimation are in Table
Table 2: Sentiment probabilities of different words using Bayesian estimation
W, "good"
W, "good"
W; "then"
Probability and Ratio
W, "bad"
W, "great"
W, = "home"
B
0.5872
0.5872
0.3819
B₁
0.0886
0.7053
0.4864
BB/BB
6.6272
0.8326
0.7852
(1 - B B,)
0.1509
1.2011
1.2735
12


Reviews:
[{"review_id": "Hyly8cjatS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper aims at extending GloVe word embedding model so that the resulting embeddings should capture sentiments (e.g. \"good\" is positive while \"bad\" is negative). The key idea is to employ an extension term to deal with the fact that some words appearing in text with sentiment information. Furthermore, to deal with the fact that many words an infrequent, besides maximum likelihood estimation, the paper proposes to use bayesian estimation. In the experiments, Stanford sentiment tree (SST) corpus is used. The word embeddings from the two models (each trained on different estimation methods) show their capability of expressing sentiments, compared with popular methods like Glove, word2vec. \n\nI would accept this paper because: \n- This paper is well written, with thoughtful maths details. \n- The proposed models, although are extensions of GloVe, gives interesting (and rigorous) points of how to add sentiment information. \n- The experiments do support what the paper claims.\n\nI would reject it because of the experiments. The dataset (SST) is so small and thus is questionable about the quality of the learned word embeddings and the comparisons. I think there should be better ways, such as word embeddings are trained on massive data (like for GloVe and word2vec), then are fine-tuned on sentiment analysis dataset. Also, I was wondering whether there's a way to collect more sentiment data (like in SE-HyRank paper).\n\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "H1lBl62aYS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nThis paper proposes a method to learn word embedding by incorporating additional sentiment information. The proposed method extends from D-GloVe by adding the probability of positive sentiment to the loss function. The paper presents three experiments: word similarity, word-level sentiment analysis, and sentence-level sentiment analysis. The experiments show that the method performs comparably with other baseline methods and outperforms in the low-frequency sentence setting (i.e. sentence containing lower frequency words).\n\nI recommend rejecting this paper because (1) the writing is unclear and hard to follow, and (2) the experiment results are not convincing.\n\nFrom what I can understand in the model part, there are many clarifications needed, not to mention the writing style. I think the re-derivations of GloVe and D-GloVe are not helpful as they cloud the main contribution of the paper. The author should clearly highlight the differences between the main subjects of the experiment: MLESWE and BESWE. In addition, it is not clearly motivated why we need Dirichlet prior for the sentiment variable.\n\nWhile the claim is to learn better embeddings for rare words, the experiments show that the proposed methods have similar results to the previous work. The only gain we can observe is in the sentence-level experiments in which other factors could affect the performance. Thus, it is hard to draw a supportive conclusion.\n\nFinally, the writing quality must be improved. The paper contains a lot of unrelated and redundant texts (it could be that I could follow the paper). \n1. I do not think eq 2 is a representative of how the paper train the model, nor attempt to compare with. \n\n2. As mention earlier, in section 3.2 and 3.3, the re-derivation is not particularly helpful. I think the paper should put more emphasis on the novelty of the work. \n\n3. Plots in the experiment results are illegible. Tables should be more suitable for Figures 1, 2, and 3. \n\nI urged the authors to revise this paper and make sure it follows the formatting guideline, especially the citations. Finally, I'd recommend the authors have a professional writer (English) review the paper before submission.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "BklVBWjCFS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposed a word embedding model to incorporate the sentiment information. The paper provided both maximum likelihood estimation and maximum posterior estimation for the proposed framework. Improved experiment results on word similarity and low frequency embeddings are presented. Overall, the paper incorporates the sentiment information in a neat way. And my main concern is the around the Bayesian inference and the prior knowledge distilled into the model.  Detail comments are as following, \n\n1. The model employed Laplace approximation for posterior distribution. Not quite sure this is a good idea for the Bernoulli case since Laplace approximation is trying to use Gaussian distribution to approximate the region around the mode. How will the MAP solution compare with a full Bayesian solution such as VB or sampling-based methods?\n\n2. Another concern is the prior introduced into the model. Normally prior information will be washed away as the training data grow. Not the case for the low frequency examples that the model performed well on. Would it possible that the improved performance on low frequency example is just a side effect of the biased introduced by the prior? How sensitive will the embedding perform with respect to the prior selected?\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a method to improve word embedding by incorporating sentiment probabilities. Reviewer appreciate the interesting and simple approach and acknowledges improved results in low-frequency words.

However, reviewers find that the paper is lacking in two major aspects:
1) Writing is unclear, and thus it is difficult to understand and judge the contributions of this research.
2) Perhaps because of 1, it is not convincing that the improvements are significant and directly resulting from the modeling contributions.

I thank the authors for submitting this work to ICLR, and I hope that the reviewers' comments are helpful in improving this research for future submission.
Decision: Reject