Title: A Novel Bayesian Estimation-Based Word Embedding Model For Sentiment Analysis

Abstract: The word embedding models have achieved state-of-the-art results in a variety of natural language processing tasks. Whereas, current word embedding models mainly focus on the rich semantic meanings while are challenged by capturing the sentiment information. For this reason, we propose a novel sentiment word embedding model. In line with the working principle, the parameter estimating method is highlighted. On the task of semantic and sentiment embeddings, the parameters in the proposed model are determined by using both the maximum likelihood estimation and the Bayesian estimation. Experimental results show the proposed model significantly outperforms the baseline methods in sentiment analysis for low-frequency words and sentences. Besides, it is also effective in conventional semantic and sentiment analysis tasks.

1. Introduction:

Word embeddings provide continuous low-dimensional vector representations of words from documents (Li et al., 2017) . Aiming to capture semantic and syntactic contextual information from large datasets, the word embedding models are extensively employed to represent words in natural language processing tasks (Levy and Goldberg, 2014) . For this reason, many modelling methods are proposed to generate dense representations of words (Rath, 2017) .

Seeing the flourish of word embeddings, Word2vec (Mikolov et al., 2013) and GloVe (Pennington et al., 2014) are considered as the edge-cutting approaches to deal with the word contexts. C&W is another most widespread method due to the progress in neural networks (Collobert and Weston, 2008) . Besides, other algorithms are integrated into the existing models. For instance, Jameel and Schockaert proposes D-GloVe by combing GloVe and Dirichlet-Multinomial language modeling (Jameel and Schockaert, 2016) . More recently, the contextualized word embedding models, which improve the accuracy to a large extent, are put forward. As such, the traditional approaches are concluded as pre-trained word embeddings. Whereas, the newly proposed methods, such as ELMo by Peters M. E. et al. (Peters et al., 2018) , BERT by Devlin J. et al. (Devlin et al., 2018) and XLNet by Yang Z et al. (Yang et al., 2019) , cost large amount of computing resource for training whilst obtain a better working performance in downstream tasks. In this way, the pre-trained word embeddings still hold a great promise in handling complicated natural language processing tasks.

The aforementioned models are effective in dealing with semantic-oriented tasks. Likewise, in sentiment analysis, research is still ongoing to capture sufficient sentiment information while the sentiment embeddings typically depend on the sentiment polarity labels provided by labeled corpora to guide learning processes via objective functions (Yu et al., 2017) . Tang et al. propose a method for learning sentiment embeddings by regulating the C&W model, which encodes sentiment information in the continuous word representations (Tang et al., 2015) . By exploiting the prior knowledge, Li et al. incorporate the sentiment information to analyze the sentiment label of each word in target and contexts (Li et al., 2017) . Maas et al. apply a semi-supervised method to get sentiment information and carry out the maximum likelihood estimation for parameter determination (Maas et al., 2011) .

Notwithstanding, the pre-trained word embeddings still have challenges in tackling sentiment analysis tasks, which are concluded as the following two aspects. On the one hand, the semantically similar word may have opposite sentiment polarities. Thus the sentiment polarity identification process has to be dedicatedly designed (Tang et al., 2015) (Li et al., 2017) (Shi et al., 2018) . On the other hand, the capturing of sentiment information from low-frequency words is most pronounced. Typically, the low-frequency words can be regarded as the derivation of entity nouns, new terms and some deformation high-frequency words, which also contain significant semantic information. Nevertheless, due to the low frequency, current models are absent of processing their sentiment.

The objective of this work is to devise a sentiment word embedding model. Specifically, the issue of parameter setting is deeply studied. Methods for effectively estimating the involving parameters based on Bayesian estimation and maximum likelihood estimation are proposed. For low-frequency word analysis, the Bayesian estimation is applied to determine the co-occurrence probabilities and the sentiment probabilities. This work describes current parameter estimation approaches and the model of GloVe in Section 2, illustrates our sentiment word embedding model in Section 3, shows the experiments in Section 4, and presents the research findings in Section 5.

2. Preliminary:

This section introduces the basic theory related to parameter estimating algorithms and the GloVe model, so as to facilitate the description of subsequent model architecture.

2.1. Parameter Estimating Algorithms:

Typically, word vectors are taken as learning variables in the word embeddings, which results in the use of parameter estimating algorithms. The way of establishing objective function is therefore be employed. According to (Tang et al., 2014) , the objective function of the Skip-Gram model is to maximize the average log probability, which is expressed as:

EQUATION

)

where T is the number of words in the corpus and c indicates the size of window. We take e i as the embedding of target word w i and w i+j as the context of w i . The outcome p (w i+j |e i ) is obtained via the hierarchical softmax. Similarly, the objective of GloVe refers to the maximum of likelihood probability and is defined as (Jameel et al., 2019) :

EQUATION

where N .; µ, σ 2 represents the normal distribution with the mean µ and the variance σ 2 . In GloVe, the variance is determined by each word couple (i, j).

In addition to objective function constructing, the estimation algorithms are applied to compute other parameters within word embeddings. In (Maas et al., 2011) , the maximum posterior probability estimation identifies the parameter to weigh the semantic information (Maas et al., 2011) . In D-GloVe, Jameel and Schockaert also use the semantic information weighing parameter, whose value corresponds to the Bayesian estimating outcome (Jameel and Schockaert, 2016) .

2.2. The GloVe model:

Basically, the GloVe model is a word-embedding method that combines evidence from the local context and the global counts. Typically, three distinguished words are used in this model, which are W i ,W j and W k . Both W i and W j are target words while W k is the context. Let x be the matrix representing the word-word co-occurrence counts. We define the element x ik as the times for word W k appearing in the context of W i . Correspondingly, x i = k x ik indicates the frequency of each word occurs in the context of W i . The cooccurrence probability of W k being the context word of W i is given as

P ik = P (W k |W i ) = x ik /x i (3)

The parameter P ik /P jk is taken to determine the relation of W i to W k and W j to W k . For W k has a similar relation to W i and W j , i.e. both relevant or irrelevant, the ratio approaches 1. The information in the ratio of co-occurrence probabilities is:

F w T i w k -w T j w k = P ik /P jk (4)

where wϵR n refers to the target word vector and wϵR n to the context vector. Commonly, GloVe extracts the semantic relation between different words by using the ratio of cooccurrence probabilities while the semantic information are identified via the maximum likelihood estimation (Maas et al., 2011) .

3. Methodology:

This section depicts the architecture of the sentiment word embedding, working principle of the parameter estimating process using two different estimation algorithms.

3.1. Sentiment Word Embedding Model Architecture:

In sentiment analysis tasks, the sentiment information is captured during processing. Aiming to identify the sentiment polarities of different words, a word embedding model, incorporating the sentiment information, is established. Typically, we tend to characterize the proposed model by the loss function.

To compute the sentiment embeddings, we define the probability of W i being positive as B i and negative as 1-B i . Assuming that W i = good and W j = bad, the value of B i /B j is larger than 1, which indicates good is more positive than bad. In turn, the value (1

-B i ) / (1 -B j )

is less than 1 since bad shows a negative polarity. In this way, the relations of the word sentiment are expressed by the ratio of sentiment probabilities. For

B i +(1-B i ) = 1, B i /B j and (1 -B i ) / (1 -B j

) make the same sense in conveying the sentiment, we take B i /B j to construct the sentiment relation of W i and W j . More details of the words' relation and the ratios are presented in Appendix 1.

Considering the bias vector corresponds to positive sentiment polarity, we take sϵR n to indicate the bias vector to match the size of word vector. By transforming W i and W j into word vectors w i and w j , the difference established upon s i and s j is written as:

F (w T i s i -w T j s j ) = B i /B j (5)

Assuming that F is confirming to the homomorphisms between groups (R, +) and (R >0 , ×), the semantic and sentiment information is combined to get:

F w T i w k -w T j w k + w T i s i -w T j s j =F w T i w k -w T j w k • F w T i s i -w T j s j = P ik P jk • B i B j (6)

Due to properties of group homomorphism, eqn.6 is transformed into

F w T i w k -w T j w k + w T i s i -w T j s j =F w T i w k + w T i s i -w T j w k + w T j s j = F w T i w k + w T i s i F w T j w k + w T j s j = P ik P jk • B i B j (7)

in line with

F w T i w k + w T i s i = P ik • B i (8)

According to eqn.7,the basic objective function F is in the form of exponential, that is

F (x) = exp (x)

. Thus, we apply the logarithm operation to each side and have:

w T i w k + w T i s i = ln (P ik • B i ) = ln P ik + ln B i (9)

By incorporating the sentiment information, the loss function of the word embedding model is defined as

loss (w i , w k , s i ) = V i,k=1 w T i w k + w T i s i -ln P ik -ln B i 2 (10)

where V indicates the size of the vocabulary. The parameters w T i , w k and s i are computed via gradient descent algorithms.

3.2. Incorporating Sentiment Information:

As pointed out in the Introduction, current models use the maximum likelihood estimating algorithm for parameter determination. In this part, we preliminarily carry out the parameter estimation based on the maximum likelihood principle.

For each target word W i , x i times Bernoulli experiments are conducted to extract the context independently with V different outcomes in each experiment (Djuric and Huang, 2000) . The occurrence number of the k th outcome and its probability are represented by x ik and P ik . If the random variable X i = (X i1 , X i2 , • • •, X iV ) stands for the occurrence times of all the possibilities, i.e. X ik is the number for the k th one, the parameter X i obeys the Multinomial distribution, i.e.

X i ∼ M ultinomial -→ x i , -→ P i with -→ P i = (P i1 , P i2 , • • • , P iV ) and -→ x i = (x i1 , x i2 , • • • , x iV ). Hence, a log-likelihood function is constructed:                            max Pi1,Pi2,••• ,P ik ,••• ,P iV ln L(P i1 , P i2 , • • • , P ik , • • • , P iV ) = max Pi1,Pi2,••• ,P ik ,••• ,P iV ln [(P i1 ) xi1 • (P i2 ) xi2 • • • (P ik ) x ik • • • (P iV ) x iV ] = max Pi1,Pi2,••• ,P ik ,••• ,P iV V k=1 x ik • ln P ik s.t. V k=1 P ik = 1 (11)

According to eqn.11, the objective function can be resolved as an optimal problem that equality constraints. Thus, the corresponding Lagrangian function is formulated as

J (P i1 , P i2 , • • • , P iV , λ) = V k=1 x ik • log P ik + λ 1 - V k=1 P ik (12)

where we have

P ik = x ik λ determined by ∂J(Pi1,Pi2,••• ,P iV ,λ) ∂P ik = x ik P ik -λ = 0. Likewise, λ = V k=1 x ik = x i is calculated with respect to V k=1 P ik = V k=1 x ik λ = V k=1 x ik /λ = 1.

Thus, the estimation of P ik is written as

P ik = x ik /x i (13)

Notably, the obtained P ik is the same with that from GloVe according to eqn.3, which demonstrates the feasibility for parameter estimation in our model. In this way, the outcome of parameter sentiment probability can also be computed by using the maximum likelihood estimator. As such, a maximum likelihood estimation-based sentiment word embedding, namely MLESWE, is put forward. In this case, the Bernoulli experiments are applied to pick up the sentiment polarity of the target word W i and the outcome can be either positive or negative.

Since B i is the probability of W i being positive, we designate the distribution of W i obeys -→ t i = (t i0 , t i1 ) where t i0 is the number of negative texts and t i1 indicates that of the positive ones. Thus, the total number of texts including W i is expressed as

t i = t i0 + t i1 . Support a random variable T i = (T i0 , T i1

) denotes the times of all the possibilities of outcomes and T i conforms to the binomial distribution, i.e.

T i ∼ Binomial -→ t i , -→ B i where -→ B i = (B i , 1 -B i ).

The log-likelihood function of sentiment probabilities is delivered as:

max Bi ln L (B i ) = max Bi ln (B i ) ti1 • (1 -B i ) ti0 = max Bi [t i1 • ln B i + t i0 • ln (1 -B i )] (14) Similarly, B i = ti1 ti is obtained based on ∂(ln L) ∂Bi = ti1 Bi -ti0 1-Bi = 0.

Combining the semantic and sentiment information, the final loss function based on maximum likelihood principle is

loss (w i , w k , s i ) = V i,k=1 w T i w k + w T i s i -ln x ik x i -ln t i1 t i 2 (15)

3.3. Parameter Estimating using Bayesian Estimation:

The Bayesian estimating method is highlighted due to its not sensitive to initialization via proper prior distributions to parameters (Ma et al., 2018) . By using the prior knowledge, the deficiency of lacking information of small datasets can be resolved, which leads to the converge to the actual value (Phoong and Ismail, 2015) . Accordingly, the generalization ability of the model can be improved (Wu et al., 2018) . The Bayesian approach, in this way, is able to present an elegant solution for automatically determining the parameters (Ferguson, 1973) . We thus employ the Bayesian estimation for the parameter estimating of the proposed model. The Bayesian estimation-based sentiment word embedding, namely BESWE, is performed.

In accordance to the assumption of maximum likelihood principle mentioned before, the prior distribution P -→ P i is assumed to obey the Dirichlet distribution of

-→ α = (α i , α i , • • • , α V ).

The prior distribution is converted to P -→

P i = Dir ( -→ α ) = Γ( ∑ k α k ) ∏ k Γ(α k ) k P α k -1 ik

, with the identical likelihood function:

P -→ x i | -→ P i = M ult -→ x i , -→ P i = x i ! V k (x ik !) • V k P x ik ik (16)

Considering the Dirichlet-Multinomial conjugate structure, the posterior distribution is

EQUATION

)

where

α k = λ 1 • n k ∑ k n k , n k

is the total number of occurrences of word W k in the corpus and λ 1 > 0 is determined by tuning data. By satisfying

EQUATION

we compute the Bayesian estimating outcome of ln P ik in the loss function provided by eqn.10, which is also the mean value of posterior probability in line with P ik . As stated in (Jameel and Schockaert, 2016) , the computation of E P

( - → P i| - → x i ) [ln P ik ] is facilitated via

Taylor expansion:

EQUATION

where we have V ar P

( - → P i| - → x i ) [P ik ] = α k +x ik ∑ k (α k +x ik ) • 1 -α k +x ik ∑ k (α k +x ik ) • 1 ∑ k (α k +x ik )+1 and E P ( - → P i| - → x i ) [P ik ] = α k +x ik ∑ k (α k +x ik )

. Note that ln P ik is estimated via Bayesian principle in eqn.18 whose form is unlike that of eqn.13. Comparing to the maximum likelihood estimation, a direct outcome is obtained without using Laplace smoothing in experiment. Comparatively,

P:

-→ B i is designed to obey Beta distribution with the parameter -→ β = (β 0 , β 1 ), along with the prior distribution given as

P -→ B i = Beta -→ β = Γ (β 0 + β 1 ) Γ (β 0 ) Γ (β 1 ) (1 -B i ) β0-1 • B β1-1 i (20)

from which the log-likelihood function is

P -→ t i | -→ B i = b -→ t i , -→ B i = C ti1 ti • (1 -B i ) ti0 • B ti1 i (21)

and the posterior distribution subject to the conjugate structure of Beta-Binomial is

EQUATION

)

where m k stands for the texts of the sentiment label k, λ 2 > 0 is a parameter depending on tuning data and

β k = λ 2 • m k ∑ k m k .

Thereupon, to determine the ln B i in eqn.10, we take the Bayesian estimation approach. The solution to the posterior probability expectation of ln B i , which is involved with B i is characterized as

EQUATION

Furthermore, the Taylor expansion is employed to update the equation:

EQUATION

)

where we have V ar P

( - → B i| - → t i ) [B i ] = β1+ti1 ∑ k (β k +t ik ) • 1 -β1+ti1 ∑ k (β k +t ik ) • 1 ∑ k (β k +t ik )+1 and E P ( - → B i| - → t i ) [B i ] = β1+ti1 ∑ k (β k +t ik )

. Hence, the final loss function of BESWE can be obtained:

EQUATION

4 Experiments

In this section, the working performance of BESWE and MLESWE are evaluated. The task of word similarity analysis is carried out. To deliver the sentiment embeddings, both wordand sentence-level sentiment analysis using different models is conducted.

4.1. Experiment Settings:

Datasets. The dataset SST (Stanford Sentiment Tree) is employed for the mode training.

There are five classes annotations within SST, which are very negative, negative, neutral, positive and very positive. Typically, we assign the value 3 and 4 to represent the positive polarity, 0 and 1 to negative and 2 to else. For our models, the word representation dimension is 50, the learning rate is 0.05 and the iteration number is 50. Besides, the loss function is optimized with the deployment of AdaGrad.

Baseline Methods. We evaluate the proposed model in comparison to other state-of-theart models. The models of word embeddings, such as C&W, word2vec and GloVe, together with models of sentiment embeddings, including SE-HyRank and DLJT2, are implemented.

For the baseline methods, we use default settings in the provided implementations or described as their papers and the word representation dimension is 50.

Word Similarity. Computing word similarity (Levy et al., 2015) aims to capture the general meanings of words. In this research, the word similarity tasks are conducted on the dataset EN-WS-353-ALL, EN-WS-353-SIM and SCWS, which are detailed illustrated in (Jameel et al., 2019) .

Word-level Sentiment Analysis. We conduct word-level sentiment analysis on two sentiment lexicons, namely MPQA and NRC. The number of positive and negative items for MPQA is 2301 and 4151 while for NRC is 2231 and 3324. The N-fold cross validation with N=5 and N=10 is performed. An SVM classifier is trained whose average accuracy is the evaluation metric. Specifically, the words from SST corpus are extracted and converted into word embeddings, which are taken as the features of SVM.

As Bayesian estimating principle is capable of tackling low-frequency words, we distinctively pick up the words with a frequency less than 5 for analysis. Statistically, the SST corpus contains 9984 low-frequency words.

Sentence-level Sentiment Analysis. Considering the sentiment analysis for sentence, the movie review polarity datasets MovieReview is employed (Pang and Lee, 2005) , which contains 10622 samples with the proportion of each polarity 1:1. We use a convolutional neural network (CNN) model, namely Text-CNN, with its online implementation (Kim, 2014) . Likewise, the inputs of Text-CNN are word embeddings as well. The training episode is set as 200 epochs using the default settings.

Similarly, we also pick the low-frequency words with the occupation over 10% as the lowfrequency sentences for testing. There are totally 1258 sentences cater to the demands and are all sent to Text-CNN classifier for processing.

4.2. Experimental Results:

Word Similarity. On the task of working performance evaluation, we first present the results for of word similarity analysis (Fig. 1 ). It can be observed BESWE outperforms other algorithms on all datasets, indicating that our model is capable to capture sufficient semantic information. Distinctively, the implementation of MLESWE, although not as good as BESWE, still achieves a better result on the average accuracy (i.e. Ave_Acc in Fig. 1 ) than the original GloVe. Yet the maximum likelihood estimating algorithm can also be applied to parameter determination of the word embeddings. Sentence-level Sentiment Analysis Results. The working performance of the proposed model is further evaluated on the sentence-level sentiment analysis task. From Fig. 3 , we see that SE-HyRank has a better performance than any other algorithms in average accuracy.

Clearly, the outcome of BESWE is anyhow decent which is comparable with that of SE-HyRank. Regarding low-frequency sentences, the minimum performance gap of over 9% against SE-HyRank is reported. Consequently, for the sentiment analysis of low-frequency words or low-frequency sentences, BESWE always obtain the best and most consistent results in the identification of sentiment polarity. The sentence-level sentiment analysis reaches the highest accuracy 71.64% at the point λ 1 = 0.05 and λ 2 = 0.02. For the analysis of low-frequency sentences, the optimal values of λ 1 and λ 2 are 0.1 and 0.01, which results in an accuracy of 79.92%.

The experimental results verify the effectiveness of the proposed sentiment word embedding. The BESWE model outperforms other state-of-the-art in word similarity identification. In the sentiment analysis of both word level and sentence level, our method still presents comparable outcomes. Specifically, by integrating the prior knowledge into sentiment probabilities estimating, the BESWE model is a better alternative for low-frequency-word sentiment capturing. It is reasonable to expect better performance in sentiment analysis tasks, as it is the case.

5. Conclusion:

In this work, the designing and deploying of the sentiment word embeddings is deeply studied. On the foundation of current word embedding models, the estimation principle of the objective function, together with other parameters, are investigated. Motivated by the significance of sentiment information, a novel word embedding model for sentiment analysis is established.

Within the proposed model, both semantic and sentiment information is integrated into the word vectors. Aiming to construct the objective function, the group homomorphism theory is applied. As for the parameter determination, the maximum likelihood estimator and the Bayesian estimator are employed. Experiments are conducted on various tasks to evaluate the working performance. In comparison to the baseline models, our model is capable of tackling word similarity tasks. For the purpose of sentiment embeddings representation, the proposed model is effective in word-level and sentence-level sentiment analysis. Specifically, it outperforms other methods on low-frequency words and sentences sentiment polarity identification to demonstrate its efficacy.

A Appendix I:

For W i = good and W j = bad, we have B i ⁄B j > 1,

(1 -B i ) / (1 -B j ) < 1, i.e. W i = good is more positive than W j = bad.

For W i = good and W j = great, we have B i /B j ≈ 1,

(1 -B i ) / (1 -B j ) ≈ 1, i.e. W i = good and W j = great are of positive polarities.

For W i = then and W j = home,

we have B i /B j ≈ 1,

(1 -B i ) / (1 -B j ) ≈ 1, i.e. W i = then and W j = home are of neutral polarities.

The specific sentiment probabilities calculated by maximum likelihood estimation are presented in Table 1 .

Table 1 : Sentiment probabilities of different words using maximum likelihood estimation Probability and Ratio W i = "good" W j = "bad" W i = "good" W j = "great" 

W i = "then" W j = "

:



Figures and Tables:

Figure 1: Word similarity results in Spearman rank correlation Word-level Sentiment Analysis Results. The word-level sentiment analysis task is conducted on the dataset of single-word entries. The DLJT2 model outperforms other word embedding models by incorporating sentiment information into the learning processes, as shown in Fig.2. Compared to the state-of-the-arts, our model fails to exceed the outcome of the best method on average accuracy. Encouragingly, the BESWE model shows an even better performance in tackling the low-frequency words.

Figure 2: Word-level sentiment analysis results

Figure 3: Sentence-level sentiment analysis resultsEffects of λ 1 and λ 2 . The hyperparameters in BESWE, i.e. regulatory factors λ 1 and λ 2 , are used to represent the semantic and the sentiment information. The settings of the involving parameter can be therefore determined. The values of λ 1 and λ 2 are varied within the collection of {1, 0.75, 0.5, 0.25, 0.1, 0.05, 0.02, 0.01}. Firstly, the value of λ 1 is set as {1, 0.75, 0.5, 0.25, 0.1, 0.05, 0.02, 0.01}. When λ 2 = 1, we name BESWE as BESWE#1 and λ 2 = 0.75 as BESWE#2, and so on so forth. Correspondingly, the value of λ 2 is also picked from the same set and named from BESWE#9 to BESWE#16 in the same order. Totally, we get 16 different models.The results on the sentence-level sentiment analysis against different hyperparameter settings are shown in Fig.4(a) and Fig.4(c). Likewise, the results for low-frequency sentences are in Fig.4(b) and Fig.4(d). We take LowFreSentence#n to nominate the outcomes from low-frequency sentences.

Figure 4: Sensitivity of λ 1 and λ 2 on BESWE with Sentence-level Sentiment Analysis

home" Likewise, the outcomes based on Bayesian estimation are in Table2. Sentiment probabilities of different words using Bayesian estimation

Table content: <table><tr><td>B i</td><td>0.5879</td><td>0.5879</td><td>0.3855</td></tr><tr><td>B j</td><td>0.0905</td><td>0.7063</td><td>0.4909</td></tr><tr><td>B i /B j</td><td>6.4944</td><td>0.8323</td><td>0.7854</td></tr><tr><td>(1 -B i ) / (1 -B j )</td><td>0.1540</td><td>1.2015</td><td>1.2733</td></tr><tr><td>Probability and Ratio</td><td>W i = "good" W j = "bad"</td><td>W i = "good" W j = "great"</td><td>W i = "then" W j = "home"</td></tr><tr><td>B i</td><td>0.5872</td><td>0.5872</td><td>0.3819</td></tr><tr><td>B j</td><td>0.0886</td><td>0.7053</td><td>0.4864</td></tr><tr><td>B i /B j</td><td>6.6272</td><td>0.8326</td><td>0.7852</td></tr><tr><td>(1 -B i ) / (1 -B j )</td><td>0.1509</td><td>1.2011</td><td>1.2735</td></tr></table>

Peer review (iclr2020):
The paper proposed a word embedding model to incorporate the sentiment information. The paper provided both maximum likelihood estimation and maximum posterior estimation for the proposed framework. Improved experiment results on word similarity and low frequency embeddings are presented. Overall, the paper incorporates the sentiment information in a neat way. And my main concern is the around the Bayesian inference and the prior knowledge distilled into the model. Detail comments are as following, 1. The model employed Laplace approximation for posterior distribution. Not quite sure this is a good idea for the Bernoulli case since Laplace approximation is trying to use Gaussian distribution to approximate the region around the mode. How will the MAP solution compare with a full Bayesian solution such as VB or sampling-based methods? 2. Another concern is the prior introduced into the model. Normally prior information will be washed away as the training data grow. Not the case for the low frequency examples that the model performed well on. Would it possible that the improved performance on low frequency example is just a side effect of the biased introduced by the prior? How sensitive will the embedding perform with respect to the prior selected?