Title: MODEL COMPARISON FOR SEMANTIC GROUPING Anonymous authors Paper under double-blind review

Abstract: We introduce a probabilistic framework for quantifying the semantic similarity between two groups of embeddings. We formulate the task of semantic similarity as a model comparison task in which we contrast a generative model which jointly models two sentences versus one that does not. We illustrate how this framework can be used for the Semantic Textual Similarity tasks using clear assumptions about how the embeddings of words are generated. We apply information criteria based model comparison to overcome the shortcomings of Bayesian model comparison, whilst still penalising model complexity. We achieve competitive results by applying the proposed framework with an appropriate choice of likelihood on the STS datasets.

1. INTRODUCTION:

The problem of Semantic Textual Similarity (STS), measuring how closely the meaning of one piece of text corresponds to that of another, has been studied in the hope of improving performance across various problems in Natural Language Processing (NLP), including information retrieval (Zheng & Callan, 2015) . Recent progress in the generation of word embeddings (Mikolov et al., 2013) has allowed the encoding of words using distributed vector representations, which capture semantic information through their location in the learned embedding space. Despite the extent to which semantic relations between words are captured in this space, it remains a challenge to adapt these individual word embeddings to express semantic similarity between word groups, like documents, sentences, and other textual formats.

Recent methods for STS rely on additive composition of word vectors (Arora et al., 2016; Blacoe & Lapata, 2012; Mitchell & Lapata, 2008; 2010) or deep learning architectures (Kiros et al., 2015) , which summarise a sentence through a single embedding. The resulting sentence vectors are then compared using cosine similarity -a choice that is not theoretically justified, stemming only from the fact that cosine similarity gives good empirical results. It is difficult for a practitioner to utilize word vectors efficiently if the underlying assumptions in the similarity measure are not well understood.

The main contribution of our work is the proposal of a framework that addresses these issues by explicitly deriving the similarity measure through a chosen generative model of embeddings, instead of empirically motivating it. Via this design process, a practitioner can encode suitable assumptions and constraints that may be favourable to the application of interest. Furthermore, this framework puts forward a new research direction that could help understand semantic similarity, in which practitioners can study suitable embedding distributions and assess how these perform.

The secondary contribution of our work is a derivation of a similarity measure that performs well in an online setting1 . Online settings are both practical and key to use-cases that involve information retrieval in dialogue systems. For example, in a chat-bot application new queries will arrive one at a time and methods such as the one proposed in Arora et al. (2016) will not perform as strongly as they do on the benchmark datasets. This is because one cannot perform the required data preprocessing on the entire query dataset, which will not be available a priori in online settings. Whilst our framework produces an online similarity metric, it remains competitive to offline methods such as Arora et al. (2016) . We achieve results comparable to Arora et al. (2016) on the STS dataset in Figure 1 : The generation of embeddings of words in a sentence through a latent variable (the semantic group). Above we visualize the semantic group space of embeddings (with D i , D j , D k representing different semantic groups).

O(nd) time -compared to the O(nd 2 ) average complexity of their method (where n is the number of words in a sentence, and d is the embedding size).

2. BACKGROUND:

The compositional nature of distributed representations demonstrated in Mikolov et al. (2013) and Pennington et al. (2014) indicate the presence of semantic groups in the representation space of word embeddings; an idea which has been further explored in Athiwaratkun & Wilson (2017) . Under this assumption, the task of semantic similarity can be formulated as the following question: "Are the two sentences (as groups of words) partitions of the same semantic group?" Figure 1 illustrates this concept. Utilising this rephrasing, this work formulates the task of semantic similarity between two arbitrary groups of objects as a model comparison problem. Taking inspiration from Ghahramani & Heller (2006) and Marshall et al. (2006) we propose the generative models for groups (e.g. sentences) D 1 , D 2 seen in Figure 2 .

The Bayes Factor for this graphical model is then formally defined as

sim(D 1 , D 2 ) = log p(D 1 , D 2 |M 1 ) p(D 1 , D 2 |M 2 ) = log p(D 1 , D 2 |M 1 ) p(D 1 |M 2 )p(D 2 |M 2 )

.

(1)

Figure 2 : On the left, M 1 assumes that both datasets are generated i.i.d. from the same parametric distribution. On the right, M 2 assumes that the datasets are generated i.i.d. from distinct parametric distributions.

To obtain the evidence p(D|M i ) the parameters of the model must be marginalised

p(D 1 , D 2 |M 1 ) = p(D 1 , D 2 |θ)p(θ)dθ = w k ∈D1⊕D2 p(w k |θ)p(θ)dθ, p(D i |M 2 ) = w k ∈Di p(w k |θ)p(θ)dθ,

where ⊕ denotes concatenation.

Computing the semantic similarity score of the two groups D 1 , D 2 under the Bayesian framework requires selecting a reasonable model likelihood p(w k |θ), prior density on the parameters p(θ), and computing the marginal evidence specified above. Computing the evidence can be computationally intensive and usually requires approximation. What is more, the Bayes factor is very sensitive to the choice of prior and can result in estimates that heavily underfit the data (especially under a vague prior, as shown in Appendix E), having the tendency to select the simpler model; which is argued further by M. S. Bartlett (1957) and Akaike et al. (1981) . This is handled in Ghahramani & Heller (2006) by using the empirical Bayes procedure, a shortcoming of which is the issue of double counting (Berger, 2000) and thus being prone to over-fitting. We address these issues by choosing to work with information criteria based model comparison as opposed to using the Bayes factor. The details are described in Section 3.

We are not aware of any prior work on sentence similarity that uses our approach. We employ a generative model for sentences similar to Arora et al. (2016) , but our contribution differs from theirs in that our similarity is based on the aforementioned model comparison test, whilst theirs is based on the inner product of sentence embeddings derived as maximum likelihood estimators. The method by Marshall et al. (2006) applies the same score as in Equation 1 in the context of merging datasets whilst we focus on information retrieval and semantic textual similarity.

3. METHODOLOGY:

We address the shortcomings of the Bayes Factor described in Section 2 by proposing model comparison criteria that minimise Kullback-Leibler (KL) divergence (denoted in equations as D KL ) across a candidate set of models. This results in a penalised likelihood ratio test which gives competitive results. This approach relies on Information Theoretic Criteria to design a log-ratio-like test that is prior free and robust to overfitting.

We seek to compare models M 1 and M 2 using Information Criteria (IC) to assess the goodness of fit of each model. There are multiple IC used for model selection, each with different settings to which they are better suited. The IC which we will be working with have the general form

IC(D, M) = -αL( θ|D, M) + Ω (D, M) , where L( θ|D, M) = n i L( θ|w i , M)

is the maximised value of the log likelihood function for model M, α is a scalar derived for each IC, and Ω (D, M) represents a model complexity penalty term which is model and IC specific. Using this general formulation for the involved information criteria yields the following similarity score:

sim(D 1 , D 2 ) = -IC({D 1 , D 2 }, M 1 ) + IC({D 1 , D 2 }, M 2 ) = α L( θ1,2 |M 1 ) -( L( θ1 |M 2 ) + L( θ2 |M 2 )) -Ω ({D 1 , D 2 }, M 1 ) + Ω ({D 1 , D 2 }, M 2 ) .

The paragraph surrounding Equation (9) of Tenenbaum & Griffiths (2001) (Tversky's contrast model) describes what a good similarity is from a cognitive science perspective. Interestingly, the semantic interpretation of the similarity we have derived is similar to the one described in that work. We want to contrast the commonalities of the two datasets (through shared parameters) to the distinctive features of each dataset (through independently fit parameters).

Examples of these criteria can be put into two broad classes. The Bayesian Information Criterion (BIC) is an example of an IC that approximates the model evidence directly, as defined in Schwarz et al. (1978) . Empirically it has been shown that the BIC is likely to underfit the data, especially when the number of samples is small (Dziak et al., 2012) . We provide additional empirical results in Appendix E, showing BIC is not a good fit for the STS task as sentences contain a relatively small number of words (samples). Thus we focus on the second class -Information Theoretic Criteria.

3.1. INFORMATION THEORETIC CRITERIA:

The Information Theoretic Criteria (ITC) are a family of model selection criteria. The task they address is evaluating the expected quality of an estimated model specified by L( θ|w) when it is used to generate unseen data from the true distribution G(w), as defined in Konishi & Kitagawa (2008b) . This family of criteria perform this evaluation by using the KL divergence between the true model G(w) and the fitted model L( θ|w), with the aim of selecting the model (from a given set of models) that minimizes the quantity

D KL G(w) p(w| θ) = E G ln G(w) p(w| θ) = -H G (w) -E G ln p(w| θ) .

The entropy of the true model H G (w) will remain constant across different likelihoods. Thus, the quantity of interest in the definition of the information criterion under consideration is given by the expected log likelihood under the true model E G [ln p(w| θ)]. The goal then, is to find a good estimator for this quantity. One such estimator is given by the normalized maximum likelihood:

E Ĝ ln p(w| θ) = 1 n n i=1 ln p(w i | θ),

where Ĝ represents the empirical distribution. This estimator introduces a bias that varies with respect to the dimension of the model's parameter vector θ and requires a correction in order to carry out a fair comparison of information criteria between models. A model specific correction is derived resulting in the following IC, called Takeuchi Information Criterion (TIC) in Takeuchi (1976)

Ĵ = - 1 n n i=1 ∇ 2 θ L(θ|w i ) θ= θ , Î, = 1 n n i=1 ∇ θ L(θ|w i )∇ θ L (θ|w i ) θ= θ , TIC(D, M) = -2 L( θ|D, M) -tr Î Ĵ -1 .

Then, under the TIC we have the following similarity (full derivation in Appendix A)

sim(D 1 , D 2 ) = 2 L( θ1,2 |M 1 ) -L( θ1 |M 2 ) -L( θ2 |M 2 ) -tr Î1,2 Ĵ -1 1,2 + tr Î1 Ĵ1 -1 + tr Î2 Ĵ2 -1

.

For the case where we assume our model has the same parametric form as the true model and as n → ∞, the equality Î = Ĵ holds resulting in a penalty of tr Î Ĵ -1 = tr(I k ) = k where k is the number of model parameters, as shown in Konishi & Kitagawa (2008a) . This results in the Akaike Information Criterion (AIC) (Akaike, 1974 )

AIC(D, M) = -2(L( θ|D, M) -k).

The AIC simplification of TIC relies on several assumptions that only hold true in the big data limit. AIC also assumes our model M has the same parametric form as the true model (Konishi & Kitagawa (2008a) ). In general, TIC is a more robust approximation. However, as shown in Appendix F, for models with a high number of parameters, TIC may prove unstable and thus AIC will generally perform better. In this study we will consider and contrast both.

4. LIKELIHOODS:

In this section we will illustrate how to derive a similarity score under our ITC framework by choosing a likelihood function that incorporates our prior assumptions about the generating process of the data. Adopting the viewpoint of a practitioner, we would like to compare the performance of two models -one that ignores word embedding magnitude, and one that makes use of it. Our modelling choices for each assumption are the von Mises-Fisher (vMF) and Gaussian likelihoods respectively. The comparison between the two likelihoods we provide in Section 5 provides empirical evidence as to which approach is better suited to modelling word embeddings. The rest of this section outlines the details of each of the two modelling choices.

4.1. VON MISES-FISHER LIKELIHOOD:

Word embeddings are d dimensional vectors of real numbers, that are traditionally learned by optimizing a dot product between target words and context vectors (Mikolov et al., 2013) . Due to this training setup, cosine similarity is often used to measure the semantic similarity of words in various information retrieval tasks. Thus, we want to explore a distribution induced by the cosine similarity measure. We model our embeddings as vectors lying on the surface of the d -1 dimensional unit hyper-sphere w ∈ S d-1 and i.i.d. according to a vMF likelihood (Fisher et al. (1993) )

p(w|µ, κ) = κ d 2 -1 (2π) d 2 I d 2 -1 (κ) exp κµ w = 1 Z(κ) exp κµ w ,

where µ is the mean direction vector and κ is the concentration, with supports ||µ|| = ||w|| = 1, κ ≥ 0. The term I ν (κ) Corresponds to a modified Bessel function of the first kind with order ν.

In this work we reparameterise the random variable to polar hypersphericals w(φ) (φ = (φ 1 , ..., φ d-1 ) ) as adopted in Mabdia (1975) . Further details can be found in Appendix B.

The first and second order partial derivatives of the vMF log likelihood are derived in Appendix C.

We prove (in Appendix C) that the mixed derivatives of the vMF log likelihood are a constant (with respect to φ) times ∂L(θ, κ|φ)/∂θ k . Thus, evaluated at the MLE, these entries are zero. Assuming

l < k ∂ 2 L(θ, κ|D) ∂κ∂θ k θ= θ,κ=κ = ∂ 2 L(θ, κ|D) ∂θ k ∂θ l θ= θ,κ=κ = 0.

Thus, Ĵ is a diagonal matrix, with diagonal j = ( Ĵ11 , ..., Ĵdd )

EQUATION

)

This quantity only requires O(nd) operations to compute and thus does not increase the asymptotic complexity of the algorithm.

The closed form of the similarity measure for two sentences D 1 , D 2 of length m and l respectively under this model is then

sim(D 1 , D 2 ) =(m + l)κ 1,2 R1,2 -mκ 1 R1 -lκ 2 R2 -(m + l) log Z(κ 1,2 ) + m log Z(κ 1 ) + l log Z(κ 2 ) -tr( Î1,2 Ĵ -1 1,2 ) + tr( Î1 Ĵ1 -1

) + tr( Î2 Ĵ2

-1

),

where the Jacobian terms (from the reparametrisation) cancel out. The subscripts indicate the sentence, with 1, 2 meaning the concatenation of the two sentences.

4.2. GAUSSIAN LIKELIHOOD:

Schakel & Wilson (2015) show that some frequency information is contained in the magnitude of word embeddings. This motivates a choice of a likelihood function that is not constrained to the unit hyper-sphere and possibly the simplest such choice is the Gaussian likelihood. Due to the small size of sentences2 we choose a diagonal covariance Gaussian models. The Gaussian likelihood is then

p (w|µ, Σ) = N (w|µ, Σ) ,

where Σ is a diagonal matrix.

As with the vMF likelihood, we prove in Appendix D that the Hessian of the log likelihood evaluated at the MLE is diagonal. The TIC correction is a sum of O(d) terms, similar to the form in Equation 2. This can be nicely written in terms of biased sample kurtosis (denoted κ)

tr( Î Ĵ -1 ) = 1 2 d + d i=1 ( μ4 ) i σ4 i = d 2 + d i=1 κi 2

The closed form of the similarity measure for two sentences D 1 , D 2 of length m and l respectively under this model is then

sim(D 1 , D 2 ) = d i=1 -(m + l) ln( σ1,2 ) i + m ln( σ1 ) i + l ln( σ2 ) i + d 2 + 1 2 d i=1 -( κ 1,2 ) i + ( κ1 ) i + ( κ2 ) i

where the subscripts indicate the sentence, with 1, 2 meaning the concatenation of the two sentences.

5. EXPERIMENTS:

We assess our methods' performance on the Semantic Textual Similarity (STS) datasets3 (Agirre et al., 2012; 2013; 2014; 2015; 2016) . The objective of these tasks is to estimate the similarity between two given sentences, validated against human scores. In our experiments, we assess on the pre-trained GloVe (Pennington et al., 2014) , FastText (Bojanowski et al., 2016) , and Word2Vec (Mikolov et al., 2013) word embeddings. For the vMF distribution, we centre the distribution of word embeddings and normalize them to be of length 1. Some of the sentences are left with a single word after querying the word embeddings, making the MLE of the κ parameter of the vMF and Σ parameter of the Gaussian ill-defined. We overcome this issue by padding each sentence with an arbitrary embedding of a word or punctuation symbol (i.e. '.' or '?').

We first compare our methods: vMF likelihood with TIC correction (vMF+TIC) and diagonal Gaussian likelihood with AIC correction (Diag+AIC) against each other. Then, the better method is compared against mean word vector (MWV), word mover's distance (WMD) (Kusner et al., 2015) 4 , smooth inverse frequency (SIF), and SIF with principal component removal as defined in Arora et al. ( 2016)5 . We re-ran these models under our experimental setup, to ensure a fair comparison. The metric used is the average Spearman correlation score over each dataset, weighted by the number of sentences. The choice of Spearman correlation is given by its non-parametric nature (assumes no distribution over the scores), as well as measuring any monotonic relationship between the two compared quantities.

5.1. EMBEDDING MAGNITUDE:

A practitioner may want to learn more about a given set of word embeddings, and the way these embeddings were trained may not allow the user to understand the importance of certain features -say embedding magnitude. This is where our framework can be used to help build intuition by comparing a likelihood that implicitly incorporates embedding magnitude to one that does not. We present the comparison between the similarities derived from the vMF and Gaussian likelihoods in Table 5 .1. We note that the Gaussian is a much better modelling choice, beating the vMF on every dataset by a margin of at least 0.05 (5%) on average, with each of the three word embeddings. This is strong evidence that the information encoded in an embedding's magnitude is useful for tasks such as semantic similarity. This further motivates the conjecture that frequency information is contained in word embedding magnitude, as explored in Schakel & Wilson (2015) .

5.2. ONLINE SCENARIO:

In Section 1, it was mentioned that in an online setting, one cannot perform the principal component removal described in Arora et al. (2016) , as that pre-processing requires access to the entire query dataset a priori. This scenario arises in the context of information retrieval, e.g. when creating a chat-bot.

The comparison against the baseline methods are presented in Table 5 .2. We are able to out-perform the standard weighting induced by MWV, as well as the WMD approach on all datasets, with each of the three word embeddings considered. We outperform SIF using the GloVe embeddings by 0.0278 (2.78%), effectively tie when using the FastText embeddings with a difference of 0.0021 (0.21%) and we are marginally below using Word2Vec by 0.0028 (0.28%). The latter two differences are small enough to be considered insignificant. augmented with the additional pre-processing technique seen in Arora et al. (2016) . We need only consider this baseline, as it outperforms all others by a large margin.

5.3. OFFLINE SCENARIO:

The results are shown in Table 5 .3. We remaing competitive with SIF+PCA on all three word embeddings, being able to match very closely on GloVe embeddings. On the FastText and Word2Vec embeddings, our method is less than 0.01 (1%) lower on average than SIF+PCA.

6. CONCLUSION:

We've presented a new approach to similarity measurement that achieves competitive performance to standard methods in both online and offline settings. Our method requires a set of clear choices -model, likelihood and information criterion. From that, a comparison framework is naturally derived, which supplies us with a statistically justified similarity measure (by utilizing ITC to reduce the resulting model-comparison bias). This framework is suitable for a variety of modelling scenarios, due to the freedom in specifying the generative process. The graphical model we employ is adaptable to encode structural dependencies beyond the i.i.d. data-generating process we have assumed throughout this study -for example, an auto-regressive (sequential) model may be assumed if the practitioner suspects that word order matters (i.e. compare "Does she want to get pregnant?" to "She does want to get pregnant.")

In this study, we conjecture that the von Mises-Fisher distribution lends itself to representing word embeddings well, if their magnitude is disregarded and a unimodal distribution over individual sentences is assumed. Relaxing the former assumption, we also model word embeddings with a Gaussian likelihood. As this improves results, this is empirical evidence that word embedding magnitude carries relevant information, which agrees with prior intuition built from Schakel & Wilson (2015) . We hope that this framework could be a stepping stone in using more complex and accurate generative models of text to assess semantic similarity. For example, relaxing the assumption of unimodality is an interesting area for future research.

A DERIVATION OF TIC FOR M 2

The MLE for p(φ 1,2 |θ, M 2 ) can be derived by simply estimating the separate MLE solutions for p(φ 1 |θ 1 , M 2 ) and p(φ 2 |θ 2 , M 2 ). What is not as obvious is that the penalty term follows the estimation pattern.

Gradient vectors for M 2 are given by (where ⊕ is concatenation)

∇ θ L(θ|φ 1,2 ) = ∇ θ1 L(θ 1 |φ 1 ) ⊕ ∇ θ2 L(θ 2 |φ 2 ),

and Hessian results in a block diagonal matrix

∇ 2 θ L(θ|φ 1,2 ) = ∇ 2 θ1 L(θ 1 |φ 1 ) 0 0 ∇ 2 θ2 L(θ 2 |φ 2 )

, with inverse

∇ 2 θ L(θ|φ 1,2 ) -1 = ∇ 2 θ1 L(θ 1 |φ 1 ) -1 0 0 ∇ 2 θ2 L(θ 2 |φ 2 ) -1 . Computing tr( Î Ĵ -1 ) then yields tr( Î Ĵ -1 ) = tr ∇ θ L(θ|φ 1,2 )∇ θ L(θ|φ 1,2 ) ∇ 2 θ L(θ|φ 1,2 ) -1 = tr Î11 ∇ 2 θ1 L(θ 1 |φ 1 ) -1 Î12 ∇ 2 θ2 L(θ 2 |φ 2 ) -1 Î12 ∇ 2 θ1 L(θ 1 |φ 1 ) -1 Î22 ∇ 2 θ2 L(θ 2 |φ 2 ) -1 = tr Î11 ∇ 2 θ1 L(θ 1 |φ 1 ) -1 + tr Î22 ∇ 2 θ2 L(θ 2 |φ 2 ) -1 = tr Î1 Ĵ1 -1 + tr Î2 Ĵ2 -1

.

B REPARAMETRISATION OF THE VMF DISTRIBUTION:

We reparametrise the random variable to polar hypersphericals w(φ) (φ = (φ 1 , ..., φ d-1 ) ) as adopted in Mabdia ( 1975)

p(φ|θ, κ) = κ d 2 -1 (2π) d 2 I d 2 -1 (κ) ∂w ∂φ exp κµ(θ) w(φ) ,

where

w i (φ) = ((1 -δ id ) cos φ i + δ id ) i-1 k=1 sin φ k , µ i (θ) = ((1 -δ id ) cos θ i + δ id ) i-1 k=1 sin θ k , ∂w ∂φ = d-2 k=1 (sin φ k ) d-k-1 .

This reparametrisation simplifies the calculation of partial derivatives. The maxima of the likelihood remains unchanged since |∂w/∂φ| does not depend on θ thus the MLE estimate in the hypershperical coordinates parametrisation is given by applying the map from the cartesian MLE to the polars.

μ = n i=1 w i || n i=1 w i || , θ = µ -1 ( μ), A d (κ) = I d/2 I d/2-1 , R = || n i=1 w i || n , κ = A -1 d ( R) ≈ R(d -R2 ) 1 -R2 .

where both the derivation and approximation for the MLE estimates are derived in Banerjee et al. (2005) . Let D = {φ i } n i=1 be the dataset. The log likelihood is then

L(θ, κ|φ) = κw(φ) T µ(θ) -log Z(κ) + log ∂w ∂φ , L(θ, κ|D) = n i=1 L(θ, κ|φ i ).

C PARTIAL DERIVATIVE CALULATIONS (VMF LIKELIHOOD):

We first show the following result, which is useful for the full derivation. For k ≤ j

∂ ∂θ k µ j (θ) = ∂ ∂θ k ((1 -δ kd ) cos θ k + δ kd ) j-1 i=1 sin θ i = (1 -δ kj ) cos θ k sin θ k -δ kj sin θ k cos θ k ((1 -δ kd ) cos θ k + δ kd ) i-1 i=1 sin θ i = ((1 -δ kj ) cot θ k -δ kj tan θ k )µ j (θ),

where the second line comes from the fact that sin θ k (or cos θ k ) gets transformed into a cos θ k (or -sin θ k ), and thus we can revert to the original definition of µ j by multiplying with a cot θ k (or -tan θ k ). If k > j, this derivative is 0. Thus, for a single data point w(φ)

∂ ∂θ k L(θ, κ|φ) = ∂ ∂θ k κw(φ) µ(θ) - ∂ ∂θ k log Z(κ) = κw(φ) ∂µ(θ) ∂θ k = κ d j=k w j (φ)µ j (θ)((1 -δ kj ) cot θ k -δ kj tan θ k ),

where the sum starts from k, as for j < k, the derivative is zero.

The derivative with respect to κ is derived as follows

∂ ∂κ L(θ, κ|φ) = ∂ ∂κ κw(φ) µ(θ) - ∂ ∂κ log Z(κ) = w(φ) µ(θ) - I d 2 (κ) I d 2 -1 (κ)

, where the derivative of the second term is a known result.

We next focus on second order derivatives

∂ 2 ∂θ 2 k µ j (θ) = ∂ 2 ∂θ 2 k ((1 -δ id ) cos θ i + δ id ) j-1 i=1 sin θ i .

Unless this derivative is zero, we notice that we take the derivative

∂ 2 cos θ k ∂θ 2 k or ∂ 2 sin θ k ∂θ 2 k

, both of which result in the negative of the original function. Thus

∂ 2 ∂θ 2 k L(θ, κ|φ) = -κ d j=k w j (φ)µ j (θ).

The below result is given

∂ 2 ∂κ 2 L(θ, κ|φ) = ∂ ∂κ (- I d 2 (κ) I d 2 -1 (κ) ) = I d 2 (κ)(I d 2 -2 (κ) + I d 2 (κ)) -I d 2 -1 (κ)(I d 2 -1 (κ) + I d 2 +1 (κ)) 2I d 2 -1 (κ) 2

.

Next, we show that the second order mixed derivatives are a constant (with respect to φ) times ∂L(θ, κ|φ)/∂θ k , i.e.

∂ 2 L(θ, κ|φ) ∂κ∂θ k = ∂ 2 ∂κ∂θ k κw(φ) µ(θ) - ∂ ∂κ∂θ k log Z(κ) = w(φ) ∂µ(θ) ∂θ k = κ -1 ∂L(θ, κ|φ) ∂θ k , Evaluated at the MLE by definition κ -1 ∂L(θ,κ|D) ∂θ k θ= θ,κ=κ = 0 Assuming l < k (Hessian is symmetric) ∂L(θ, κ|φ) ∂θ k ∂θ l = κw(φ) ∂µ(θ) ∂θ k ∂θ l = κ d j=max(k,l) w j (φ)µ j (θ)((1 -δ kj ) cot θ k -δ kj tan θ k )((1 -δ lj ) cot θ l -δ lj tan θ l ) = κ d j=k w j (φ)µ j (θ) cot θ l ((1 -δ kj ) cot θ k -δ kj tan θ k ) = cot θ l κ d j=l w j (φ)µ j (θ)((1 -δ kj ) cot θ k -δ kj tan θ k ) = cot θ l ∂L(θ, κ|φ) ∂θ k .

Which is also zero evaluated at the MLE cot θ l ∂L(θ,κ|D) ∂θ k θ= θ,κ=κ = 0.

D PARTIAL DERIVATIVES CALCULATION (GAUSSIAN LIKELIHOOD):

The partial derivatives for the diagonal Gaussian likelihood are (we take derivatives with respect to precision λ

2 k = 1/σ 2 k ) ∂ ∂µ k L(µ, λ|w) = n i=1 λ 2 k x (i) k -µ k , ∂ 2 ∂µ 2 k L(µ, λ|w) = -nλ 2 k , ∂ ∂λ 2 k L(µ, λ|w) = n 2λ 2 k - 1 2 n i=1 x (i) k -µ k 2 , ∂ 2 ∂(λ 2 k ) 2 L(µ, λ|w) = - n 2λ 4 k , ∂ 2 ∂λ 2 k ∂µ k L(µ, λ|w) = n i=1 x (i) k -µ k .

Evaluating at the MLE we get where n, m are defined as above.

∂ 2 ∂µ 2 k L(µ, λ|D) µ= μ,λ= λ = -n λ2 k = - n σ2 k , ∂ 2 ∂(λ 2 k ) 2 L(µ, λ|D) µ= μ,λ= λ = - n 2 λ4 k = - nσ 4 k 2 , ∂ 2 ∂λ 2 k ∂µ k L(µ, λ|D) µ= μ,λ= λ = n i=1 x (i) k -μk = 0.

Equations 3 and 4 represents our similarity score under a Gaussian likelihood, for the Bayes Factor and BIC respectively.

Table E compares BIC and the Bayes Factor using a Gaussian likelihood to the approach presented in the paper. We see that while these approaches are competitive on STS-13, STS-14 and STS-15, they both give severely worse results on STS-12 and STS-16, with more than 0.08 difference. This motivates our choice to do a penalised likelihood ratio test instead of doing full Bayesian inference of the evidences.

F TIC ROBUSTNESS:

In this section, we compare the TIC and AIC on the two likelihoods described in the main text. Table F presents the results of that comparison. As we can see, with each word embedding the Gaussian AIC correction outperforms the TIC correction on average. Looking at Figure 3 , it becomes apparent why -the more parameters a Gaussian has, the more dependent its correction is on the number of words in the sentence. This is reminiscent of the linear scaling with number of words in the BIC penalty discussed in Appendix E, which was shown to perform badly. On the other hand, looking at Figure 4 , we see that the TIC for the vMF distribution has very low variance, and is generally not dependent on the number of word embeddings in the word group. This gives intuition why the AIC and TIC for the vMF give very similar results. 

:

The difference between an online and an offline setting is whether or not one has access to the entire dataset at evaluation time. That is, in an online setting, one has access to only a single query pair at a time.

The covariance matrix of n samples with d dimensions such that n < d results in a low rank matrix.

The STS13 dataset does not include the proprietary SMT dataset that was available with the original release of STS.

https://github.com/mkusner/wmd

https://github.com/PrincetonML/SIF

Figures and Tables:

Figure 3: TIC Correction penalty for varying sample sizes, samples generated from standardised normal N (0, I d ).

Comparison of Spearman correlations on the STS datasets between the two similarity measures we introduce in the text. The average is weighted according to dataset size.

Table content: <table><tr><td colspan="2">Embedding Method</td><td>STS12 STS13 STS14 STS15 STS16 Average</td></tr><tr><td>FastText</td><td colspan="2">vMF+TIC 0.5578 0.5532 0.5897 0.6660 0.6596 0.6023</td></tr><tr><td/><td colspan="2">Diag+AIC 0.6193 0.6335 0.6721 0.7328 0.7518 0.6765</td></tr><tr><td>GloVe</td><td colspan="2">vMF+TIC 0.5516 0.5873 0.5865 0.6605 0.6083 0.5977</td></tr><tr><td/><td colspan="2">Diag+AIC 0.6031 0.6132 0.6445 0.7171 0.7346 0.6564</td></tr><tr><td>Word2Vec</td><td colspan="2">vMF+TIC 0.5161 0.4997 0.5661 0.6477 0.5984 0.5683</td></tr><tr><td/><td colspan="2">Diag+AIC 0.5630 0.5799 0.6291 0.6951 0.6701 0.6265</td></tr></table>

Comparison of Spearman correlations on the STS datasets between our best model (Diag+AIC) and SIF,WMD and MWV for three different word vectors. The average is weighted according to dataset size.

Table content: <table><tr><td colspan="2">Embedding Method</td><td>STS12 STS13 STS14 STS15 STS16 Average</td></tr><tr><td>FastText</td><td colspan="2">Diag+AIC 0.6193 0.6335 0.6721 0.7328 0.7518 0.6765</td></tr><tr><td/><td>SIF</td><td>0.6003 0.6921 0.6729 0.7473 0.7012 0.6777</td></tr><tr><td/><td>MWV</td><td>0.5994 0.6494 0.6473 0.7114 0.6814 0.6542</td></tr><tr><td/><td>WMD</td><td>0.5576 0.5146 0.5915 0.6800 0.6402 0.5997</td></tr><tr><td>GloVe</td><td colspan="2">Diag+AIC 0.6031 0.6132 0.6445 0.7171 0.7346 0.6564</td></tr><tr><td/><td>SIF</td><td>0.5754 0.6269 0.6113 0.6899 0.6699 0.6286</td></tr><tr><td/><td>MWV</td><td>0.5526 0.5643 0.5625 0.6314 0.5804 0.5784</td></tr><tr><td/><td>WMD</td><td>0.5516 0.5007 0.5811 0.6704 0.6246 0.5896</td></tr><tr><td>Word2Vec</td><td colspan="2">Diag+AIC 0.5630 0.5799 0.6291 0.6951 0.6701 0.6265</td></tr><tr><td/><td>SIF</td><td>0.5400 0.6352 0.6386 0.7027 0.6413 0.6293</td></tr><tr><td/><td>MWV</td><td>0.5329 0.5796 0.5886 0.6385 0.5776 0.5846</td></tr><tr><td/><td>WMD</td><td>0.5162 0.4966 0.5778 0.6630 0.6022 0.5755</td></tr></table>

Comparison of Spearman correlations on the STS datasets between our best model (Diag+AIC) and SIF+PCA for three different word vectors.

Table content: <table/>

Spearman correlations using GloVe embeddings and Gaussian likelihood. The AIC and BIC use a diagonal covariance matrix, while the Bayes Factor uses a full covariance matrix.

Table content: <table><tr><td/><td>AIC</td><td>Bayes Factor</td><td>BIC</td></tr><tr><td>STS-12</td><td>0.6031</td><td>0.5183</td><td>0.5009</td></tr><tr><td colspan="2">STS-13 (-SMT) 0.6132</td><td>0.6297</td><td>0.6011</td></tr><tr><td>STS-14</td><td>0.6445</td><td>0.6428</td><td>0.5926</td></tr><tr><td>STS-15</td><td>0.7171</td><td>0.7429</td><td>0.6625</td></tr><tr><td>STS-16</td><td>0.7346</td><td>0.6328</td><td>0.5826</td></tr></table>

Diag+TIC 0.5366 0.5932 0.6258 0.6814 0.6215 0.6127 Diag+AIC 0.5630 0.5799 0.6291 0.6951 0.6701 0.6265 Comparison of Spearman correlations on the STS datasets between the TIC and AIC corrections for the diagonal covariance Gaussian and vMF likelihood functions.

Table content: <table><tr><td colspan="2">Embedding Method</td><td>STS12 STS13 STS14 STS15 STS16 Average</td></tr><tr><td>FastText</td><td colspan="2">vMF+TIC 0.5578 0.5532 0.5897 0.6660 0.6596 0.6023</td></tr><tr><td/><td colspan="2">vMF+AIC 0.5443 0.5445 0.5853 0.6636 0.6654 0.5966</td></tr><tr><td/><td colspan="2">Diag+TIC 0.5883 0.6553 0.6678 0.7197 0.7052 0.6626</td></tr><tr><td/><td colspan="2">Diag+AIC 0.6193 0.6335 0.6721 0.7328 0.7518 0.6764</td></tr><tr><td>GloVe</td><td colspan="2">vMF+TIC 0.5516 0.5873 0.5865 0.6605 0.6083 0.5977</td></tr><tr><td/><td colspan="2">vMF+AIC 0.5541 0.5739 0.5824 0.6576 0.6599 0.5997</td></tr><tr><td/><td colspan="2">Diag+TIC 0.5802 0.6420 0.6535 0.7179 0.6966 0.6534</td></tr><tr><td/><td colspan="2">Diag+AIC 0.6031 0.6132 0.6445 0.7171 0.7346 0.6564</td></tr><tr><td>Word2Vec</td><td colspan="2">vMF+TIC 0.5161 0.4997 0.5661 0.6477 0.5984 0.5683</td></tr><tr><td/><td colspan="2">vMF+AIC 0.5176 0.5045 0.5696 0.6495 0.6072 0.5716</td></tr></table>

Peer review (iclr2019):
 The authors propose a probabilistic model for computing the sentence similarity between two sets of representations in an online fashion (that is, they do not need to see the entire dataset at once as SIF does when using PCA). They evaluate on the STS tasks and outperform competitive baselines like WMD, averaging embeddings, and SIF (without PCA), but they have worse performance that SIF + PCA. The paper is clearly written and their model is carefully laid out along with their derivation. My concern with this paper however, is that I feel the paper lacks a motivation, was it derive an online similarity metric that outperforms SIF(without PCA)? A few experimental questions/comments: What happens to all methods when stop words are not removed? How far does performance fall? I think one reason it might fall (in addition to the reasons given in the paper) is that all vectors are set to have the same norm. For STS tasks, often the norms of these vectors are reduced during training which lessens their influence. What mechanism was used to identify the stop words and does removing these help the other methods (I know in the paper, stop words were removed in the baseline, did this unilaterally improve performance for these methods)? Overall I do like the paper, however I do find the results to be lackluster. There are many papers on combining word embeddings trained in various ways that have much stronger numbers on STS, but these methods won*t be effective with this type of similarity (namely because embeddings must have unit norm in their model). Therefore, I think the paper needs some more motivation and experimental evidence of its superiority over related methods like SIF+PCA in order for it to be accepted. PROS - Probabilistic model with clear design assumptions from which a similarity metric can be derived. - Derived similarity metric doesn*t require knowledge of the entire dataset (in comparison to SIF + PCA) CONS - Performance seems to be slightly better than SIF, WMD, and averaging word embeddings, but below that of SIF + PCA - Unclear motivation for the model, was it derive an online similarity metric that outperforms SIF(without PCA)? - Requires the removal of stop words, but doesn*t state how these were defined. Minor point, but tuning this could be enough to cause the improvement over related methods.