Venue: ACL 2020

# Estimating Mutual Information Between Dense Word Embeddings

Vitalii Zhelezniak, Aleksandar Savkov & Nils Hammerla

Babylon Health

{firstname.lastname}@babylonhealth.com

###### Abstract

Word embedding-based similarity measures are currently among the top-performing methods on unsupervised semantic textual similarity (STS) tasks. Recent work has increasingly adopted a statistical view on these embeddings, with some of the top approaches being essentially various correlations (which include the famous cosine similarity). Another excellent candidate for a similarity measure is mutual information (MI), which can capture arbitrary dependencies between the variables and has a simple and intuitive expression. Unfortunately, its use in the context of dense word embeddings has so far been avoided due to difficulties with estimating MI for continuous data. In this work we go through a vast literature on estimating MI in such cases and single out the most promising methods, yielding a simple and elegant similarity measure for word embeddings. We show that mutual information is a viable alternative to correlations, gives an excellent signal that correlates well with human judgements of similarity and rivals existing state-of-the-art unsupervised methods.

## 1 Introduction

Neural text embeddings learned from unlabeled data are a key component of modern approaches to semantic textual similarity (STS). Despite the impressive performance of large pretrained models Kiros et al. (2015); Conneau et al. (2017); Subramanian et al. (2018); Cer et al. (2018); Peters et al. (2018); Radford (2018); Devlin et al. (2018); Dai et al. (2019); Yang et al. (2019) on a plethora of hard NLP tasks, deep models do not currently offer a clear advantage over much simpler static word embeddings Bengio et al. (2003); Mikolov et al. (2013); Pennington et al. (2014); Bojanowski et al. (2017); Joulin et al. (2017) on standard unsupervised STS benchmarks Hill et al. (2016); Arora et al. (2017); Wieting et al. (2016); Wieting and Gimpel (2018); Zhelezniak et al. (2019, 2019). Instead, the main sources of improvement here have come from training on supervised paraphrastic corpora Wieting et al. (2015, 2016); Wieting and Gimpel (2018), designing better composition functions Mitchell and Lapata (2008); De Boom et al. (2016); Arora et al. (2017); Zhao and Mao (2017); Ruckle et al. (2018); Zhelezniak et al. (2019, 2019) and exploring novel similarity measures between word embeddings, in particular those inspired by optimal transport Kusner et al. (2015); Huang et al. (2016), soft and fuzzy sets Jimenez et al. (2010, 2015); Zhelezniak et al. (2019), and statistics Lev et al. (2015); Nikolentzos et al. (2017); Torki (2018); Zhelezniak et al. (2019, 2019).

Recently, Zhelezniak et al. (2019, 2019) advocated for a new statistical perspective on word embeddings where each word embedding itself is viewed as a sample of (e.g. 300) observations from some scalar random variable. They conducted a statistical analysis of several popular pretrained word embeddings and their compositions and established that the ubiquitous cosine similarity is practically equivalent to Pearson correlation. They also demonstrated significant gains in performance when one instead uses non-parametric rank correlation coefficients Spearman's \(\rho\), Kendall's \(\tau\)) and cross-covariance operators between reproducing kernel Hilbert spaces Hilbert-Schmidt independence criterion (HSIC) Gretton et al. (2005), Centered Kernel Alignment (CKA) Cortes et al. (2012); Kornblith et al. (2019).

One prominent alternative to those correlation-based approaches is mutual information (MI), which is of great importance in information theory and statistics. In some sense, mutual information is an excellent candidate for a similarity measure between word embeddings as it can capture arbitrary dependencies between the variables and has a simple and intuitive expression. Unfortunately,

its use in the context of continuous dense word representations has so far been avoided due to the difficulties in estimating MI for continuous random variables (joint and marginal densities are not known in practice).

In this work we make the first steps towards the adoption of MI as a measure of semantic similarity between dense word embeddings. We begin our discussion with how to apply MI for this purpose in principle. Next we carefully summarise the vast literature on estimation of MI for continuous random variables and identify approaches most suitable for our use case. Our chief goal here is to identify the estimators that yield elegant, almost closed-form expressions for the resulting similarity measure as opposed to complicated estimation procedures. Finally, we show that such estimators of mutual information give an excellent signal that correlates very well with human judgements and comfortably rivals existing state-of-the-art unsupervised STS approaches.

## 2 Background: Statistical Approaches to Word Embeddings

Suppose we are given a word embedding matrix \({\bf W}\in\mathbb{R}^{N\times D}\), where \(N\) is the vocabulary size and \(D\) is the embedding dimension (commonly \(D=300\)). Ultimately, the matrix \({\bf W}\) is simply a table of some numbers and just like any dataset, it is subject to a statistical analysis. There are essentially two ways we can proceed: we can either choose to view \({\bf W}\) as \(N\) observations from \(D\) random variables or we can instead consider \({\bf W}^{T}\) and view it as \(D\) observations from \(N\) randomly variables. The first approach allows us to study 'global' properties of the word embedding space (e.g. via PCA, clustering, etc.) and defines 'global' similarity structures, such as Mahalanobis distance, Fisher kernel [15], etc.

In the second approach we study the distribution \(P(W_{1},W_{2},\ldots,W_{N})\), where a word embedding \({\bf w}_{i}\) is a sample of \(D\) (\(=300\)) observations from some scalar random variable \(W_{i}\) corresponding to the word \(w_{i}\)[64, c]. The 'local' similarity between two words \(w_{i}\) and \(w_{j}\) is then encoded in the dependencies between the corresponding random variables \(W_{i},W_{j}\). Since the distribution \(P(W_{i},W_{j})\) is unknown, we estimate these dependences based on the sample \({\bf w}_{i},{\bf w}_{j}\). Certain dependencies can be captured by Pearson, Spearman and Kendall correlation coefficients between word embeddings \(\widehat{\rho}({\bf w}_{i},{\bf w}_{j})\), where the choice of the coefficient depends on the statistics of each word embedding model [78].

Conveniently, correlations can also be used to measure semantic similarity between two sets of words (e.g. phrases and sentences) if one considers the correlations between random vectors \({\bf X}=(X_{1},X_{2},\ldots,X_{l_{x}})\) and \({\bf Y}=(Y_{1},Y_{2},\ldots,Y_{l_{y}})\), where scalar random variables \(X_{i}\) correspond to the words in the first sentence and \(Y_{j}\) to the words in the second sentence. This, for example, can be done by first pooling (e.g. mean- or max-pooling) random vectors into scalar variables \(X_{\text{pool}}\) and \(Y_{\text{pool}}\) and then estimating univariate correlations corr\((X_{\text{pool}},Y_{\text{pool}})\) as before. Alternatively, we can measure correlations between random vectors directly using norms of cross-covariance matrices/operators (e.g. the Hilbert-Schmidt independence criterion [9]). Both approaches are known to give excellent results on standard STS benchmarks [78]. A viable alternative to correlations is mutual information (MI), which can detect any kind of dependence between random variables, but which has so far not been explored for this problem.

## 3 Mutual Information between Dense Word Embeddings

We operate within the previous setting where we consider two sentences \(x=x_{1}x_{2}\dots x_{l_{x}}\) and \(y=y_{1}y_{2}\dots y_{l_{y}}\). Our goal now is to estimate the mutual information I\(({\bf X};{\bf Y})\) between the corresponding random vectors \({\bf X}=(X_{1},X_{2},\ldots,X_{l_{x}})\) and \({\bf Y}=(Y_{1},Y_{2},\ldots,Y_{l_{y}})\)

\[\text{I}({\bf X};{\bf Y})=\iint p_{{\bf XY}}(x,y)\text{log}\frac{p_{{\bf XY} }(x,y)}{p_{{\bf X}}(x)p_{{\bf Y}}(y)}\text{d}x\text{d}y, \tag{1}\]

where \(p_{{\bf XY}}(x,y)\) is the joint density of \({\bf X}\) and \({\bf Y}\) and \(p_{{\bf X}}(x)=\int_{\mathcal{Y}}p_{{\bf XY}}(x,y)\text{d}y\) and \(p_{{\bf Y}}(y)=\int_{\mathcal{X}}p_{{\bf XY}}(x,y)\text{d}x\) are the marginal densities. Unfortunately, these theoretical quantities are not available to us and we must somehow estimate \(\widehat{\text{I}}({\bf X};{\bf Y})\) directly from the word embeddings \(\widehat{\textbf{X}}=({\bf x}_{(1)},{\bf x}_{(2)},\ldots,{\bf x}_{(l_{x})})\) and \(\widehat{\mathbf{Y}}=({\bf y}_{(1)},{\bf y}_{(2)},\ldots,{\bf y}_{(l_{y})})\). Luckily, there is a vast literature on how to estimate mutual information between continuous random variables based on the sample. The first class of methods partitions the supports \(\mathcal{X},\mathcal{Y}\) into a finite number of bins of equal or unequal (adaptive) size and estimates 

based on discrete counts in each bin (Moddemeijer, 1989; Fraser and Swinney, 1986; Darbellay and Vajda, 1999; Reshef et al., 2011; Ince et al., 2016). While such methods are easy to understand conceptually, they might suffer from the curse of dimensionality (especially when sentences are long) and in some sense violate our desire for an elegant closed-form similarity measure. The next class of methods constructs kernel density estimates (KDE) and then numerically integrates such approximate densities to obtain MI (Moon et al., 1995; Steuer et al., 2002). These methods might require a careful choice of kernels and the bandwidth parameters and also violate our simplicity requirement. The third class of methods that has recently gained popularity in the deep learning community is based on neural-network-based estimation of various bounds on mutual information (e.g. by training a critic to estimate the density ratio in (1)) (Suzuki et al., 2008; Alemi et al., 2017; Belghazi et al., 2018; Hjelm et al., 2019; Poole et al., 2019). Such estimators are usually differentiable and scale well to high dimensions and large sample sizes (Belghazi et al., 2018). However, in our case the sample size (e.g. 300) and dimensionality are not too large (at least for short phrases and sentences), and thus training a separate neural network for a simple similarity computation is hardly justified. This leaves us with the last class of methods that estimates mutual information from the \(k\)-nearest neighbour statistics (Kraskov et al., 2004; Ver Steeg and Galstyan, 2013; Ver Steeg, 2014; Ross, 2014; Gao et al., 2015; Gao et al., 2018). These approaches are not without problems (Gao et al., 2015) and inherit the weaknesses of \(k\)NN in large dimensions but are very simple to implement. In particular, we focus on the Kraskov-Stogbauer-Grassberger (KSG) estimator (Kraskov et al., 2004) which admits a particularly elegant expression for the resulting similarity measure.

### The KSG Similarity Measure

It can be verified that the mutual information is given by \({\rm I}({\bf X};{\bf Y})={\rm H}({\bf X})+{\rm H}({\bf Y})-{\rm H}({\bf X},{ \bf Y})\), i.e. the difference between the sum of marginal entropies and the joint entropy. Thus, in order to estimate MI, it is sufficient to be able to estimate various entropies in the above equation. In their seminal work, Kozachenko and Leonenko (1987) show how to estimate such differential entropies based on the nearest neighbour statistics. Concretely, these methods approximate the log-density at a point by a uniform density in a e.g. Euclidean or Chebyshev norm ball containing its \(k\)-nearest neighbours. Kraskov et al. (2004) modify this idea to construct their famous KSG estimator of mutual information given by

\[\begin{split}&{\rm KSG}({\bf X};{\bf Y})=\psi(D)+\psi(k)-\\ &\sum_{d=1}^{D}\left(\psi(n_{x}[d]+1)+\psi(n_{y}[d]+1)\right), \end{split} \tag{2}\]

where \(D\) is the embedding dimension, \(k\) is the number of nearest neighbours, \(\psi(x)=\Gamma^{\prime}(x)/\Gamma(x)\) is the digamma function and \(n_{x}[d],n_{y}[d]\) are certain nearest neighbour statistics. These statistics are obtained by counting the number of neighbours that fall within less than \(\epsilon[d]\) from \({\bf x}^{d}\) and \({\bf y}^{d}\) in the marginal spaces \({\bf X}\) and \({\bf Y}\) respectively, where \(\epsilon[d]\) is the distance from \({\bf z}^{d}=({\bf x}^{d},{\bf y}^{d})\) to its \(k\)-nearest neighbour in the joint space \(({\bf X},{\bf Y})\). We illustrate how the estimator can be applied to measure similarity between sets of word embeddings in Algorithm 1 and refer the reader to Kraskov et al. (2004) for its full derivation and justification as well as an alternative version 



## 4 Experiments

We now explore the empirical performance of the KSG similarity measure on a standard suite of Semantic Textual Similarity (STS) benchmarks Agirre et al. [2013, 2014, 2015, 2016] and report Spearman correlation between the system and human scores. Our focus here is on fastText vectors Bojanowski et al. [2017] trained on Common Crawl (600B tokens), as previous literature suggests that among unsupervised vectors fastText yields the best performance for all tasks and similarity measures Conneau et al. [2017]; Perone et al. [2018]; Zhelezniak et al. [2019a, b, c]. We defer evaluations and significance analysis on all 24 STS subtasks for other word vectors (word2vec and GloVe) to the Appendix. Our evaluations are run in the SentEval toolkit Conneau and Kiela [2018] and our code is available on GitHub1. Note that we do not report results on the STS13 SMT subtask as it is no longer publicly available.

Footnote 1: [https://github.com/babylonhealth/corrsim](https://github.com/babylonhealth/corrsim)

The number of nearest neighbours for KSG that is known to work well in practice on a variety of datasets is \(k=3\)Kraskov et al. [2004]; Khan et al. [2007]. This value seems to strike a good balance between the bias and variance of the estimator. We also run experiments for \(k=10\) to show that KSG is not very sensitive to this hyperparameter, at least in our setting. As an interesting addition, we also run KSG (\(k=10\)) for max-pooled scalar random variables MaxPool+KSG 10). We compare KSG to the following approaches from the literature: Universal Sentence Encoder Transformer) Cer et al. [2018], BERT (penultimate layer, mean-pooling) Devlin et al. [2018], Word Mover's Distance (WMD) Kusner et al. [2015], soft cardinality Jimenez et al. [2010, 2015] with cosine similarity and the softness parameter \(p=1\), DynaMax-Jaccard Zhelezniak et al. [2019b], mean-pooling with cosine similarity (MeanPool+COS) and Smooth Inverse Frequency (SIF) + PCA Arora et al. [2017]. Next we compare KSG with the following top-performing correlations: max-pooling with Spearman correlation (MaxPool+SPR), Centered Kernel Alignment (Gaussian kernel with median estimation for \(\sigma^{2}\)) and distance correlation Zhelezniak et al. [2019c]. The evaluation results are given in Table 1.

In summary, we can see that similarity measures based on mutual information (KSG) perform on par with top correlation-based measures and other leading methods from the literature. Moreover, KSG between pooled variables (MaxPool) is faster and performs only slightly worse than multivariate KSG.

\begin{table}
\begin{tabular}{l l l l l l} \hline \hline
**Similarity** & **STS** & **12** & **13** & **14** & **15** & **16** \\ \hline \multicolumn{6}{l}{_Popular approaches_} \\ \hline USE (Transf.) & 63.8 & 63.1 & 66.0 & 77.1 & 76.4 \\ BERT Small & 50.8 & 50.4 & 54.0 & 62.9 & 63.8 \\ BERT Large & 51.0 & 47.2 & 51.8 & 58.0 & 62.7 \\ WMD & 54.8 & 47.0 & 57.7 & 65.8 & 63.2 \\ SoftCard & 54.8 & 50.6 & 58.1 & 66.5 & 65.9 \\ DynaMax & 61.3 & 61.7 & 66.9 & 76.5 & 74.7 \\ MeanPool+COS & 58.8 & 58.8 & 63.4 & 69.1 & 68.3 \\ SIF+PCA & 58.1 & 67.2 & 66.5 & 73.8 & 73.0 \\ \hline \multicolumn{6}{l}{_Correlation-based Approaches_} \\ \hline MaxPool+SPR & 61.4 & 63.8 & 68.0 & 75.8 & 75.9 \\ CKA Gaussian & 60.8 & 64.6 & 68.0 & 76.4 & 73.8 \\ CKA dCorr & 60.9 & 63.4 & 67.8 & 76.2 & 73.4 \\ \hline \multicolumn{6}{l}{_Mutual Information (KSG)_} \\ \hline KSG \(k=3\) & 59.9 & 61.6 & 67.8 & 76.7 & 74.7 \\ KSG \(k=10\) & 60.4 & 61.5 & 68.3 & 77.0 & 75.1 \\ MaxPool+KSG 10 & 59.5 & 60.2 & 67.5 & 75.0 & 74.1 \\ \hline \end{tabular}
\end{table}
Table 1: Average Spearman correlation between system and human scores on STS 12–16 tasks. FastText is used for all methods that rely on word embeddings. Similarity measures based on Mutual Information (KSG) perform on par with correlation-based measures and other popular methods from the literature.

\begin{table}
\begin{tabular}{l l l l l l} \hline \hline
**Similarity** & **Time complexity** \\ \hline WMD & \(O(m^{2}D+m^{3}\log m)\) \\ WMD (relaxed) & \(O(m^{2}D)\) \\ SoftCard & \(O(m^{2}D)\) \\ DynaMax & \(O(m^{2}D)\) \\ MaxPool+SPR & \(O(mD+D\log D)\) \\ MaxPool+KSG & \(O(mD+D^{3/2})\) \\ CKA & \(O(mD^{2})\) \\ KSG & \(O(mD^{2})\) \\ \hline \hline \end{tabular}
\end{table}
Table 2: Computational complexity of some word embedding-based methods, where \(m\) is the length of the longer sentence and \(D\) is the word embedding dimension.





## 5 Conclusion

In this work we explored how to apply mutual information (MI) as a semantic similarity measure for continuous dense word embeddings. We have summarised the vast literature on estimating MI for continuous random variables from the sample and singled out a simple and elegant KSG estimator which is based on elementary nearest-neighbour statistics. We showed empirically that this estimator and mutual information in general can be an excellent candidate for a similarity measure between dense word embeddings.

## Acknowledgements

We would like to thank Adam Bozson and the four anonymous reviewers for their useful feedback and suggestions.

## References

* Agirre et al. (2015) Eneko Agirre, Carmen Banea, Claire Cardie, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Weiwei Guo, Inigo Lopez-Gazpio, Montse Maritxalar, Rada Mihalcea, German Rigau, Larraitz Uria, and Janyvee Wiebe. 2015. Semeval-2015 task 2: Semantic textual similarity, english, spanish and pilot on interpretability. In _Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015)_, pages 252-263. Association for Computational Linguistics.
* Agirre et al. (2014) Eneko Agirre, Carmen Banea, Claire Cardie, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Weiwei Guo, Rada Mihalcea, German Rigau, and Junyce Wiebe. 2014. Semeval-2014 task 10: Multilingual semantic textual similarity. In _Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014)_, pages 81-91. Association for Computational Linguistics.
* Agirre et al. (2016) Eneko Agirre, Carmen Banea, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Rada Mihalcea, German Rigau, and Junyce Wiebe. 2016. Semeval-2016 task 1: Semantic textual similarity, monolingual and cross-lingual evaluation. In _Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016)_, pages 497-511. Association for Computational Linguistics.
* Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012)_, pages 385-393. Association for Computational Linguistics.
* Agirre et al. (2013) Eneko Agirre, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, and Weiwei Guo. 2013. "sem 2013 shared task: Semantic textual similarity. In _Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 1: Proceedings of the Main Conference and the Shared Task: Semantic Textual Similarity_, pages 32-43. Association for Computational Linguistics.
* Alemi et al. (2017) Alex Alemi, Ian Fischer, Josh Dillon, and Kevin Murphy. 2017. Deep variational information bottleneck. In _ICLR_.
* Arora et al. (2017) Sanjeev Arora, Yingyu Liang, and Tengyu Ma. 2017. A Simple but Tough-to-Beat Baseline for Sentence Embeddings. _International Conference on Learning Representations_.
* Belghazi et al. (2018) Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and Devon Hjelm. 2018. Mutual information neural estimation. In _Proceedings of the 35th International Conference on Machine Learning_, volume 80 of _Proceedings of Machine Learning Research_, pages 531-540, Stockholmsmassan, Stockholm Sweden. PMLR.
* Bengio et al. (2003) Yoshua Bengio, Rjean Ducharme, Pascal Vincent, and Christian Jauvin. 2003. A neural probabilistic language model. _Journal of Machine Learning Research_, 3:1137-1155.
* Bojanowski et al. (2017) Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. _Transactions of the Association for Computational Linguistics_, 5:135-146.
* Cer et al. (2018) Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhommi St. John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, Brian Strope, and Ray Kurzweil. 2018. Universal sentence encoder for english. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_, pages 169-174. Association for Computational Linguistics.
* Conneau and Kiela (2018) Alexis Conneau and Douwe Kiela. 2018. Senteval: An evaluation toolkit for universal sentence representations. In _Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC-2018)_. European Language Resource Association.
* Conneau et al. (2017) Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, and Antoine Bordes. 2017. Supervised learning of universal sentence representations from natural language inference data. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_, pages 670-680. Association for Computational Linguistics.
* Cortes et al. (2012) Corinna Cortes, Mehryar Mohri, and Afshin Rostamizadeh. 2012. Algorithms for learning kernels based on centered alignment. _Journal of Machine Learning Research_, 13(Mar):795-828

Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov. 2019. Transformer-XL: Attentive language models beyond a fixed-length context. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 2978-2988, Florence, Italy. Association for Computational Linguistics.
* Darbellay and Vajda (1999) G.A. Darbellay and I. Vajda. 1999. Estimation of the information by an adaptive partitioning of the observation space. _IEEE Transactions on Information Theory_, 45(4):1315-1321.
* De Boom et al. (2016) Cedric De Boom, Steven Van Canneyt, Thomas Demeester, and Bart Dhoedt. 2016. Representation learning for very short texts using weighted word embedding aggregation. _Pattern Recogn. Lett._, 80(C):150-156.
* Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. _arXiv preprint arXiv:1810.04805_.
* Efron (1987) Bradley Efron. 1987. Better bootstrap confidence intervals. _Journal of the American Statistical Association_, 82(397):171-185.
* Fraser and Swinney (1986) Andrew M. Fraser and Harry L. Swinney. 1986. Independent coordinates for strange attractors from mutual information. _Phys. Rev. A_, 33:1134-1140.
* Gao et al. (2015) Shuyang Gao, Greg Ver Steeg, and Aram Galstyan. 2015. Efficient Estimation of Mutual Information for Strongly Dependent Variables. In _Proceedings of the Eighteenth International Conference on Artificial Intelligence and Statistics_, volume 38 of _Proceedings of Machine Learning Research_, pages 277-286, San Diego, California, USA. PMLR.
* Gao et al. (2018) W. Gao, S. Oh, and P. Viswanath. 2018. Demystifying fixed \(k\) -nearest neighbor information estimators. _IEEE Transactions on Information Theory_, 64(8):5629-5661.
* Gretton et al. (2005) Arthur Gretton, Olivier Bousquet, Alex Smola, and Bernhard Scholkopf. 2005. Measuring statistical dependence with hilbert-schmidt norms. In _International conference on algorithmic learning theory_, pages 63-77. Springer.
* Hill et al. (2016) Felix Hill, Kyunghyun Cho, and Anna Korhonen. 2016. Learning distributed representations of sentences from unlabelled data. In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 1367-1377. Association for Computational Linguistics.
* Hjelm et al. (2019) R Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. 2019. Learning deep representations by mutual information estimation and maximization. In _International Conference on Learning Representations_.
* Huang et al. (2016) Gao Huang, Chuan Quo, Matt J. Kusner, Yu Sun, Kilian Q. Weinberger, and Fei Sha. 2016. Supervised word mover's distance. In _Proceedings of the 30th International Conference on Neural Information Processing Systems_, NIPS'16, pages 4869-4877, USA. Curran Associates Inc.
* Inc et al. (2016) Robin A.A. Ince, Bruno L. Giordano, Christoph Kayser, Guillaume A. Rousselet, Joachim Gross, and Philippe G. Schwns. 2016. A statistical framework for neuroimaging data analysis based on mutual information estimated via a gaussian copula. _Human Brain Mapping_, 38(3):1541-1573.
* Jimenez et al. (2010) Sergio Jimenez, Fabio Gonzalez, and Alexander Gelbukh. 2010. Text comparison using soft cardinality. In _String Processing and Information Retrieval_, pages 297-302, Berlin, Heidelberg. Springer Berlin Heidelberg.
* Jimenez et al. (2015) Sergio Jimenez, Fabio A. Gonzalez, and Alexander Gelbukh. 2015. Soft cardinality in semantic text processing: Experience of the SemEval international competitions. _Polibits_, 51:63-72.
* Joulin et al. (2017) Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2017. Bag of tricks for efficient text classification. In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers_, pages 427-431. Association for Computational Linguistics.
* Khan et al. (2007) Shiraj Khan, Sharba Bandyopadhyay, Auroop R. Ganguly, Sunil Saigal, David J. Erickson, Vladimir Protopopescu, and George Ostrouchov. 2007. Relative performance of mutual information estimation methods for quantifying the dependence among short and noisy data. _Physical Review E_, 76(2).
* Kiros et al. (2015) Ryan Kiros, Yukun Zhu, Ruslan R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Skip-Thought Vectors. In _Advances in Neural Information Processing Systems_, pages 3294-3302.
* Kornblith et al. (2019) Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. 2019. Similarity of neural network representations revisited. In _ICML_.
* Kozachenko and Leonenko (1987) LF Kozachenko and Nikolai N Leonenko. 1987. Sample estimate of the entropy of a random vector. _Problemy Peredachi Informatsii_, 23(2):9-16.
* Kraskov et al. (2004) Alexander Kraskov, Harald Stogbauer, and Peter Grassberger. 2004. Estimating mutual information. _Phys. Rev. E_, 69:066138.
* Kusner et al. (2015) Matt J. Kusner, Yu Sun, Nicholas I. Kolkin, and Kilian Q. Weinberger. 2015. From word embeddings to document distances. In _Proceedings of the 32Nd International Conference on International Conference on Machine Learning_, volume 37 of _ICML'15_, pages 957-966. JMLR.org 

Guy Lev, Benjamin Klein, and Lior Wolf. 2015. In defense of word embedding for generic text representation. In _Natural Language Processing and Information Systems_, pages 35-50, Cham. Springer International Publishing.
* Mikolov et al. (2013) Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient Estimation of Word Representations in Vector Space. _arXiv preprint arXiv:1301.3781_.
* Mitchell and Lapata (2008) Jeff Mitchell and Mirella Lapata. 2008. Vector-based models of semantic composition. In _Proceedings of ACL-08: HLT_, pages 236-244. Association for Computational Linguistics.
* Moddemeijer (1989) R. Moddemeijer. 1989. On estimation of entropy and mutual information of continuous distributions. _Signal Processing_, 16(3):233-248.
* Moon et al. (1995) Young-Il Moon, Balaji Rajagopalan, and Upmanu Lall. 1995. Estimation of mutual information using kernel density estimators. _Physical Review E_, 52(3):2318-2321.
* Nikolentzos et al. (2017) Giannis Nikolentzos, Polykarpos Meladianos, Francois Rousseau, Yannis Stavrakas, and Michalis Vazirgiannis. 2017. Multivariate Gaussian document representation from word embeddings for text categorization. In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers_, pages 450-455, Valencia, Spain. Association for Computational Linguistics.
* Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 1532-1543. Association for Computational Linguistics.
* Perone et al. (2018) Christian S Perone, Roberto Silveira, and Thomas S Paula. 2018. Evaluation of sentence embeddings in downstream and linguistic probing tasks. _arXiv preprint arXiv:1806.06259_.
* Peters et al. (2018) Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_, pages 2227-2237. Association for Computational Linguistics.
* Poole et al. (2019) Ben Poole, Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, and George Tucker. 2019. On variational bounds of mutual information. In _Proceedings of the 36th International Conference on Machine Learning_, volume 97 of _Proceedings of Machine Learning Research_, pages 5171-5180, Long Beach, California, USA. PMLR.
* Radford (2018) Alec Radford. 2018. Improving language understanding by generative pre-training.
* Reshef et al. (2011) D. N. Reshef, Y. A. Reshef, H. K. Finucane, S. R. Grossman, G. McVean, P. J. Turnbaugh, E. S. Lander, M. Mitzenmacher, and P. C. Sabetti. 2011. Detecting novel associations in large data sets. _Science_, 334(6062):1518-1524.
* Ross (2014) Brian C. Ross. 2014. Mutual information between discrete and continuous data sets. _PLoS ONE_, 9(2):e87357.
* Ruckle et al. (2018) Andreas Ruckle, Steffen Eger, Maxime Peyrard, and Iryna Gurevych. 2018. Concatenated p-mean word embeddings as universal cross-lingual sentence representations. _CoRR_, abs/1803.01400.
* Steuer et al. (2002) R. Steuer, J. Kurths, C. O. Daub, J. Weise, and J. Selbig. 2002. The mutual information: Detecting and evaluating dependencies between variables. _Bioinformatics_, 18(Suppl 2):S231-S240.
* Subramanian et al. (2018) Sandeep Subramanian, Adam Trischler, Yoshua Bengio, and Christopher J Pal. 2018. Learning general purpose distributed sentence representations via large scale multi-task learning. In _International Conference on Learning Representations_.
* Suzuki et al. (2008) Taiji Suzuki, Masashi Sugiyama, Jun Sese, and Takafumi Kanamori. 2008. Approximating mutual information by maximum likelihood density ratio estimation. In _Proceedings of the Workshop on New Challenges for Feature Selection in Data Mining and Knowledge Discovery at ECML/PKDD 2008_, volume 4 of _Proceedings of Machine Learning Research_, pages 5-20, Antwerp, Belgium. PMLR.
* Torki (2018) Marwan Torki. 2018. A document descriptor using covariance of word vectors. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_, pages 527-532, Melbourne, Australia. Association for Computational Linguistics.
* Ver Steeg (2014) Greg Ver Steeg. 2014. Non-parametric entropy estimation toolbox (NPEET).
* Veteg and Galstyan (2013) Greg Ver Steeg and Aram Galstyan. 2013. Information-theoretic measures of influence based on content dynamics. In _Proceedings of the Sixth ACM International Conference on Web Search and Data Mining_, WSDM '13, pages 3-12, New York, NY, USA. ACM.
* Wieting et al. (2015) John Wieting, Mohit Bansal, Kevin Gimpel, and Karen Livescu. 2015. From paraphrase database to compositional paraphrase model and back. _Transactions of the Association for Computational Linguistics_, 3:345-358.
* Wieting et al. (2016) John Wieting, Mohit Bansal, Kevin Gimpel, and Karen Livescu. 2016. Towards Universal Paraphrasic Sentence Embeddings. In _International Conference on Learning Representations_ 

John Wieting and Kevin Gimpel. 2018. Parammt-50m: Pushing the limits of paraphrastic sentence embeddings with millions of machine translations. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 451-462. Association for Computational Linguistics.
* Yang et al. (2019a) Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019a. Xlnet: Generalized autoregressive pretraining for language understanding. _CoRR_, abs/1906.08237.
* Yang et al. (2019b) Ziyi Yang, Chenguang Zhu, and Weizhu Chen. 2019b. Parameter-free sentence embedding via orthogonal basis. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 638-648, Hong Kong, China. Association for Computational Linguistics.
* Zhao and Mao (2017) Rui Zhao and Kezhi Mao. 2017. Fuzzy bag-of-words model for document representation. _IEEE Transactions on Fuzzy Systems_, pages 1-1.
* Zhelezniak et al. (2019a) Vitalii Zhelezniak, Aleksandar Savkov, April Shen, Nils Hammerla. 2019a. Correlation coefficients and semantic textual similarity. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 951-962, Minneapolis. Association for Computational Linguistics.
* Zhelezniak et al. (2019b) Vitalii Zhelezniak, Aleksandar Savkov, April Shen, Francesco Moramarco, Jack Flann, and Nils Y. Hammerla. 2019b. Don't settle for average, go for the max: Fuzzy sets and max-pooled word vectors. In _International Conference on Learning Representations_.
* Zhelezniak et al. (2019c) Vitalii Zhelezniak, April Shen, Daniel Busbridge, Aleksandar Savkov, and Nils Hammerla. 2019c. Correlations between word vector sets. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 77-87, Hong Kong, China. Association for Computational Linguistics.
 

* [14] M. C. Bender, J. R. Friedman, and A. S. Gheorghi, "Theoretical Population of the Universe: The Role of Life in the Early Universe," _Astrophys. J._**630**(2), 579-804

[MISSING_PAGE_POST]

12 

* [14] M. C. Bender, J. R. Friedman, and A. S. Gheorghi, _Theoretical Population of the Universe: An Introduction to the Theory of Evolution_, volume 207 of _Lecture Notes in Physics_, vol. 389 of _Lecture Notes in Physics_, vol. 563 of _Lecture Notes in Physics_, vol. 564 of _Lecture Notes in Physics_, vol. 565 of _Lecture Notes in Physics_, vol. 566 of _Lecture Notes in Physics_, vol. 567 of _Lecture Notes in Physics_, vol. 568 of _Lecture Notes in Physics_, vol. 569 of _Lecture Notes in Physics_, vol. 569 of _Lecture Notes in Physics_, vol. 569 of _Lecture Notes in Physics_, vol. 570 of _Lecture Notes in Physics_, vol. 571 of _Lecture Notes in Physics_, vol. 572 of _Lecture Notes in Physics_, vol. 573 of _Lecture Notes in Physics_, vol. 574 of _Lecture Notes in Physics_, vol. 575 of _Lecture Notes in Physics_, vol. 576 of _Lecture Notes in Physics_, vol. 577 of _Lecture Notes in Physics_, vol. 578 of _Lecture Notes in Physics_, vol.

 

* [14] J. M. Cannon, R. A. Shannon and S. W. H. Freeman, _The Theory of Computation_, vol. 20, no. 3, pp. 57-68, 1970.
* [15] D. B. Cox, P. L. Johnson, and T. E. Barnes, _A Practical Guide to the Statistical Sciences of America_, vol. 1, p. 103-113, 1970.

[MISSING_PAGE_POST]