Introduction
Representing the meanings of words is a fundamental task in Natural Language Processing (NLP). One popular approach to represent the meaning of a word is to embed it in some fixed-dimensional vector space (). In contrast to sparse and high-dimensional counting-based distributional word representation methods that use co-occurring contexts of a word as its representation (), dense and low-dimensional prediction-based distributed word representations have obtained impressive performances in numerous NLP tasks such as sentiment classification (), and machine translation (). Several distributed word embedding learning methods based on different learning strategies have been proposed (;;;;).
Previous works studying the differences in word embedding learning methods (;) have shown that word embeddings learnt using different methods and from different resources have significant variation in quality and characteristics of the semantics captured. For example, Hill:NIPS:2014,Hill:ICLR:2015 showed that the word embeddings trained from monolingual vs. bilingual corpora capture different local neighbourhoods. Bansal:ACL:2014 showed that an ensemble of different word representations improves the accuracy of dependency parsing, implying the complementarity of the different word embeddings. This suggests the importance of meta-embedding – creating a new embedding by combining different existing embeddings. We refer to the input word embeddings to the meta-embedding process as the source embeddings. Yin:ACL:2016 showed that by meta-embedding five different pre-trained word embeddings, we can overcome the out-of-vocabulary problem, and improve the accuracy of cross-domain part-of-speech (POS) tagging. Encouraged by the above-mentioned prior results, we expect an ensemble containing multiple word embeddings to produce better performances than the constituent individual embeddings in NLP tasks.
There are three main challenges a meta-embedding learning method must overcome.
First, the vocabularies covered by the source embeddings might be different because they have been trained on different text corpora. Therefore, not all words will be equally represented by all the source embeddings. Even in situations where the implementations of the word embedding learning methods are publicly available, it might not be possible to retrain those embeddings because the text corpora on which those methods were originally trained might not be publicly available. Moreover, it is desirable if the meta-embedding method does not require the original resources upon which they were trained such as corpora or lexicons, and can directly work with the pre-trained word embeddings. This is particularly attractive from a computational point of view because re-training source embedding methods on large corpora might require significant processing times and resources.
Second, the vector spaces and their dimensionalities of the source embeddings might be different. In most prediction-based word embedding learning methods the word vectors are randomly initialised. Therefore, there is no obvious correspondence between the dimensions in two word embeddings learnt even from two different runs of the same method, let alone from different methods (). Moreover, the pre-trained word embeddings might have different dimensionalities, which is often a hyperparameter set experimentally. This becomes a challenging task when incorporating multiple source embeddings to learn a single meta-embedding because the alignment between the dimensionalities of the source embeddings is unknown.
Third, the local neighbourhoods of a particular word under different word embeddings show a significant diversity. For example, as the nearest neighbours of the word bank, GloVe (), a word sense insensitive embedding, lists credit, financial, cash, whereas word sense sensitive embeddings created by Huang:ACL:2012 lists river, valley, marsh when trained on the same corpus. We see that the nearest neighbours for the different senses of the word bank (i.e. financial institution vs. river bank) are captured by the different word embeddings. Meta-embedding learning methods that learn a single global projection over the entire vocabulary are insensitive to such local variations in the neighbourhoods ().
To overcome the above-mentioned challenges, we propose a locally-linear meta-embedding learning method that (a) requires only the words in the vocabulary of each source embedding, without having to predict embeddings for missing words, (b) can meta-embed source embeddings with different dimensionalities, (c) is sensitive to the diversity of the neighbourhoods of the source embeddings.
Our proposed method comprises of two steps: a neighbourhood reconstruction step (Section "Nearest Neighbour Reconstruction" ), and a projection step (Section "Projection to Meta-Embedding Space" ). In the reconstruction step, we represent the embedding of a word by the linearly weighted combination of the embeddings of its nearest neighbours in each source embedding space. Although the number of words in the vocabulary of a particular source embedding can be potentially large, the consideration of nearest neighbours enables us to limit the representation to a handful of parameters per each word, not exceeding the neighbourhood size. The weights we learn are shared across different source embeddings, thereby incorporating the information from different source embeddings in the meta-embedding. Interestingly, vector concatenation, which has found to be an accurate meta-embedding method, can be derived as a special case of this reconstruction step.
Next, the projection step computes the meta-embedding of each word such that the nearest neighbours in the source embedding spaces are embedded closely to each other in the meta-embedding space. The reconstruction weights can be efficiently computed using stochastic gradient descent, whereas the projection can be efficiently computed using a truncated eigensolver.
It is noteworthy that we do not directly compare different source embeddings for the same word in the reconstruction step nor in the projection step. This is important because the dimensions in source word embeddings learnt using different word embedding learning methods are not aligned. Moreover, a particular word might not be represented by all source embeddings. This property of the proposed method is attractive because it obviates the need to align source embeddings, or predict missing source word embeddings prior to meta-embedding. Therefore, all three challenges described above are solved by the proposed method.
The above-mentioned properties of the proposed method enables us to compute meta-embeddings for five different source embeddings covering 2.7 million unique words. We evaluate the meta-embeddings learnt by the proposed method on semantic similarity prediction, analogy detection, relation classification, and short-text classification tasks. The proposed method significantly outperforms several competitive baselines and previously proposed meta-embedding learning methods () on multiple benchmark datasets.
Related Work
Yin:ACL:2016 proposed a meta-embedding learning method (1TON) that projects a meta-embedding of a word into the source embeddings using separate projection matrices. The projection matrices are learnt by minimising the sum of squared Euclidean distance between the projected source embeddings and the corresponding original source embeddings for all the words in the vocabulary. They propose an extension (1TON+) to their meta-embedding learning method that first predicts the source word embeddings for out-of-vocabulary words in a particular source embedding, using the known word embeddings. Next, 1TON method is applied to learn the meta-embeddings for the union of the vocabularies covered by all of the source embeddings.
Experimental results in semantic similarity prediction, word analogy detection, and cross-domain POS tagging tasks show the effectiveness of both 1TON and 1TON+. In contrast to our proposed method which learns locally-linear projections that are sensitive to the variations in the local neighbourhoods in the source embeddings, 1TON and 1TON+ can be seen as globally linear projections between meta and source embedding spaces. As we see later in Section "Meta-Embedding Results" , our proposed method outperforms both of those methods consistently in all benchmark tasks demonstrating the importance of neighbourhood information when learning meta-embeddings. Moreover, our proposed meta-embedding method does not directly compare different source embeddings, thereby obviating the need to predict source embeddings for out-of-vocabulary words. Locally-linear embeddings are attractive from a computational point-of-view as well because during optimisation we require information from only the local neighbourhood of each word.
Although not learning any meta-embeddings, several prior work have shown that by incorporating multiple word embeddings learnt using different methods improve performance in various NLP tasks. For example, tsuboi:2014:EMNLP2014 showed that by using both word2vec and GloVe embeddings together in a POS tagging task, it is possible to improve the tagging accuracy, if we had used only one of those embeddings. Similarly, Turian:ACL:2010 collectively used Brown clusters, CW and HLBL embeddings, to improve the performance of named entity recognition and chucking tasks.
Luo:AAAI:2014 proposed a multi-view word embedding learning method that uses a two-sided neural network. They adapt pre-trained CBOW () embeddings from Wikipedia and click-through data from a search engine. Their problem setting is different from ours because their source embeddings are trained using the same word embedding learning method but on different resources whereas, we consider source embeddings trained using different word embedding learning methods and resources. Although their method could be potentially extended to meta-embed different source embeddings, the unavailability of their implementation prevented us from exploring this possibility.
AAAI:2016:Goikoetxea showed that concatenation of word embeddings learnt separately from a corpus and the WordNet to produce superior word embeddings. Moreover, performing Principal Component Analysis (PCA) on the concatenated embeddings slightly improved the performance on word similarity tasks. In Section "Baselines" , we discuss the relationship between the proposed method and vector concatenation.
Problem Settings
To explain the proposed meta-embedding learning method, let us consider two source word embeddings, denoted by $_{1}$ and $_{2}$ . Although we limit our discussion here to two source embeddings for the simplicity of the description, the proposed meta-embedding learning method can be applied to any number of source embeddings. Indeed in our experiments we consider five different source embeddings. Moreover, the proposed method is not limited to meta-embedding unigrams, and can be used for $n$ -grams of any length $n$ , provided that we have source embeddings for those $n$ -grams.
We denote the dimensionalities of $_{1}$ and $_{2}$ respectively by $d_{1}$ and $d_{2}$ (in general, $d_{1} \ne d_{2}$ ). The sets of words covered by each source embedding (i.e. vocabulary) are denoted by $_{1}$ and $_{2}$ . The source embedding of a word $v \in _{1}$ is represented by a vector $\vec{v}^{(1)} \in ^{d_{1}}$ , whereas the same for a word $v \in _{2}$ by a vector $_{2}$0 . Let the set union of $_{2}$1 and $_{2}$2 be $_{2}$3 containing $_{2}$4 words. In particular, note that our proposed method does not require a word $_{2}$5 to be represented by all source embeddings, and can operate on the union of the vocabularies of the source embeddings. The meta-embedding learning problem is then to learn an embedding $_{2}$6 in a meta-embedding space $_{2}$7 with dimensionality $_{2}$8 for each word $_{2}$9 .
For a word $v$ , we denote its $k$ -nearest neighbour set in embedding spaces $_{1}$ and $_{2}$ respectively by $_{1}(v)$ and $_{2}(v)$ (in general, $|_{1}(v)| \ne |_{2}(v)|$ ). As discussed already in Section "Problem Settings" , different word embedding methods encode different aspects of lexical semantics, and are likely to have different local neighbourhoods. Therefore, by requiring the meta embedding to consider different neighbourhood constraints in the source embedding spaces we hope to exploit the complementarity in the source embeddings.
Nearest Neighbour Reconstruction
The first-step in learning a locally linear meta-embedding is to reconstruct each source word embedding using a linearly weighted combination of its $k$ -nearest neighbours. Specifically, we construct each word $v \in $ separately from its $k$ -nearest neighbours $_{1}(v)$ , and $_{2}(v)$ . The reconstruction weight $w_{vu}$ assigned to a neighbour $u \in _{1}(v) \cup _{2}(v)$ is found by minimising the reconstruction error $\Phi ({W})$ defined by ( "Nearest Neighbour Reconstruction" ), which is the sum of local distortions in the two source embedding spaces. (W) = i=12v v(i) - u i(v) wvu u(i)22 Words that are not $k$ -nearest neighbours of $v$ in either of the source embedding spaces will have their weights set to zero (i.e. $v \in $0 ). Moreover, we require the sum of reconstruction weights for each $v \in $1 to be equal to one (i.e. $v \in $2 ).
To compute the weights $w_{vu}$ that minimise ( "Nearest Neighbour Reconstruction" ), we compute its error gradient $\frac{\partial \Phi ({W})}{\partial w_{vu}}$ as follows: -2i=12(v(i) - x i(v) wvx x(i))u(i)I[u i(v)] Here, the indicator function, $\mathbb {I}[x]$ , returns 1 if $x$ is true and 0 otherwise. We uniformly randomly initialise the weights $w_{vu}$ for each neighbour $u$ of $v$ , and use stochastic gradient descent (SGD) with the learning rate scheduled by AdaGrad () to compute the optimal values of the weights. The initial learning rate is set to $0.01$ and the maximum number of iterations to 100 in our experiments. Empirically we found that these settings to be adequate for convergence. Finally, we normalise the weights $w_{uv}$ for each $v$ such that they sum to 1 (i.e. $\frac{\partial \Phi ({W})}{\partial w_{vu}}$0 ).
Exact computation of $k$ nearest neighbours for a given data point in a set of $n$ points requires all pairwise similarity computations. Because we must repeat this process for each data point in the set, this operation would require a time complexity of $Ø(n^{3})$ . This is prohibitively large for the vocabularies we consider in NLP where typically $n>10^{3}$ . Therefore, we resort to approximate methods for computing $k$ nearest neighbours. Specifically, we use the BallTree algorithm () to efficiently compute the approximate $k$ -nearest neighbours, for which the time complexity of tree construction is $Ø(n \log n)$ for $n$ data points.
The solution to the least square problem given by ( "Nearest Neighbour Reconstruction" ) subjected to the summation constraints can be found by solving a set of linear equations. Time complexity of this step is $(N (d_{1} |_{1}|^{3} + d_{2} |_{2}|^{3}))$ , which is cubic in the neighbourhood size and linear in both the dimensionalities of the embeddings and vocabulary size. However, we found that the iterative estimation process using SGD described above to be more efficient in practice. Because $k$ is significantly smaller than the number of words in the vocabulary, and often the word being reconstructed is contained in the neighbourhood, the reconstruction weight computation converges after a small number (less than 5 in our experiments) of iterations.
Projection to Meta-Embedding Space
In the second step of the proposed method, we compute the meta-embeddings $\vec{v}^{()}, \vec{u}^{()} \in ^{d_{}}$ for words $v, u \in $ using the reconstruction weights $w_{vu}$ we computed in Section "Nearest Neighbour Reconstruction" . Specifically, the meta-embeddings must minimise the projection cost, $\Psi ()$ , defined by ( 4 ).
$$\Psi () = \sum _{v \in } {\vec{v}^{()} - \sum _{i=1}^{2}\sum _{u \in _{i}(v)} w_{vu}\vec{u}^{()}}_{2}^{2}$$   (Eq. 4)
By finding a $$ space that minimises ( 4 ), we hope to preserve the rich neighbourhood diversity in all source embeddings within the meta-embedding. The two summations in ( 4 ) over $N_{1}(v)$ and $N_{2}(v)$ can be combined to re-write ( 4 ) as follows:
$$\Psi () = \sum _{v \in } {\vec{v}^{()} - \sum _{u \in _{1}(v) \cup _{2}(v)} w^{\prime }_{vu} \vec{u}^{()}}_{2}^{2}$$   (Eq. 5)
Here, $w^{\prime }_{uv}$ is computed using ( 6 ).
$$w^{\prime }_{vu} = w_{vu}\sum _{i=1}^{2} \mathbb {I}[u \in _{i}(v)]$$   (Eq. 6)
The $d_{}$ dimensional meta-embeddings are given by the eigenvectors corresponding to the smallest $(d_{} + 1)$ eigenvectors of the matrix ${M}$ given by ( 7 ).
$${M} = ({I} - {W}^{\prime }){I} - {W}^{\prime })$$   (Eq. 7)
Here, ${W}^{\prime }$ is a matrix with the $(v,u)$ element set to $w^{\prime }_{vu}$ . The smallest eigenvalue of ${M}$ is zero and the corresponding eigenvector is discarded from the projection. The eigenvectors corresponding to the next smallest $d_{}$ eigenvalues of the symmetric matrix ${M}$ can be found without performing a full matrix diagonalisation (). Operations involving ${M}$ such as the left multiplication by ${M}$ , which is required by most sparse eigensolvers, can exploit the fact that ${M}$ is expressed in ( 7 ) as the product between two sparse matrices. Moreover, truncated randomised methods () can be used to find the smallest eigenvectors, without performing full eigen decompositions. In our experiments, we set the neighbourhood sizes for all words in all source embeddings equal to $n$ (i.e $(v,u)$0 ), and project to a $(v,u)$1 dimensional meta-embedding space.
Source Word Embeddings
We use five previously proposed pre-trained word embedding sets as the source embeddings in our experiments:
(a) HLBL – hierarchical log-bilinear () embeddings released by Turian:ACL:2010 (246,122 word embeddings, 100 dimensions, trained on Reuters Newswire (RCV1) corpus),
(b) Huang – Huang:ACL:2012 used global contexts to train multi-prototype word embeddings that are sensitive to word senses (100,232 word embeddings, 50 dimensions, trained on April 2010 snapshot of Wikipedia),
(c) GloVe – Pennington:EMNLP:2014 used global co-occurrences of words over a corpus to learn word embeddings (1,193,514 word embeddings, 300 dimensions, trained on 42 billion corpus of web crawled texts),
(d) CW – Collobert:ICML:2008 learnt word embeddings following a multitask learning approach covering multiple NLP tasks (we used the version released by () trained on the same corpus as HLBL containing 268,810 word embeddings, 200 dimensions),
(e) CBOW – Mikolov:NIPS:2013 proposed the continuous bag-of-words method to train word embeddings (we discarded phrase embeddings and selected 929,922 word embeddings, 300 dimensions, trained on the Google News corpus containing ca. 100 billion words).
The intersection of the five vocabularies is 35,965 words, whereas their union is 2,788,636. Although any word embedding can be used as a source we select the above-mentioned word embeddings because (a) our goal in this paper is not to compare the differences in performance of the source embeddings, and (b) by using the same source embeddings as in prior work (), we can perform a fair evaluation. In particular, we could use word embeddings trained by the same algorithm but on different resources, or different algorithms on the same resources as the source embeddings. We defer such evaluations to an extended version of this conference submission.
Evaluation Tasks
The standard protocol for evaluating word embeddings is to use the embeddings in some NLP task and to measure the relative increase (or decrease) in performance in that task. We use four such extrinsic evaluation tasks:
We measure the similarity between two words as the cosine similarity between the corresponding embeddings, and measure the Spearman correlation coefficient against the human similarity ratings. We use Rubenstein and Goodenough's dataset () (RG, 65 word-pairs), rare words dataset (RW, 2034 word-pairs) (), Stanford's contextual word similarities (SCWS, 2023 word-pairs) (), the MEN dataset (3000 word-pairs) (), and the SimLex dataset () (SL 999 word-pairs).
In addition, we use the Miller and Charles' dataset () (MC, 30 word-pairs) as a validation dataset to tune various hyperparameters such as the neighbourhood size, and the dimensionality of the meta-embeddings for the proposed method and baselines.
Using the CosAdd method, we solve word-analogy questions in the Google dataset (GL) () (19544 questions), and in the SemEval (SE) dataset (). Specifically, for three given words $a$ , $b$ and $c$ , we find a fourth word $d$ that correctly answers the question $a$ to $b$ is $c$ to what? such that the cosine similarity between the two vectors $(\vec{b} - \vec{a} + \vec{c})$ and $\vec{d}$ is maximised.
We use the DiffVec (DV) () dataset containing 12,458 triples of the form $(\textrm {relation}, \textrm {word}_{1}, \textrm {word}_{2})$ covering 15 relation types. We train a 1-nearest neighbour classifer where for each target tuple we measure the cosine similarity between the vector offset for its two word embeddings, and those of the remaining tuples in the dataset. If the top ranked tuple has the same relation as the target tuple, then it is considered to be a correct match. We compute the (micro-averaged) classification accuracy over the entire dataset as the evaluation measure.
We use two binary short-text classification datasets: Stanford sentiment treebank (TR) (903 positive test instances and 903 negative test instances), and the movie reviews dataset (MR) () (5331 positive instances and 5331 negative instances). Each review is represented as a bag-of-words and we compute the centroid of the embeddings of the words in each bag to represent that review. Next, we train a binary logistic regression classifier with a cross-validated $\ell _{2}$ regulariser using the train portion of each dataset, and evaluate the classification accuracy using the test portion of the dataset.
Baselines
A simple baseline method for combining pre-trained word embeddings is to concatenate the embedding vectors for a word $w$ to produce a meta-embedding for $w$ . Each source embedding of $w$ is $\ell _{2}$ normalised prior to concatenation such that each source embedding contributes equally (a value in $[-1,1]$ ) when measuring the word similarity using the dot product. As also observed by Yin:ACL:2016 we found that CONC performs poorly without emphasising GloVe and CBOW by a constant factor (which is set to 8 using MC as a validation dataset) when used in conjunction with HLBL, Huang, and CW source embeddings.
Interestingly, concatenation can be seen as a special case in the reconstruction step described in Section "Nearest Neighbour Reconstruction" . To see this, let us denote the concatenation of column vectors $\vec{v}^{(1)}$ and $\vec{v}^{(2)}$ by $\vec{x} = (\vec{v}^{(1)}; \vec{v}^{(2)})$ , and $\vec{u}^{(1)}$ and $\vec{u}^{(2)}$ by $\vec{y} = (\vec{u}^{(1)}; \vec{u}^{(2)})$ , where $\vec{x}, \vec{y} \in ^{d_{1} + d_{2}}$ . Then, the reconstruction error defined by ( "Nearest Neighbour Reconstruction" ) can be written as follows:
$$\Phi ({W}) = \sum _{v \in } {\vec{x} - \sum _{u \in (v)} w_{vu}\vec{y} }_{2}^{2}$$   (Eq. 23)
Here, the vocabulary $$ is constrained to the intersection $_{} \cap _{}$ because concatenation is not defined for missing words in a source embedding. Alternatively, one could use zero-vectors for missing words or (better) predict the word embeddings for missing words prior to concatenation. However, we consider such extensions to be beyond the simple concatenation baseline we consider here. On the other hand, the common neighbourhood $(v)$ in ( 23 ) can be obtained by either limiting $(v)$ to $_{1}(v) \cap _{2}(v)$ or, by extending the neighbourhoods to the entire vocabulary ( $(v) = $ ). ( 23 ) shows that under those neighbourhood constraints, the first step in our proposed method can be seen as reconstructing the neighbourhood of the concatenated space. The second step would then find meta-embeddings that preserve the locally linear structure in the concatenated space.
One drawback of concatenation is that it increases the dimensionality of the meta-embeddings compared to the source-embeddings, which might be problematic when storing or processing the meta-embeddings (for example, for the five source embeddings we use here $d_{} = 100 + 50 + 300 + 200 + 300 = 950$ ).
We create an $N \times 950$ matrix ${C}$ by arranging the CONC vectors for the union of all source embedding vocabularies. For words that are missing in a particular source embedding, we assign zero vectors of that source embedding's dimensionality. Next, we perform SVD on ${C} = {U} {D} {V}, where $ U $ and $ V $ are unitary matrices and the diagonal matrix $ D $ contains the singular values of $ C $. We then select the $ d $ largest left singular vectors from $ U $ to create a $ d $ dimensional embeddings for the $ N ${C}$0 d = 300 ${C}$1 U ${C}$2
Meta-Embedding Results
Using the MC dataset, we find the best values for the neighbourhood size $n = 1200$ and dimensionality $d_{} = 300$ for the Proposed method. We plan to publicly release our meta-embeddings on acceptance of the paper.
We summarise the experimental results for different methods on different tasks/datasets in Table 1 . In Table 1 , rows 1-5 show the performance of the individual source embeddings. Next, we perform ablation tests (rows 6-20) where we hold-out one source embedding and use the other four with each meta-embedding method. We evaluate statistical significance against best performing individual source embedding on each dataset. For the semantic similarity benchmarks we use Fisher transformation to compute $p < 0.05$ confidence intervals for Spearman correlation coefficients. In all other (classification) datasets, we used Clopper-Pearson binomial exact confidence intervals at $p < 0.05$ .
Among the individual source embeddings, we see that GloVe and CBOW stand out as the two best embeddings. This observation is further confirmed from ablation results, where the removal of GloVe or CBOW often results in a decrease in performance. Performing SVD (rows 11-15) after concatenating, does not always result in an improvement. SVD is a global projection that reduces the dimensionality of the meta-embeddings created via concatenation. This result indicates that different source embeddings might require different levels of dimensionality reductions, and applying a single global projection does not always guarantee improvements. Ensemble methods that use all five source embeddings are shown in rows 21-25. 1TON and 1TON+ are proposed by Yin:ACL:2016, and were detailed in Section "Nearest Neighbour Reconstruction" . Because they did not evaluate on all tasks that we do here, to conduct a fair and consistent evaluation we used their publicly available meta-embeddings without retraining by ourselves.
Overall, from Table 1 , we see that the Proposed method (row 25) obtains the best performance in all tasks/datasets. In 6 out of 12 benchmarks, this improvement is statistically significant over the best single source embedding. Moreover, in the MEN dataset (the largest among the semantic similarity benchmarks compared in Table 1 with 3000 word-pairs), and the Google dataset, the improvements of the Proposed method over the previously proposed 1TON and 1TON+ are statistically significant.
The ablation results for the Proposed method show that, although different source embeddings are important to different degrees, by using all source embeddings we can obtain the best results. Different source embeddings are trained from different resources and by optimising different objectives. Therefore, for different words, the local neighbours predicted by different source embeddings will be complementary. Unlike the other methods, the Proposed method never compares different source embeddings' vectors directly, but only via the neighbourhood reconstruction weights. Consequently, the Proposed method is unaffected by relative weighting of source embeddings. In contrast, the CONC is highly sensitive against the weighting. In fact, we confirmed that the performance scores of the CONC method were decreased by 3–10 points when we did not do the weight tuning described in Section "Evaluation Tasks" . The unnecessity of the weight tuning is thus a clear advantage of the Proposed method.
To investigate the effect of the dimensionality $d^{}$ on the meta-embeddings learnt by the proposed method, in fig:k, we fix the neighbourhood size $N = 1200$ and measure the performance on semantic similarity measurement tasks when varying $d^{}$ . Overall, we see that the performance peaks around $d^{} = 300$ . Such behaviour can be explained by the fact that smaller $d^{}$ dimensions are unable to preserve information contained in the source embeddings, whereas increasing $d^{}$ beyond the rank of the weight matrix ${W}$ is likely to generate noisy eigenvectors.
In fig:n, we study the effect of increasing the neighbourhood size $n$ equally for all words in all source embeddings, while fixing the dimensionality of the meta-embedding $d^{} = 300$ . Initially, performance increases with the neighbourhood size and then saturates. This implies that in practice a small local neighbourhood is adequate to capture the differences in source embeddings.
Complementarity of Resources
We have shown empirically in Section "Meta-Embedding Results" that using the proposed method it is possible to obtain superior meta-embeddings from a diverse set of source embeddings. One important scenario where meta-embedding could be potentially useful is when the source embeddings are trained on different complementary resources, where each resource share little common vocabulary. For example, one source embedding might have been trained on Wikipedia whereas a second source embedding might have been trained on tweets.
To evaluate the effectiveness of the proposed meta-embedding learning method under such settings, we design the following experiment. We select MEN dataset, the largest among all semantic similarity benchmarks, which contains 751 unique words in 3000 human-rated word-pairs for semantic similarity. Next, we randomly split the set of words into two sets with different overlap ratios. We then select sentences from 2017 January dump of Wikipedia that contains words from only one of the two sets. We create two corpora of roughly equal number of sentences via this procedure for different overlap ratios. We train skip-gram with negative sampling (SGNS) () on one corpus to create source embedding $S_{1}$ and GloVe () on the other corpus to create source embedding $S_{2}$ . Finally, we use the proposed method to meta-embed $S_{1}$ and $S_{2}$ .
Figure 2 shows the Spearman correlation between the human similarity ratings and cosine similarities computed using the word embeddings on the MEN dataset for $S_{1}$ , $S_{2}$ and their meta-embeddings created using the proposed method (Meta) and concatenation baseline (CONC). From Figure 2 , we see that the meta embeddings obtain the best performance across all overlap ratios. The improvements are larger when the overlap between the corpora is smaller, and diminishes when the two corpora becomes identical. This result shows that our proposed meta-embedding learning method captures the complementary information available in different source embeddings to create more accurate word embeddings. Moreover, it shows that by considering the local neighbourhoods in each of the source embeddings separately, we can obviate the need to predict embeddings for missing words in a particular source embedding, which was a limitation in the method proposed by Yin:ACL:2016.
Conclusion
We proposed an unsupervised locally linear method for learning meta-embeddings from a given set of pre-trained source embeddings. Experiments on several NLP tasks show the accuracy of the proposed method, which outperforms previously proposed meta-embedding learning methods on multiple benchmark datasets. In future, we plan to extend the proposed method to learn cross-lingual meta-embeddings by incorporating both cross-lingual as well as monolingual information.

Question: What is the introduced meta-embedding method introduced in this paper?
Answer: proposed method comprises of two steps: a neighbourhood reconstruction step (Section "Nearest Neighbour Reconstruction" ), and a projection step (Section "Projection to Meta-Embedding Space" ). In the reconstruction step, we represent the embedding of a word by the linearly weighted combination of the embeddings of its nearest neighbours in each source embedding space.