Introduction
Word embeddings have been used to improve the performance of many NLP tasks including language modelling BIBREF1 , machine translation BIBREF2 , and sentiment analysis BIBREF3 . The broad applicability of word embeddings to NLP implies that improvements to their quality will likely have widespread benefits for the field.
The word embedding problem is to learn a mapping INLINEFORM0 ( INLINEFORM1 100-300 in most applications) that encodes meaningful semantic and/or syntactic information. For instance, in many word embeddings, INLINEFORM2 car INLINEFORM3 truck INLINEFORM4 , since the words are semantically similar.
More complex relationships than similarity can also be encoded in word embeddings. For example, we can answer analogy queries of the form INLINEFORM0 ? using simple arithmetic in many state-of-the-art embeddings BIBREF4 . The answer to bed INLINEFORM1 sleep INLINEFORM2 chair INLINEFORM3 INLINEFORM4 is given by the word whose vector representation is closest to INLINEFORM5 sleep INLINEFORM6 bed INLINEFORM7 chair INLINEFORM8 ( INLINEFORM9 sit INLINEFORM10 ). Other embeddings may encode such information in a nonlinear way BIBREF5 .
BIBREF4 demonstrates the additive compositionality of their word2vec vectors: one can sum vectors produced by their embedding to compute vectors for certain phrases rather than just vectors for words. Later in this paper, we will show that our embeddings naturally give rise to a form of multiplicative compositionality that has not yet been explored in the literature.
Almost all recent word embeddings rely on the distributional hypothesis BIBREF6 , which states that a word's meaning can be inferred from the words that tend to surround it. To utilize the distributional hypothesis, many embeddings are given by a low-rank factor of a matrix derived from co-occurrences in a large unsupervised corpus, see BIBREF7 , BIBREF8 , BIBREF9 and BIBREF10 .
Approaches that rely on matrix factorization only utilize pairwise co-occurrence information in the corpus. We aim to extend this approach by creating word embeddings given by factors of tensors containing higher order co-occurrence data.
Related work
Some common word embeddings related to co-occurrence based matrix factorization include GloVe BIBREF7 , word2vec BIBREF9 , LexVec BIBREF10 , and NNSE BIBREF8 . In contrast, our work studies word embeddings given by factorization of tensors. An overview of tensor factorization methods is given in BIBREF11 .
Our work uses factorization of symmetric nonnegative tensors, which has been studied in the past BIBREF12 , BIBREF13 . In general, factorization of tensors has been applied to NLP in BIBREF14 and factorization of nonnegative tensors BIBREF15 . Recently, factorization of symmetric tensors has been used to create a generic word embedding BIBREF16 but the idea was not explored extensively. Our work studies this idea in much greater detail, fully demonstrating the viability of tensor factorization as a technique for training word embeddings.
Composition of word vectors to create novel representations has been studied in depth, including additive, multiplicative, and tensor-based methods BIBREF17 , BIBREF18 . Typically, composition is used to create vectors that represent phrases or sentences. Our work, instead, shows that pairs of word vectors can be composed multiplicatively to create different vector representations for the various meanings of a single polysemous word.
Notation
Throughout this paper we will write scalars in lowercase italics INLINEFORM0 , vectors in lowercase bold letters INLINEFORM1 , matrices with uppercase bold letters INLINEFORM2 , and tensors (of order INLINEFORM3 ) with Euler script notation INLINEFORM4 , as is standard in the literature.
Pointwise Mutual Information
Pointwise mutual information (PMI) is a useful property in NLP that quantifies the likelihood that two words co-occur BIBREF9 . It is defined as: INLINEFORM0
where INLINEFORM0 is the probability that INLINEFORM1 and INLINEFORM2 occur together in a given fixed-length context window in the corpus, irrespective of order.
It is often useful to consider the positive PMI (PPMI), defined as: INLINEFORM0
since negative PMI values have little grounded interpretation BIBREF19 , BIBREF9 , BIBREF15 .
Given an indexed vocabulary INLINEFORM0 , one can construct a INLINEFORM1 PPMI matrix INLINEFORM2 where INLINEFORM3 . Many existing word embedding techniques involve factorizing this PPMI matrix BIBREF9 , BIBREF8 , BIBREF10 .
PMI can be generalized to INLINEFORM0 variables. While there are many ways to do so BIBREF20 , in this paper we use the form defined by: INLINEFORM1
where INLINEFORM0 is the probability that all of INLINEFORM1 occur together in a given fixed-length context window in the corpus, irrespective of their order.
In this paper we study 3-way PPMI tensors INLINEFORM0 , where INLINEFORM1 , as this is the natural higher-order generalization of the PPMI matrix. We leave the study of creating word embeddings with INLINEFORM2 -dimensional PPMI tensors ( INLINEFORM3 ) to future work.
Tensor factorization
Just as the rank- INLINEFORM0 matrix decomposition is defined to be the product of two factor matrices ( INLINEFORM1 ), the canonical rank- INLINEFORM2 tensor decomposition for a third order tensor is defined to be the product of three factor matrices BIBREF11 : DISPLAYFORM0
where INLINEFORM0 is the outer product: INLINEFORM1 . This is also commonly referred to as the rank-R CP Decomposition. Elementwise, this is written as: INLINEFORM2
where INLINEFORM0 is elementwise vector multiplication and INLINEFORM1 is the INLINEFORM2 row of INLINEFORM3 . In our later section on multiplicative compositionality, we will see this formulation gives rise to a meaningful interpretation of the elementwise product between vectors in our word embeddings.
Symmetric CP Decomposition. In this paper, we will consider symmetric CP decomposition of nonnegative tensors BIBREF21 , BIBREF11 . Since our INLINEFORM0 -way PPMI is nonnegative and invariant under permutation, the PPMI tensor INLINEFORM1 is nonnegative and supersymmetric, i.e. INLINEFORM2 for any permutation INLINEFORM3 .
In the symmetric CP decomposition, instead of factorizing INLINEFORM0 , we factorize INLINEFORM1 as the triple product of a single factor matrix INLINEFORM2 such that INLINEFORM3
In this formulation, we use INLINEFORM0 to be the word embedding so the vector for INLINEFORM1 is the INLINEFORM2 row of INLINEFORM3 similar to the formulations in BIBREF9 , BIBREF8 , BIBREF7 .
It is known that the optimal rank- INLINEFORM0 CP decomposition exists for symmetric nonnegative tensors such as the PPMI tensor BIBREF21 . However, finding such a decomposition is NP hard in general BIBREF22 so we must consider approximate methods.
In this work, we only consider the symmetric CP decomposition, leaving the study of other tensor decompositions (such as the Tensor Train or HOSVD BIBREF23 , BIBREF11 ) to future work.
Computing the Symmetric CP Decomposition
The INLINEFORM0 size of the third order PPMI tensor presents a number of computational challenges. In practice, INLINEFORM1 can vary from INLINEFORM2 to INLINEFORM3 , resulting in a tensor whose naive representation requires at least INLINEFORM4 bytes = 4 TB of floats. Even the sparse representation of the tensor takes up such a large fraction of memory that standard algorithms such as successive rank-1 approximation BIBREF12 , BIBREF24 and alternating least-squares BIBREF11 are infeasible for our uses. Thus, in this paper we will consider a stochastic online formulation similar to that of BIBREF25 .
We optimize the CP decomposition in an online fashion, using small random subsets INLINEFORM0 of the nonzero tensor entries to update the decomposition at time INLINEFORM1 . In this minibatch setting, we optimize the decomposition based on the current minibatch and the previous decomposition at time INLINEFORM2 . To update INLINEFORM3 (and thus the symmetric decomposition), we first define a decomposition loss INLINEFORM4 and minimize this loss with respect to INLINEFORM5 using Adam BIBREF26 .
At each time INLINEFORM0 , we take INLINEFORM1 to be all co-occurrence triples (weighted by PPMI) in a fixed number of sentences (around 1,000) from the corpus. We continue training until we have depleted the entire corpus.
For INLINEFORM0 to accurately model INLINEFORM1 , we also include a certain proportion of elements with zero PPMI (or “negative samples”) in INLINEFORM2 , similar to that of BIBREF10 . We use an empirically found proportion of negative samples for training, and leave discovery of the optimal negative sample proportion to future work.
Word Embedding Proposals
CP-S. The first embedding we propose is based on symmetic CP decomposition of the PPMI tensor INLINEFORM0 as discussed in the mathematical preliminaries section. The optimal setting for the word embedding INLINEFORM1 is: INLINEFORM2
Since we cannot feasibly compute this exactly, we minimize the loss function defined as the squared error between the values in INLINEFORM0 and their predicted values: INLINEFORM1
using the techniques discussed in the previous section.
JCP-S. A potential problem with CP-S is that it is only trained on third order information. To rectify this issue, we propose a novel joint tensor factorization problem we call Joint Symmetric Rank- INLINEFORM0 CP Decomposition. In this problem, the input is the fixed rank INLINEFORM1 and a list of supersymmetric tensors INLINEFORM2 of different orders but whose axis lengths all equal INLINEFORM3 . Each tensor INLINEFORM4 is to be factorized via rank- INLINEFORM5 symmetric CP decomposition using a single INLINEFORM6 factor matrix INLINEFORM7 .
To produce a solution, we first define the loss at time INLINEFORM0 to be the sum of the reconstruction losses of each different tensor: INLINEFORM1
where INLINEFORM0 is an INLINEFORM1 -dimensional supersymmetric PPMI tensor. We then minimize the loss with respect to INLINEFORM2 . Since we are using at most third order tensors in this work, we assign our word embedding INLINEFORM3 to be: INLINEFORM4
This problem is a specific instance of Coupled Tensor Decomposition, which has been studied in the past BIBREF27 , BIBREF28 . In this problem, the goal is to factorize multiple tensors using at least one factor matrix in common. A similar formulation to our problem can be found in BIBREF29 , which studies blind source separation using the algebraic geometric aspects of jointly factorizing numerous supersymmetric tensors (to unknown rank). In contrast to our work, they outline some generic rank properties of such a decomposition rather than attacking the problem numerically. Also, in our formulation the rank is fixed and an approximate solution must be found. Exploring the connection between the theoretical aspects of joint decomposition and quality of word embeddings would be an interesting avenue for future work.
To the best of our knowledge this is the first study of Joint Symmetric Rank- INLINEFORM0 CP Decomposition.
Shifted PMI
In the same way BIBREF9 considers factorization of positive shifted PMI matrices, we consider factorization of positive shifted PMI tensors INLINEFORM0 , where INLINEFORM1 for some constant shift INLINEFORM2 . We empirically found that different levels of shifting resulted in different qualities of word embeddings – the best shift we found for CP-S was a shift of INLINEFORM3 , whereas any nonzero shift for JCP-S resulted in a worse embedding across the board. When we discuss evaluation we report the results given by factorization of the PPMI tensors shifted by the best value we found for each specific embedding.
Computational notes
When considering going from two dimensions to three, it is perhaps necessary to discuss the computational issues in such a problem size increase. However, it should be noted that the creation of pre-trained embeddings can be seen as a pre-processing step for many future NLP tasks, so if the training can be completed once, it can be used forever thereafter without having to take training time into account. Despite this, we found that the training of our embeddings was not considerably slower than the training of order-2 equivalents such as SGNS. Explicitly, our GPU trained CBOW vectors (using the experimental settings found below) in 3568 seconds, whereas training CP-S and JCP-S took 6786 and 8686 seconds respectively.
Evaluation
In this section we present a quantitative evaluation comparing our embeddings to an informationless embedding and two strong baselines. Our baselines are:
For a fair comparison, we trained each model on the same corpus of 10 million sentences gathered from Wikipedia. We removed stopwords and words appearing fewer than 2,000 times (130 million tokens total) to reduce noise and uninformative words. Our word2vec and NNSE baselines were trained using the recommended hyperparameters from their original publications, and all optimizers were using using the default settings. Hyperparameters are always consistent across evaluations.
Because of the dataset size, the results shown should be considered a proof of concept rather than an objective comparison to state-of-the-art pre-trained embeddings. Due to the natural computational challenges arising from working with tensors, we leave creation of a full-scale production ready embedding based on tensor factorization to future work.
As is common in the literature BIBREF4 , BIBREF8 , we use 300-dimensional vectors for our embeddings and all word vectors are normalized to unit length prior to evaluation.
Quantitative tasks
Outlier Detection. The Outlier Detection task BIBREF0 is to determine which word in a list INLINEFORM0 of INLINEFORM1 words is unrelated to the other INLINEFORM2 which were chosen to be related. For each INLINEFORM3 , one can compute its compactness score INLINEFORM4 , which is the compactness of INLINEFORM5 . INLINEFORM6 is explicitly computed as the mean similarity of all word pairs INLINEFORM7 . The predicted outlier is INLINEFORM8 , as the INLINEFORM9 related words should form a compact cluster with high mean similarity.
We use the WikiSem500 dataset BIBREF30 which includes sets of INLINEFORM0 words per group gathered based on semantic similarity. Thus, performance on this task is correlated with the amount of semantic information encoded in a word embedding. Performance on this dataset was shown to be well-correlated with performance at the common NLP task of sentiment analysis BIBREF30 .
The two metrics associated with this task are accuracy and Outlier Position Percentage (OPP). Accuracy is the fraction of cases in which the true outlier correctly had the highest compactness score. OPP measures how close the true outlier was to having the highest compactness score, rewarding embeddings more for predicting the outlier to be in 2nd place rather than INLINEFORM0 when sorting the words by their compactness score INLINEFORM1 .
3-way Outlier Detection. As our tensor-based embeddings encode higher order relationships between words, we introduce a new way to compute INLINEFORM0 based on groups of 3 words rather than pairs of words. We define the compactness score for a word INLINEFORM1 to be: INLINEFORM2
where INLINEFORM0 denotes similarity between a group of 3 vectors. INLINEFORM1 is defined as: INLINEFORM2
We call this evaluation method OD3.
The purpose of OD3 is to evaluate the extent to which an embedding captures 3rd order relationships between words. As we will see in the results of our quantitative experiments, our tensor methods outperform the baselines on OD3, which validates our approach.
This approach can easily be generalized to OD INLINEFORM0 INLINEFORM1 , but again we leave the study of higher order relationships to future work.
Simple supervised tasks. BIBREF5 points out that the primary application of word embeddings is transfer learning to NLP tasks. They argue that to evaluate an embedding's ability to transfer information to a relevant task, one must measure the embedding's accessibility of information for actual downstream tasks. To do so, one must cite the performance of simple supervised tasks as training set size increases, which is commonly done in transfer learning evaluation BIBREF5 . If an algorithm using a word embedding performs well with just a small amount of training data, then the information encoded in the embedding is easily accessible.
The simple supervised downstream tasks we use to evaluate the embeddings are as follows:
Supervised Analogy Recovery. We consider the task of solving queries of the form a : b :: c : ? using a simple neural network as suggested in BIBREF5 . The analogy dataset we use is from the Google analogy testbed BIBREF4 .
Sentiment analysis. We also consider sentiment analysis as described by BIBREF31 . We use the suggested Large Movie Review dataset BIBREF32 , containing 50,000 movie reviews.
All code is implemented using scikit-learn or TensorFlow and uses the suggested train/test split.
Word similarity. To standardize our evaluation methodology, we evaluate the embeddings using word similarity on the common MEN and MTurk datasets BIBREF33 , BIBREF34 . For an overview of word similarity evaluation, see BIBREF31 .
Quantitative results
Outlier Detection results. The results are shown in Table TABREF20 . The first thing to note is that CP-S outperforms the other methods across each Outlier Detection metric. Since the WikiSem500 dataset is semantically focused, performance at this task demonstrates the quality of semantic information encoded in our embeddings.
On OD2, the baselines perform more competitively with our CP Decomposition based models, but when OD3 is considered our methods clearly excel. Since the tensor-based methods are trained directly on third order information and perform much better at OD3, we see that OD3 scores reflect the amount of third order information in a word embedding. This is a validation of OD3, as our 3rd order embeddings would naturally out perform 2nd order embeddings at a task that requires third order information. Still, the superiority of our tensor-based embeddings at OD2 demonstrates the quality of the semantic information they encode.
Supervised analogy results. The results are shown in Figure FIGREF18 . At the supervised semantic analogy task, CP-S vastly outperforms the baselines at all levels of training data, further signifying the amount of semantic information encoded by this embedding technique.
Also, when only 10% of the training data is presented, our tensor methods are the only ones that attain nonzero performance – even in such a limited data setting, use of CP-S's vectors results in nearly 40% accuracy. This phenomenon is also observed in the syntactic analogy tasks: our embeddings consistently outperform the others until 100% of the training data is presented. These two observations demonstrate the accessibility of the information encoded in our word embeddings. We can thus conclude that this relational information encoded in the tensor-based embeddings is more easily accessible than that of CBOW and NNSE. Thus, our methods would likely be better suited for transfer learning to actual NLP tasks, particularly those in data-sparse settings.
Sentiment analysis results. The results are shown in Figure FIGREF19 . In this task, JCP-S is the dominant method across all levels of training data, but the difference is more obvious when training data is limited. This again indicates that for this specific task the information encoded by our tensor-based methods is more readily available as that of the baselines. It is thus evident that exploiting both second and third order co-occurrence data leads to higher quality semantic information being encoded in the embedding. At this point it is not clear why JCP-S so vastly outperforms CP-S at this task, but its superiority to the other strong baselines demonstrates the quality of information encoded by JCP-S. This discrepancy is also illustrative of the fact that there is no single “best word embedding” BIBREF5 – different embeddings encode different types of information, and thus should be used where they shine rather than for every NLP task.
Word Similarity results.
We show the results in Table TABREF21 . As we can see, our embeddings very clearly outperform the random embedding at this task. They even outperform CBOW on both of these datasets. It is worth including these results as the word similarity task is a very common way of evaluating embedding quality in the literature. However, due to the many intrinsic problems with evaluating word embeddings using word similarity BIBREF35 , we do not discuss this further.
Multiplicative Compositionality
We find that even though they are not explicitly trained to do so, our tensor-based embeddings capture polysemy information naturally through multiplicative compositionality. We demonstrate this property qualitatively and provide proper motivation for it, leaving automated utilization to future work.
In our tensor-based embeddings, we found that one can create a vector that represents a word INLINEFORM0 in the context of another word INLINEFORM1 by taking the elementwise product INLINEFORM2 . We call INLINEFORM3 a “meaning vector” for the polysemous word INLINEFORM4 .
For example, consider the word star, which can denote a lead performer or a celestial body. We can create a vector for star in the “lead performer” sense by taking the elementwise product INLINEFORM0 . This produces a vector that lies near vectors for words related to lead performers and far from those related to star's other senses.
To motivate why this works, recall that the values in a third order PPMI tensor INLINEFORM0 are given by: INLINEFORM1
where INLINEFORM0 is the word vector for INLINEFORM1 . If words INLINEFORM2 have a high PPMI, then INLINEFORM3 will also be high, meaning INLINEFORM4 will be close to INLINEFORM5 in the vector space by cosine similarity.
For example, even though galaxy is likely to appear in the context of the word star in in the “celestial body” sense, INLINEFORM0 PPMI(star, actor, galaxy) is low whereas INLINEFORM1 PPMI(star, actor, drama) is high. Thus , INLINEFORM2 represents the meaning of star in the “lead performer” sense.
In Table TABREF22 we present the nearest neighbors of multiplicative and additive composed vectors for a variety of polysemous words. As we can see, the words corresponding to the nearest neighbors of the composed vectors for our tensor methods are semantically related to the intended sense both for multiplicative and additive composition. In contrast, for CBOW, only additive composition yields vectors whose nearest neighbors are semantically related to the intended sense. Thus, our embeddings can produce complementary sets of polysemous word representations that are qualitatively valid whereas CBOW (seemingly) only guarantees meaningful additive compositionality. We leave automated usage of this property to future work.
Conclusion
Our key contributions are as follows:
Tensor factorization appears to be a highly applicable and effective tool for learning word embeddings, with many areas of potential future work. Leveraging higher order data in training word embeddings is useful for encoding new types of information and semantic relationships compared to models that are trained using only pairwise data. This indicates that such techniques will prove useful for training word embeddings to be used in downstream NLP tasks.

Question: Do they test their word embeddings on downstream tasks?
Answer: Yes

Question: What are the main disadvantages of their proposed word embeddings?
Answer: Unanswerable

Question: What dimensions of word embeddings do they produce using factorization?
Answer: 300-dimensional vectors

Question: On which dataset(s) do they compute their word embeddings?
Answer: 10 million sentences gathered from Wikipedia

Question: Do they measure computation time of their factorizations compared to other word embeddings?
Answer: Yes