Quantitative tasks	Outlier Detection. The Outlier Detection task  is to determine which word in a list INLINEFORM0 of INLINEFORM1 words is unrelated to the other INLINEFORM2 which were chosen to be related. For each INLINEFORM3 , one can compute its compactness score INLINEFORM4 , which is the compactness of INLINEFORM5 . INLINEFORM6 is explicitly computed as the mean similarity of all word pairs INLINEFORM7 . The predicted outlier is INLINEFORM8 , as the INLINEFORM9 related words should form a compact cluster with high mean similarity. We use the WikiSem500 dataset  which includes sets of INLINEFORM0 words per group gathered based on semantic similarity. Thus, performance on this task is correlated with the amount of semantic information encoded in a word embedding. Performance on this dataset was shown to be well-correlated with performance at the common NLP task of sentiment analysis  . The two metrics associated with this task are accuracy and Outlier Position Percentage (OPP). Accuracy is the fraction of cases in which the true outlier correctly had the highest compactness score.

They argue that to evaluate an embedding's ability to transfer information to a relevant task, one must measure the embedding's accessibility of information for actual downstream tasks. To do so, one must cite the performance of simple supervised tasks as training set size increases, which is commonly done in transfer learning evaluation  . If an algorithm using a word embedding performs well with just a small amount of training data, then the information encoded in the embedding is easily accessible. The simple supervised downstream tasks we use to evaluate the embeddings are as follows: Supervised Analogy Recovery. We consider the task of solving queries of the form a : b :: c : ? using a simple neural network as suggested in  . The analogy dataset we use is from the Google analogy testbed  . Sentiment analysis. We also consider sentiment analysis as described by  . We use the suggested Large Movie Review dataset  , containing 50,000 movie reviews. All code is implemented using scikit-learn or TensorFlow and uses the suggested train/test split. Word similarity. To standardize our evaluation methodology, we evaluate the embeddings using word similarity on the common MEN and MTurk datasets  ,  . For an overview of word similarity evaluation, see  .

At this point it is not clear why JCP-S so vastly outperforms CP-S at this task, but its superiority to the other strong baselines demonstrates the quality of information encoded by JCP-S. This discrepancy is also illustrative of the fact that there is no single “best word embedding”  – different embeddings encode different types of information, and thus should be used where they shine rather than for every NLP task. Word Similarity results. We show the results in Table TABREF21 . As we can see, our embeddings very clearly outperform the random embedding at this task. They even outperform CBOW on both of these datasets. It is worth including these results as the word similarity task is a very common way of evaluating embedding quality in the literature. However, due to the many intrinsic problems with evaluating word embeddings using word similarity  , we do not discuss this further.

OPP measures how close the true outlier was to having the highest compactness score, rewarding embeddings more for predicting the outlier to be in 2nd place rather than INLINEFORM0 when sorting the words by their compactness score INLINEFORM1 . 3-way Outlier Detection. As our tensor-based embeddings encode higher order relationships between words, we introduce a new way to compute INLINEFORM0 based on groups of 3 words rather than pairs of words. We define the compactness score for a word INLINEFORM1 to be: INLINEFORM2  where INLINEFORM0 denotes similarity between a group of 3 vectors. INLINEFORM1 is defined as: INLINEFORM2  We call this evaluation method OD3. The purpose of OD3 is to evaluate the extent to which an embedding captures 3rd order relationships between words. As we will see in the results of our quantitative experiments, our tensor methods outperform the baselines on OD3, which validates our approach. This approach can easily be generalized to OD INLINEFORM0 INLINEFORM1 , but again we leave the study of higher order relationships to future work. Simple supervised tasks.  points out that the primary application of word embeddings is transfer learning to NLP tasks.

demonstrates the additive compositionality of their word2vec vectors: one can sum vectors produced by their embedding to compute vectors for certain phrases rather than just vectors for words. Later in this paper, we will show that our embeddings naturally give rise to a form of multiplicative compositionality that has not yet been explored in the literature. Almost all recent word embeddings rely on the distributional hypothesis  , which states that a word's meaning can be inferred from the words that tend to surround it. To utilize the distributional hypothesis, many embeddings are given by a low-rank factor of a matrix derived from co-occurrences in a large unsupervised corpus, see  ,  ,  and  . Approaches that rely on matrix factorization only utilize pairwise co-occurrence information in the corpus. We aim to extend this approach by creating word embeddings given by factors of tensors containing higher order co-occurrence data.

Multiplicative Compositionality	We find that even though they are not explicitly trained to do so, our tensor-based embeddings capture polysemy information naturally through multiplicative compositionality. We demonstrate this property qualitatively and provide proper motivation for it, leaving automated utilization to future work. In our tensor-based embeddings, we found that one can create a vector that represents a word INLINEFORM0 in the context of another word INLINEFORM1 by taking the elementwise product INLINEFORM2 . We call INLINEFORM3 a “meaning vector” for the polysemous word INLINEFORM4 . For example, consider the word star, which can denote a lead performer or a celestial body. We can create a vector for star in the “lead performer” sense by taking the elementwise product INLINEFORM0 . This produces a vector that lies near vectors for words related to lead performers and far from those related to star's other senses. To motivate why this works, recall that the values in a third order PPMI tensor INLINEFORM0 are given by: INLINEFORM1  where INLINEFORM0 is the word vector for INLINEFORM1 .

Evaluation	In this section we present a quantitative evaluation comparing our embeddings to an informationless embedding and two strong baselines. Our baselines are: For a fair comparison, we trained each model on the same corpus of 10 million sentences gathered from Wikipedia. We removed stopwords and words appearing fewer than 2,000 times (130 million tokens total) to reduce noise and uninformative words. Our word2vec and NNSE baselines were trained using the recommended hyperparameters from their original publications, and all optimizers were using using the default settings. Hyperparameters are always consistent across evaluations. Because of the dataset size, the results shown should be considered a proof of concept rather than an objective comparison to state-of-the-art pre-trained embeddings. Due to the natural computational challenges arising from working with tensors, we leave creation of a full-scale production ready embedding based on tensor factorization to future work. As is common in the literature  ,  , we use 300-dimensional vectors for our embeddings and all word vectors are normalized to unit length prior to evaluation.

Experimentally, we show that all of our word embeddings either outperform or are competitive with state-of-the-art baselines commonly used today on a variety of recent datasets. Suggested applications of tensor factorization-based word embeddings are given, and all source code and pre-trained vectors are publicly available online.

To produce a solution, we first define the loss at time INLINEFORM0 to be the sum of the reconstruction losses of each different tensor: INLINEFORM1  where INLINEFORM0 is an INLINEFORM1 -dimensional supersymmetric PPMI tensor. We then minimize the loss with respect to INLINEFORM2 . Since we are using at most third order tensors in this work, we assign our word embedding INLINEFORM3 to be: INLINEFORM4  This problem is a specific instance of Coupled Tensor Decomposition, which has been studied in the past  ,  . In this problem, the goal is to factorize multiple tensors using at least one factor matrix in common. A similar formulation to our problem can be found in  , which studies blind source separation using the algebraic geometric aspects of jointly factorizing numerous supersymmetric tensors (to unknown rank). In contrast to our work, they outline some generic rank properties of such a decomposition rather than attacking the problem numerically. Also, in our formulation the rank is fixed and an approximate solution must be found. Exploring the connection between the theoretical aspects of joint decomposition and quality of word embeddings would be an interesting avenue for future work.

Computational notes	When considering going from two dimensions to three, it is perhaps necessary to discuss the computational issues in such a problem size increase. However, it should be noted that the creation of pre-trained embeddings can be seen as a pre-processing step for many future NLP tasks, so if the training can be completed once, it can be used forever thereafter without having to take training time into account. Despite this, we found that the training of our embeddings was not considerably slower than the training of order-2 equivalents such as SGNS. Explicitly, our GPU trained CBOW vectors (using the experimental settings found below) in 3568 seconds, whereas training CP-S and JCP-S took 6786 and 8686 seconds respectively.

Word Embeddings via Tensor Factorization	Most popular word embedding techniques involve implicit or explicit factorization of a word co-occurrence based matrix into low rank factors. In this paper, we aim to generalize this trend by using numerical methods to factor higher-order word co-occurrence based arrays, or \textit{tensors}. We present four word embeddings using tensor factorization and analyze their advantages and disadvantages. One of our main contributions is a novel joint symmetric tensor factorization technique related to the idea of coupled tensor factorization. We show that embeddings based on tensor factorization can be used to discern the various meanings of polysemous words without being explicitly trained to do so, and motivate the intuition behind why this works in a way that doesn't with existing methods. We also modify an existing word embedding evaluation metric known as Outlier Detection [Camacho-Collados and Navigli, 2016] to evaluate the quality of the order-$N$ relations that a word embedding captures, and show that tensor-based methods outperform existing matrix-based methods at this task.

Pointwise Mutual Information	Pointwise mutual information (PMI) is a useful property in NLP that quantifies the likelihood that two words co-occur  . It is defined as: INLINEFORM0  where INLINEFORM0 is the probability that INLINEFORM1 and INLINEFORM2 occur together in a given fixed-length context window in the corpus, irrespective of order. It is often useful to consider the positive PMI (PPMI), defined as: INLINEFORM0  since negative PMI values have little grounded interpretation  ,  ,  . Given an indexed vocabulary INLINEFORM0 , one can construct a INLINEFORM1 PPMI matrix INLINEFORM2 where INLINEFORM3 . Many existing word embedding techniques involve factorizing this PPMI matrix  ,  ,  . PMI can be generalized to INLINEFORM0 variables. While there are many ways to do so  , in this paper we use the form defined by: INLINEFORM1   where INLINEFORM0 is the probability that all of INLINEFORM1 occur together in a given fixed-length context window in the corpus, irrespective of their order.

Related work	Some common word embeddings related to co-occurrence based matrix factorization include GloVe  , word2vec  , LexVec  , and NNSE  . In contrast, our work studies word embeddings given by factorization of tensors. An overview of tensor factorization methods is given in  . Our work uses factorization of symmetric nonnegative tensors, which has been studied in the past  ,  . In general, factorization of tensors has been applied to NLP in  and factorization of nonnegative tensors  . Recently, factorization of symmetric tensors has been used to create a generic word embedding  but the idea was not explored extensively. Our work studies this idea in much greater detail, fully demonstrating the viability of tensor factorization as a technique for training word embeddings. Composition of word vectors to create novel representations has been studied in depth, including additive, multiplicative, and tensor-based methods  ,  . Typically, composition is used to create vectors that represent phrases or sentences. Our work, instead, shows that pairs of word vectors can be composed multiplicatively to create different vector representations for the various meanings of a single polysemous word.

Word Embedding Proposals	CP-S. The first embedding we propose is based on symmetic CP decomposition of the PPMI tensor INLINEFORM0 as discussed in the mathematical preliminaries section. The optimal setting for the word embedding INLINEFORM1 is: INLINEFORM2  Since we cannot feasibly compute this exactly, we minimize the loss function defined as the squared error between the values in INLINEFORM0 and their predicted values: INLINEFORM1   using the techniques discussed in the previous section. JCP-S. A potential problem with CP-S is that it is only trained on third order information. To rectify this issue, we propose a novel joint tensor factorization problem we call Joint Symmetric Rank- INLINEFORM0 CP Decomposition. In this problem, the input is the fixed rank INLINEFORM1 and a list of supersymmetric tensors INLINEFORM2 of different orders but whose axis lengths all equal INLINEFORM3 . Each tensor INLINEFORM4 is to be factorized via rank- INLINEFORM5 symmetric CP decomposition using a single INLINEFORM6 factor matrix INLINEFORM7 .

Computing the Symmetric CP Decomposition	The INLINEFORM0 size of the third order PPMI tensor presents a number of computational challenges. In practice, INLINEFORM1 can vary from INLINEFORM2 to INLINEFORM3 , resulting in a tensor whose naive representation requires at least INLINEFORM4 bytes = 4 TB of floats. Even the sparse representation of the tensor takes up such a large fraction of memory that standard algorithms such as successive rank-1 approximation  ,  and alternating least-squares  are infeasible for our uses. Thus, in this paper we will consider a stochastic online formulation similar to that of  . We optimize the CP decomposition in an online fashion, using small random subsets INLINEFORM0 of the nonzero tensor entries to update the decomposition at time INLINEFORM1 . In this minibatch setting, we optimize the decomposition based on the current minibatch and the previous decomposition at time INLINEFORM2 . To update INLINEFORM3 (and thus the symmetric decomposition), we first define a decomposition loss INLINEFORM4 and minimize this loss with respect to INLINEFORM5 using Adam  .

Quantitative results	Outlier Detection results. The results are shown in Table TABREF20 . The first thing to note is that CP-S outperforms the other methods across each Outlier Detection metric. Since the WikiSem500 dataset is semantically focused, performance at this task demonstrates the quality of semantic information encoded in our embeddings. On OD2, the baselines perform more competitively with our CP Decomposition based models, but when OD3 is considered our methods clearly excel. Since the tensor-based methods are trained directly on third order information and perform much better at OD3, we see that OD3 scores reflect the amount of third order information in a word embedding. This is a validation of OD3, as our 3rd order embeddings would naturally out perform 2nd order embeddings at a task that requires third order information. Still, the superiority of our tensor-based embeddings at OD2 demonstrates the quality of the semantic information they encode. Supervised analogy results. The results are shown in Figure FIGREF18 . At the supervised semantic analogy task, CP-S vastly outperforms the baselines at all levels of training data, further signifying the amount of semantic information encoded by this embedding technique.

Introduction	Word embeddings have been used to improve the performance of many NLP tasks including language modelling  , machine translation  , and sentiment analysis  . The broad applicability of word embeddings to NLP implies that improvements to their quality will likely have widespread benefits for the field. The word embedding problem is to learn a mapping INLINEFORM0 ( INLINEFORM1 100-300 in most applications) that encodes meaningful semantic and/or syntactic information. For instance, in many word embeddings, INLINEFORM2 car INLINEFORM3 truck INLINEFORM4 , since the words are semantically similar. More complex relationships than similarity can also be encoded in word embeddings. For example, we can answer analogy queries of the form INLINEFORM0 ? using simple arithmetic in many state-of-the-art embeddings  . The answer to bed INLINEFORM1 sleep INLINEFORM2 chair INLINEFORM3 INLINEFORM4 is given by the word whose vector representation is closest to INLINEFORM5 sleep INLINEFORM6 bed INLINEFORM7 chair INLINEFORM8 ( INLINEFORM9 sit INLINEFORM10 ). Other embeddings may encode such information in a nonlinear way  .

Conclusion	Our key contributions are as follows: Tensor factorization appears to be a highly applicable and effective tool for learning word embeddings, with many areas of potential future work. Leveraging higher order data in training word embeddings is useful for encoding new types of information and semantic relationships compared to models that are trained using only pairwise data. This indicates that such techniques will prove useful for training word embeddings to be used in downstream NLP tasks.

In this paper we study 3-way PPMI tensors INLINEFORM0 , where INLINEFORM1 , as this is the natural higher-order generalization of the PPMI matrix. We leave the study of creating word embeddings with INLINEFORM2 -dimensional PPMI tensors ( INLINEFORM3 ) to future work.

Shifted PMI	In the same way  considers factorization of positive shifted PMI matrices, we consider factorization of positive shifted PMI tensors INLINEFORM0 , where INLINEFORM1 for some constant shift INLINEFORM2 . We empirically found that different levels of shifting resulted in different qualities of word embeddings – the best shift we found for CP-S was a shift of INLINEFORM3 , whereas any nonzero shift for JCP-S resulted in a worse embedding across the board. When we discuss evaluation we report the results given by factorization of the PPMI tensors shifted by the best value we found for each specific embedding.

If words INLINEFORM2 have a high PPMI, then INLINEFORM3 will also be high, meaning INLINEFORM4 will be close to INLINEFORM5 in the vector space by cosine similarity. For example, even though galaxy is likely to appear in the context of the word star in in the “celestial body” sense, INLINEFORM0 PPMI(star, actor, galaxy) is low whereas INLINEFORM1 PPMI(star, actor, drama) is high. Thus , INLINEFORM2 represents the meaning of star in the “lead performer” sense. In Table TABREF22 we present the nearest neighbors of multiplicative and additive composed vectors for a variety of polysemous words. As we can see, the words corresponding to the nearest neighbors of the composed vectors for our tensor methods are semantically related to the intended sense both for multiplicative and additive composition. In contrast, for CBOW, only additive composition yields vectors whose nearest neighbors are semantically related to the intended sense. Thus, our embeddings can produce complementary sets of polysemous word representations that are qualitatively valid whereas CBOW (seemingly) only guarantees meaningful additive compositionality. We leave automated usage of this property to future work.

Also, when only 10% of the training data is presented, our tensor methods are the only ones that attain nonzero performance – even in such a limited data setting, use of CP-S's vectors results in nearly 40% accuracy. This phenomenon is also observed in the syntactic analogy tasks: our embeddings consistently outperform the others until 100% of the training data is presented. These two observations demonstrate the accessibility of the information encoded in our word embeddings. We can thus conclude that this relational information encoded in the tensor-based embeddings is more easily accessible than that of CBOW and NNSE. Thus, our methods would likely be better suited for transfer learning to actual NLP tasks, particularly those in data-sparse settings. Sentiment analysis results. The results are shown in Figure FIGREF19 . In this task, JCP-S is the dominant method across all levels of training data, but the difference is more obvious when training data is limited. This again indicates that for this specific task the information encoded by our tensor-based methods is more readily available as that of the baselines. It is thus evident that exploiting both second and third order co-occurrence data leads to higher quality semantic information being encoded in the embedding.

In the symmetric CP decomposition, instead of factorizing INLINEFORM0 , we factorize INLINEFORM1 as the triple product of a single factor matrix INLINEFORM2 such that INLINEFORM3  In this formulation, we use INLINEFORM0 to be the word embedding so the vector for INLINEFORM1 is the INLINEFORM2 row of INLINEFORM3 similar to the formulations in  ,  ,  . It is known that the optimal rank- INLINEFORM0 CP decomposition exists for symmetric nonnegative tensors such as the PPMI tensor  . However, finding such a decomposition is NP hard in general  so we must consider approximate methods. In this work, we only consider the symmetric CP decomposition, leaving the study of other tensor decompositions (such as the Tensor Train or HOSVD  ,  ) to future work.

Tensor factorization	Just as the rank- INLINEFORM0 matrix decomposition is defined to be the product of two factor matrices ( INLINEFORM1 ), the canonical rank- INLINEFORM2 tensor decomposition for a third order tensor is defined to be the product of three factor matrices  : DISPLAYFORM0   where INLINEFORM0 is the outer product: INLINEFORM1 . This is also commonly referred to as the rank-R CP Decomposition. Elementwise, this is written as: INLINEFORM2  where INLINEFORM0 is elementwise vector multiplication and INLINEFORM1 is the INLINEFORM2 row of INLINEFORM3 . In our later section on multiplicative compositionality, we will see this formulation gives rise to a meaningful interpretation of the elementwise product between vectors in our word embeddings. Symmetric CP Decomposition. In this paper, we will consider symmetric CP decomposition of nonnegative tensors  ,  . Since our INLINEFORM0 -way PPMI is nonnegative and invariant under permutation, the PPMI tensor INLINEFORM1 is nonnegative and supersymmetric, i.e. INLINEFORM2 for any permutation INLINEFORM3 .

At each time INLINEFORM0 , we take INLINEFORM1 to be all co-occurrence triples (weighted by PPMI) in a fixed number of sentences (around 1,000) from the corpus. We continue training until we have depleted the entire corpus. For INLINEFORM0 to accurately model INLINEFORM1 , we also include a certain proportion of elements with zero PPMI (or “negative samples”) in INLINEFORM2 , similar to that of  . We use an empirically found proportion of negative samples for training, and leave discovery of the optimal negative sample proportion to future work.

To the best of our knowledge this is the first study of Joint Symmetric Rank- INLINEFORM0 CP Decomposition.

Notation	Throughout this paper we will write scalars in lowercase italics INLINEFORM0 , vectors in lowercase bold letters INLINEFORM1 , matrices with uppercase bold letters INLINEFORM2 , and tensors (of order INLINEFORM3 ) with Euler script notation INLINEFORM4 , as is standard in the literature.

Question: On which dataset(s) do they compute their word embeddings?
Answer: 10 million sentences gathered from Wikipedia