Venue: NAACL 2019

# Better Word Embeddings by Disentangling Contextual n-Gram Information

Prakhar Gupta*

EPFL

Switzerland

prakhar.gupta@epfl.ch &Matteo Pagliardini*

Iprova SA, Switzerland mpgliardini@iprova.com &Martin Jaggi

EPFL

Switzerland

martin.jaggi@epfl.ch

###### Abstract

Pre-trained word vectors are ubiquitous in Natural Language Processing applications. In this paper, we show how training word embeddings jointly with bigram and even trigram embeddings, results in improved unigram embeddings. We claim that training word embeddings along with higher n-gram embeddings helps in the removal of the contextual information from the unigrams, resulting in better stand-alone word embeddings. We empirically show the validity of our hypothesis by outperforming other competing word representation models by a significant margin on a wide variety of tasks. We make our models publicly available.

+
Footnote †: indicates equal contribution

## 1 Introduction

Distributed word representations are essential building blocks of modern NLP systems. Used as features in downstream applications, they often enhance generalization of models trained on a limited amount of data. They do so by capturing relevant distributional information about words from large volumes of unlabeled text.

Efficient methods to learn word vectors have been introduced in the past, most of them based on the distributional hypothesis of Harris (1954); Firth (1957): _"a word is characterized by the company it keeps"_. While a standard approach relies on global corpus statistics Pennington et al. (2014) formulated as a matrix factorization using mean square reconstruction loss, other widely used methods are the bilinear word2vec architectures introduced by Mikolov et al. (2013): While skip-gram aims to predict nearby words from a given word, CBOW predicts a target word from its set of context words.

Recently, significant improvements in the quality of the word embeddings were obtained by augmenting word-context pairs with sub-word information in the form of character n-grams Bojanowski et al. (2017), especially for morphologically rich languages. Nevertheless, to the best of our knowledge, no method has been introduced leveraging collocations of words with higher order word n-grams such as bigrams or trigrams as well as character n-grams together.

In this paper, we show how using higher order word n-grams along with unigrams during training can significantly improve the quality of obtained word embeddings. The addition furthermore helps to disentangle contextual information present in the training data from the unigrams and results in overall better distributed word representations.

To validate our claim, we train two modifications of CBOW augmented with word-n-gram information during training. One is a recent sentence embedding method, Sent2Vec Pagilardini et al. (2018), which we repurpose to obtain word vectors. The second method we propose is a modification of CBOW enriched with character n-gram information Bojanowski et al. (2017) that we again augment with word n-gram information. In both cases, we compare the resulting vectors with the most widely used word embedding methods on word similarity and analogy tasks and show significant quality improvements. The code used to train the models presented in this paper as well as the models themselves are made available to the public1.

Footnote 1: publicly available on [http://github.com/](http://github.com/) epfml/sent2vec

## 2 Model Description

Before introducing our model, we recapitulate fundamental existing word embeddings methods.

**CBOW and skip-gram models**. Continuous bag-of-words (CBOW) and skip-gram models are

standard log-bilinear models for obtaining word embeddings based on word-context pair information (Mikolov et al., 2013a). Context here refers to a symmetric window centered on the target word \(w_{t}\), containing the surrounding tokens at a distance less than some window size \(\mathsf{ws}\): \(C_{t}=\left\{w_{k}\,|\,k\in[t-\mathsf{ws},t+\mathsf{ws}]\right\}\). The CBOW model tries to predict the target word given its context, maximizing the likelihood \(\prod_{t=1}^{T}p(w_{t}|C_{t})\), whereas skip-gram learns by predicting the context for a given target word maximizing \(\prod_{t=1}^{T}p(C_{t}|w_{t})\). To model those probabilities, a softmax activation is used on top of the inner product between a target vector \(\mathbf{u}_{w_{t}}\) and its context vector \(\frac{1}{|C_{t}|}\sum_{w\in C_{t}}\mathbf{v}_{w}\).

To overcome the computational bottleneck of the softmax for large vocabulary, negative sampling or noise contrastive estimation are well-established (Mikolov et al., 2013b), with the idea of employing simpler pairwise binary classifier loss functions to differentiate between the valid context \(C_{t}\) and fake contexts \(N_{C_{t}}\) sampled at random. While generating target-context pairs, both CBOW and skip-gram also use input word subsampling, discarding higher-frequency words with higher probability during training, in order to prevent the model from overfitting the most frequent tokens. Standard CBOW also uses a dynamic context window size: for each subsampled target word \(w\), the size of its associated context window is sampled uniformly between 1 and \(\mathsf{ws}\)(Mikolov et al., 2013b).

**Adding character n-grams**. Bojanowski et al. (2017) have augmented CBOW and skip-gram by adding character n-grams to the context representations. Word vectors are expressed as the sum of its unigram and average of its character n-gram embeddings \(W_{w}\):

\[\mathbf{v}:=\mathbf{v}_{w}+\frac{1}{|W_{w}|}\sum_{c\in W_{w}}\mathbf{v}_{c}\]

Character n-grams are hashed to an index in the embedding matrix . The training remains the same as for CBOW and skip-gram. This approach greatly improves the performances of CBOW and skip-gram on morpho-syntactic tasks. For the rest of the paper, we will refer to the CBOW and skip-gram methods enriched with subword-information as _CBOW-char_ and _skip-gram-char_ respectively.

**GloVe**. Instead of training online on local window contexts, GloVe vectors (Pennington et al., 2014) are trained using global co-occurrence statistics by factorizing the word-context co-occurrence matrix.

**Ngram2vec**. In order to leverage the performance of word vectors, training of word vectors using the skip-gram objective function with negative sampling is augmented with n-gram co-occurrence information (Zhao et al., 2017).

### Improving unigram embeddings by adding higher order word-n-grams to contexts

**CBOW-char with word n-grams**. We propose to augment CBOW-char to additionally use word n-gram context vectors (in addition to char n-grams and the context word itself). More precisely, during training, the context vector for a given word \(w_{t}\) is given by the average of all word-n-grams \(N_{t}\), all char-n-grams, and all unigrams in the span of the current context window \(C_{t}\):

\[\mathbf{v}:=\frac{\sum_{w\in C_{t}}\mathbf{v}_{w}+\sum_{n\in N_{t}}\mathbf{ v}_{n}+\sum_{w\in C_{t}}\sum_{c\in W_{w}}\mathbf{v}_{c}}{|C_{t}|+|N_{t}|+\sum_{ w\in C_{t}}|W_{w}|} \tag{1}\]

For a given sentence, we apply input subsampling and a sliding context window as for standard CBOW. In addition, we keep the mapping from the subsampled sentence to the original sentence for the purpose of extracting word n-grams from the original sequence of words, within the span of the context window. Word n-grams are added to the context using the hashing trick in the same way char-n-grams are handled. We use two different hashing index ranges to ensure there is no collision between char n-gram and word n-gram representations.

**Sent2Vec for word embeddings**. Initially implemented for sentence embeddings, Sent2Vec (Pagliardini et al., 2018) can be seen as a derivative of word2vec's CBOW. The key differences between CBOW and Sent2Vec are the removal of the input subsampling, considering the entire sentence as context, as well as the addition of word-n-grams.

Here, word and n-grams embeddings from an entire sentence are averaged to form the corresponding sentence (context) embedding.

For both proposed CBOW-char and Sent2Vec models, we employ dropout on word n-grams during training. For both models, word embeddings are obtained by simply discarding the higher order n-gram embeddings after training 



## 3 Experimental Setup

### Training

We train all competing models on a wikipedia dump of 69 million sentences containing 1.7 billion words, following Pagliardini et al. (2018).

Sentences are tokenized using the Stanford NLP library Manning et al. (2014). All algorithms are implemented using a modified version of the fasttext Bojanowski et al. (2017); Joulin et al. (2017) and sent2vec Pagliaroli et al. (2018) libraries respectively. Detailed training hyperparameters for all models included in the comparison are provided in Table 3 in the supplementary material. During training, we save models checkpoints at 20 equidistant intervals and found out that the best performance for CBOW models occurs around \(60-80\%\) of the total training. As a result, we also indicate the checkpoint at which we stop training the CBOW models. We use 300-dimension vectors for all our word embedding models. For the Ngram2vec model, learning source and target embeddings for all the n-grams upto bigrams was the best performing model and is included in the comparison.

For each method, we extensively tuned hyperparameters starting from the recommended values. For each model, we select the parameters which give the best averaged results on our word-similarity and analogy tasks. After selecting the best hyperparameters, we train 5 models for each method, using a different random seed. The reported results are given as mean and standard deviation for those five models.

### Evaluation

In order to evaluate our model, we use six datasets covering pair-wise word-similarity tasks and two datasets covering word-analogy tasks.

**Word-similarity tasks**. Word-similarity tasks consist of word pairs along with their human annotated similarity scores. To evaluate the performance of our models on pair-wise word-similarity tasks, we use _WordSim353_ (353 word-pairs) Finkelstein et al. (2002) divided into two datasets, _WordSim Similarity_ (203 word-pairs) and

\begin{table}
\begin{tabular}{|l|c|c|c|} \hline Model & WS 353 & WS 353 Relatedness & WS 353 Similarity \\ \hline
**CBOW-char** & \(.709\pm.006\) & \(.626\pm.009\) & \(.783\pm.004\) \\
**CBOW-char + bi. i.** & \(.719\pm.007\) & \(.652\pm.010\) & \(.778\pm.007\) \\
**CBOW-char + bi. + tri.** & \(.727\pm.008\) & \(.664\pm.008\) & \(.783\pm.004\) \\ \hline
**Sent2Vec uni.** & \(.705\pm.004\) & \(.593\pm.005\) & \(.793\pm.006\) \\
**Sent2Vec uni. + bi.** & \(.755\pm.005\) & \(.683\pm.008\) & \(.817\pm.007\) \\
**Sent2Vec uni. + bi. + tri.** & \(.780\pm.003\) & \(.721\pm.006\) & \(.828\pm.003\) \\ \hline \end{tabular} 
\begin{tabular}{|l|c|c|c|} \hline Model & SimLex-999 & MEN & Rare Words & Mechanical Turk \\ \hline
**CBOW-char** & \(.424\pm.004\) & \(.769\pm.002\) & \(.497\pm.002\) & \(.675\pm.007\) \\
**CBOW-char + bi.** & \(.436\pm.004\) & \(.786\pm.002\) & \(.506\pm.001\) & \(.671\pm.007\) \\
**CBOW-char + bi. + tri.** & \(.441\pm.003\) & \(.788\pm.002\) & \(.509\pm.003\) & \(.678\pm.010\) \\ \hline
**Sent2Vec uni.** & \(.450\pm.003\) & \(.765\pm.002\) & \(.444\pm.001\) & \(.625\pm.005\) \\
**Sent2Vec uni. + bi.** & \(.440\pm.002\) & \(.791\pm.002\) & \(.430\pm.002\) & \(.661\pm.005\) \\
**Sent2Vec uni. + bi. + tri.** & \(.464\pm.003\) & \(.798\pm.001\) & \(.432\pm.003\) & \(.658\pm.006\) \\ \hline \end{tabular} 
\begin{tabular}{|l|c|c|c|} \hline Model & Google & Google & MSR \\ \hline
**CEOW-char** & \(.920\pm.001\) & \(.799\pm.004\) & \(.842\pm.002\) \\
**CDow-char + bi.** & \(.928\pm.003\) & \(.798\pm.006\) & \(.856\pm.004\) \\
**CROW-char + bi. + tri.** & \(.929\pm.001\) & \(.794\pm.005\) & \(.857\pm.002\) \\ \hline
**Sent2Vec uni.** & \(.826\pm.003\) & \(.847\pm.003\) & \(.734\pm.003\) \\
**Sent2Vec uni. + bi.** & \(.843\pm.004\) & \(.844\pm.002\) & \(.754\pm.004\) \\
**Sent2Vec uni. + bi. + tri.** & \(.837\pm.003\) & \(.853\pm.003\) & \(.745\pm.001\) \\ \hline \end{tabular}
\end{table}
Table 1: Impact of using word n-grams: Models are compared using Spearman correlation measures for word similarity tasks and accuracy for word analogy tasks. Top performances on each dataset are shown in bold. An underline shows the best model(s) restricted to each architecture type. The abbreviations uni., bi., and tri. stand for unigrams, bigrams, and trigrams respectively.



_WordSim Relatedness_ (252 word-pairs) [1]; _MEN_ (3000 word-pairs) [4]; _Mechanical Turk_ dataset [7] (287 word-pairs); _Rare words dataset_ (2034 word-pairs) [9]; and _SimLex-999_ (99 word-pairs) [6] dataset.

To calculate the similarity between two words, we use the cosine similarity between their word representations. The similarity scores then, are compared to the human ratings using Spearman's \(\rho\)[10] correlation scores.

**Word-analogy tasks**. Word analogy tasks pose analogy relations of the form "\(x\) is to \(y\) as \(x^{\star}\) is to \(y^{\star}\)", where \(y\) is hidden and must be guessed from the dataset vocabulary.

We use the _MSR_[7] and the _Google_[7] analogy datasets. The _MSR_ dataset contains 8000 syntactic analogy quadruplets while the _Google_ set has 8869 semantic and 10675 syntactic relations.

To calculate the missing word in the relation, we use the 3CosMul method [10]:

\[y^{\star}:=\arg\max_{z\in\mathcal{V}\setminus\{x,y^{\star}\}}\frac{cos( \mathbf{v}_{z},\mathbf{v}_{y})\cdot cos(\mathbf{v}_{z},\mathbf{v}_{x^{\star} })}{cos(\mathbf{v}_{z},\mathbf{v}_{x})+\varepsilon} \tag{2}\]

where \(\varepsilon=0.0001\) is used to prevent division by 0 and \(\mathcal{V}\) is the dataset vocabulary.

We remove all the out of vocabulary words and are left with 6946 syntactic relations for the _MSR_ dataset and 1959 word-pairs for the _Rare Words_ dataset. All other datasets do not have any out of vocabulary words.

\begin{table}
\begin{tabular}{|l|c|c|c|} \hline Model & WS 353 & WS 353 Relatedness & WS 353 Similarity \\ \hline
**CBOW-char + bi. + trii.** & \(.727\pm.008\) & \(.664\pm.008\) & \(.783\pm.004\) \\
**Sent2Vec uni. + bi. + trii.** & \(.780\pm.003\) & \(.721\pm.006\) & \(.828\pm.003\) \\ \hline
**CBOW-char** & \(.709\pm.006\) & \(.626\pm.009\) & \(.783\pm.004\) \\ \hline
**CBOW** & \(.722\pm.008\) & \(.634\pm.008\) & \(.796\pm.005\) \\ \hline
**Skip-gram-char** & \(.724\pm.007\) & \(.655\pm.008\) & \(.789\pm.004\) \\ \hline
**Skip-gram** & \(.736\pm.004\) & \(.672\pm.007\) & \(.796\pm.005\) \\ \hline
**GloVe** & \(.559\pm.002\) & \(.484\pm.005\) & \(.665\pm.008\) \\ \hline
* bi.** & \(.745\pm.003\) & \(.687\pm.003\) & \(.797\pm.004\) \\ \hline \end{tabular} 
\begin{tabular}{|l|c|c|c|c|} \hline Model & SimLex-999 & MEN & Rare Words & Mechanical Turk \\ \hline
**CBOW-char + bi. + trii.** & \(.441\pm.003\) & \(.788\pm.002\) & \(.509\pm.003\) & \(.678\pm.010\) \\
**Sent2Vec uni. + bi. + trii.** & \(.464\pm.003\) & \(.798\pm.001\) & \(.432\pm.003\) & \(.658\pm.006\) \\ \hline
**CBOW-char** & \(.424\pm.004\) & \(.769\pm.002\) & \(.497\pm.002\) & \(.675\pm.007\) \\ \hline
**CEOW** & \(.432\pm.004\) & \(.757\pm.002\) & \(.454\pm.002\) & \(.674\pm.006\) \\ \hline
**Skip-gram-char** & \(.395\pm.003\) & \(.762\pm.001\) & \(.487\pm.002\) & \(.684\pm.003\) \\ \hline
**Skip-gram** & \(.405\pm.001\) & \(.770\pm.001\) & \(.468\pm.002\) & \(.684\pm.005\) \\ \hline
**GloVe** & \(.375\pm.002\) & \(.690\pm.001\) & \(.327\pm.002\) & \(.622\pm.004\) \\ \hline
* bi.** & \(.424\pm.000\) & \(.756\pm.001\) & \(.462\pm.002\) & \(.681\pm.004\) \\ \hline \end{tabular} 
\begin{tabular}{|l|c|c|c|} \hline Model & Google & Google & MSR \\  & (Syntactic Analogies) & (Semantic Analogies) & MSR \\ \hline
**CROW-char + bi. + trii.** & \(.929\pm.001\) & \(.794\pm.005\) & \(.857\pm.002\) \\
**Sent2Vec uni. + bi. + trii.** & \(.837\pm.003\) & \(.853\pm.003\) & \(.745\pm.001\) \\ \hline
**CREOW-char** & \(.920\pm.001\) & \(.799\pm.004\) & \(.842\pm.002\) \\ \hline
**CBOW** & \(.816\pm.002\) & \(.805\pm.005\) & \(.713\pm.004\) \\ \hline
**Skip-gram-char** & \(.860\pm.001\) & \(.828\pm.005\) & \(.796\pm.003\) \\ \hline
**Skip-gram** & \(.829\pm.002\) & \(.837\pm.002\) & \(.753\pm.005\) \\ \hline
**GloVe** & \(.767\pm.002\) & \(.697\pm.007\) & \(.678\pm.003\) \\ \hline
* bi.** & \(.834\pm.001\) & \(.812\pm.003\) & \(.761\pm.001\) \\ \hline \end{tabular}
\end{table}
Table 2: Improvement over existing methods: Models are compared using Spearman correlation measures for word similarity tasks and accuracy for word analogy tasks. Top performance(s) on each dataset is(are) shown in bold. The abbreviations uni., bi., and tri. stand for unigrams, bigrams, and trigrams respectively.
 

Results

**Impact of word n-grams**. In Table 1, we evaluate the impact of adding contextual word n-grams to two CBOW variations: CBOW-char and Sent2Vec. By adding n-gram information, we consistently observe a boost in the Spearman correlation on the word similarity tasks. On the few datasets where we do not observe an improvement, the results for word-n-gram augmented methods are within standard deviation reach. The _Rare Words_ dataset for Sent2Vec is the only exception, despite getting some improvement for CBOW-char based methods. This observation can be attributed to the fact that character ngrams are shared between unigrams, enhancing generalization to infrequent words. Without char n-grams, the model might underfit those rare words, even more so with word n-grams.

We also see that the boost obtained by adding n-grams on word-similarity tasks is much larger for Sent2Vec models as compared to the CBOW-char ones possibly due to the fact that during training, Sent2Vec models use a much larger context and hence can use much more n-gram information for obtaining a better context representation.

For analogy tasks, we see an improvement in the augmented CBOW-char methods for morpho-syntactic analogy datasets with little or no gain for semantic analogy datasets. Yet, for Sent2Vec models, the gain is the other way around. This observation indicates the strong role played by character n-grams in boosting the performance on the syntactic tasks as well as restricting the word n-grams from improving the performance on semantic analogies.

**Comparison with competing methods**. In Table 2, we compare word n-gram augmented methods with the most prominent word embedding models. We obtain state-of-the-art results for standalone unigram embeddings on most of the datasets confirming our hypothesis. The _Mechanical Turk_ dataset is the only exception.

We notice that Sent2Vec trigrams model dominates the word-similarity tasks as well as the semantic analogy tasks. However, character n-grams are quite helpful when it comes to syntactic analogy tasks underlining the importance of subword information. We also note that the Ngram2vec model outperforms our augmented CBOW-char model in some of the tasks but is always inferior to Sent2Vec in those cases.

## 5 Conclusion and Future Work

We empirically show how augmenting the context representations using higher-order word n-grams improves the quality of word representations. The empirical success also calls for a new theoretical model on the composite effect of training higher order n-grams simultaneously with unigrams. Also, the success of Sent2Vec on word-level tasks, a method originally geared towards obtaining general purposed sentence embeddings, hints towards the additional benefits of using compositional methods for obtaining sentence/phrase representations.

## References

* Agirre et al. (2009) Eneko Agirre, Enrique Alfonseca, Keith Hall, Jana Kravalova, Marius Pasca, and Aitor Soroa. 2009. A study on similarity and relatedness using distributional and wordnet-based approaches. In _Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics_, pages 19-27. Association for Computational Linguistics.
* Bojanowski et al. (2017) Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. _Transactions of the Association for Computational Linguistics_, 5:135-146.
* Bruni et al. (2012) Elia Bruni, Gemma Boleda, Marco Baroni, and Nam-Khanh Tran. 2012. Distributional semantics in technicolor. In _Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Long Papers-Volume 1_, pages 136-145. Association for Computational Linguistics.
* Finkelstein et al. (2002) Lev Finkelstein, Evgeniy Gabrilovich, Yossi Matias, Ehud Rivlin, Zach Solan, Gadi Wolfman, and Eytan Ruppin. 2002. Placing search in context: The concept revisited. _ACM Transactions on information systems_, 20(1):116-131.
* Firth (1957) John R Firth. 1957. A synopsis of linguistic theory, 1930-1955. _Studies in linguistic analysis_.
* Harris (1954) Z Harris. 1954. Distributional structure.(j. katz, ed.) word journal of the international linguistic association, 10 (23), 146-162.
* Hill et al. (2015) Felix Hill, Roi Reichart, and Anna Korhonen. 2015. Simlex-999: Evaluating semantic models with (genuine) similarity estimation. _Computational Linguistics_, 41(4):665-695.
* Joulin et al. (2017) Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2017. Bag of tricks for efficient text classification. In _EACL_ 

Omer Levy and Yoav Goldberg. 2014. Linguistic regularities in sparse and explicit word representations. In _Proceedings of the eighteenth conference on computational natural language learning_, pages 171-180.
* Luong et al. (2013) Thang Luong, Richard Socher, and Christopher Manning. 2013. Better word representations with recursive neural networks for morphology. In _Proceedings of the Seventeenth Conference on Computational Natural Language Learning_, pages 104-113.
* Manning et al. (2014) Christopher D Manning, Mihai Surdeanu, John Bauer, Jenny Rose Finkel, Steven Bethard, and David McClosky. 2014. The stanford corenlp natural language processing toolkit. In _ACL (System Demonstrations)_, pages 55-60.
* Mikolov et al. (2013a) Tomas Mikolov, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. 2013a. Efficient estimation of word representations in vector space. _CoRR_, abs/1301.3781.
* Advances in Neural Information Processing Systems 26_, pages 3111-3119.
* Mikolov et al. (2013c) Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. 2013c. Linguistic regularities in continuous space word representations. In _Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 746-751.
* Pagliardiini et al. (2018) Matteo Pagliardiini, Prakhar Gupta, and Martin Jaggi. 2018. Unsupervised learning of sentence embeddings using compositional n-gram features. In _NAACL-HLT_.
* Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation. In _EMNLP_, volume 14, pages 1532-1543.
* Radinsky et al. (2011) Kira Radinsky, Eugene Agichtein, Evgeniy Gabrilovich, and Shaul Markovitch. 2011. A word at a time: computing word relatedness using temporal semantic analysis. In _Proceedings of the 20th international conference on World wide web_, pages 337-346. ACM.
* Spearman (1904) Charles Spearman. 1904. The proof and measurement of association between two things. _The American journal of psychology_, 15(1):72-101.
* Zhao et al. (2017) Zhe Zhao, Tao Liu, Shen Li, Bofang Li, and Xiaoyong Du. 2017. Ngram2vec: Learning improved word representations from ngram co-occurrence statistics. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_, pages 244-253.

## Appendix A Training parameters for selected models

Training parameters for all models except GloVe and Ngram2vec are provided in Table 3. For the GloVe model , the minimum word count is set to 10; the window size is set to 10; we use 10 epochs for training; \(X_{max}\), the weighting parameter for the word-context pairs is set to 100; all other parameters are set to default. For Ngram2vec, the minimum word count is set to 10; the window size is set to 5; both source and target vectors are trained for unigrams and bigrams; overlap between the target word and source n-grams is allowed. All other features are set to default. To train the Ngram2vec models, we use the library provided by Zhao et al. (2017)2.

 



\begin{table}
\begin{tabular}{|p{42.7pt}|p{39.8pt}|p{39.8pt}|p{39.8pt}|p{39.8pt}|p{39.8pt}|p{39.8pt}|p{39.8pt}|p{39.8pt}|p{39.8pt}|p{39.8pt}|p{39.8pt}|} \hline Model & SentZVec uni. & SentZVec uni.+bi. & SentZVec uni.+bi+trii. & CBOW (char). & CBOW (char.)+bi. & CBOW (char.)+bi. & CBOW (char.)+bi. +trii. & CBOW (char.) & Skip-gram (char.) \\ \hline Embedding Dimensions & 300 & 300 & 300 & 300 & 300 & 300 & 300 & 300 & 300 & 300 \\ \hline Max Vocab. Size & 750k & 750k & 750k & 750k & 750k & 750k & 750k & 750k & 750k \\ \hline Minimum Word Count Initial Learning Rate & 5 & 5 & 5 & 5 & 5 & 5 & 5 & 5 \\ \hline Epochs & 9 & 9 & 9 & 9 & 9 & 9 & 5 & 15 & 15 \\ \hline Subsampling hyper-param. & \multirow{2}{*}{\(1\times 10^{-5}\)} & \multirow{2}{*}{\(5\times 10^{-5}\)} & \multirow{2}{*}{\(5\times 10^{-6}\)} & \multirow{2}{*}{\(1\times 10^{-4}\)} & \multirow{2}{*}{\(1\times 10^{-4}\)} & \multirow{2}{*}{\(1\times 10^{-4}\)} & \multirow{2}{*}{\(1\times 10^{-4}\)} & \multirow{2}{*}{\(1\times 10^{-4}\)} & \multirow{2}{*}{\(1\times 10^{-4}\)} \\ \cline{2-2} \cline{5-11} Word-Ngrams Bucket Size & - & 2M & 4M & - & 2M & 4M & - & - \\ \hline Char-Ngrams Bucket Size & - & - & - & 2M & 2M & 2M & - & 2M & - \\ \hline Word-Ngrams Dropped per context & - & 4 & 4 & - & 2 & 2 & - & - & - \\ \hline Window Size & - & - & - & 10 & 10 & 10 & 10 & 5 & 5 \\ \hline Number of negatives sampled & 10 & 10 & 10 & 5 & 5 & 5 & 5 & 5 & 5 \\ \hline Max Char-Ngram Size & - & - & - & 6 & 6 & 6 & - & 6 & - \\ \hline Min Char-Ngram Size & - & - & - & 3 & 3 & 3 & - & 3 & - \\ \hline Percentage at which training is halted (For CBOW models only) & - & - & - & 75\% & 80\% & 80\% & 60\% & - & - \\ \hline  & \multicolumn{1}{c}{} & \multicolumn{1}{p{39.8pt}}{} & \multicolumn{1}{p{39.8pt}}{} & \multicolumn{1}{p{39.8pt}}{} & \multicolumn{1}{p{39.8pt}}{} & \multicolumn{1}{p{39.8pt}}{} & \multicolumn{1}{p{39.8pt}}{} & \multicolumn{1}{p{39.8pt}}{} & \multicolumn{1}{p{39.8pt}}{} & \multicolumn{1}{p{39.8pt}}{} \\ \end{tabular}
\end{table}
Table 3: Training parameters for all non-GloVe models