Venue: NAACL 2019

# Transferable Neural Projection Representations

Chinnadhurai Sankar

Mila, Universite de Montreal1

Montreal, QC, Canada

chinnadhurai@gmail.com &Sujith Ravi

Google Research

Mountain View, CA, USA

sravi@google.com &Zornitsa Kozareva

Google

Mountain View, CA, USA

zornitsa@kozareva.com

Work done during internship at Google.

###### Abstract

Neural word representations are at the core of many state-of-the-art natural language processing models. A widely used approach is to pretrain, store and look up word or character embedding matrices. While useful, such representations occupy huge memory making it hard to deploy on-device and often do not generalize to unknown words due to vocabulary pruning. In this paper, we propose a skip-gram based architecture coupled with Locality-Sensitive Hashing (LSH) projections to learn efficient dynamically computable representations. Our model does not need to store lookup tables as representations are computed on-the-fly and require low memory footprint. The representations can be trained in an unsupervised fashion and can be easily transferred to other NLP tasks. For qualitative evaluation, we analyze the nearest neighbors of the word representations and discover semantically similar words even with misspellings. For quantitative evaluation, we plug our transferable projections into a simple LSTM and run it on multiple NLP tasks and show how our transferable projections achieve better performance compared to prior work.

## 1 Introduction

Pre-trained word representations are at the core of many neural language understanding models. Among the most popular and widely used word embeddings are word2vec Mikolov et al. (2013), GloVe Pennington et al. (2014) and ELMO Peters et al. (2018). The biggest challenge with word embedding is that they require lookup and a large memory footprint, as we have to store one entry (\(d\)-dim vector) per word and it blows up.

In parallel, the tremendous success of deep learning models and the explosion of mobile, IoT devices coupled together with the growing user privacy concerns have led to the need for deploying deep learning models on-device for inference. This has led to new research in compressing large and complex deep learning models for low power on-device deployment. Recently, Ravi and Kozareva (2018) developed an on-device neural text classification model. They proposed to reduce the memory footprint of large neural networks by replacing the input word embeddings with projection based representations. Ravi and Kozareva (2018) used n-gram features to generate binary LSH Charikar (2002) randomized projections on the fly surpassing the need to store word emebdding tables and reducing the memory size. The projection models reduce the memory occupied by the model from \(O(|V|)\) to \(O(n_{\mathbb{P}})\), where \(|V|\) refers to the vocabulary size and \(n_{\mathbb{P}}\) refers to number of projection operations Ravi (2017). Two key advantages of the projection based representations over word embeddings are: (1) they are fixed and have low memory size; (2) they can handle out of vocabulary words. However, the projections in Ravi and Kozareva (2018) are static and currently do not leverage pre-training on large unsupervised corpora, which is an important property to make the projections transferable to new tasks.

In this paper, we propose to combine the best of both worlds by learning transferable neural projection representations over randomized LSH projections. We do this by introducing new neural architecture inspired by the skip gram model of Mikolov et al. (2013) and combined with a deep MLP plugged on top of LSH projections. In order to make this model train better, we introduce new regularizing loss function, which minimizes the cosine similarities of the words within a mini-batch. The loss function is critical for generalization.

In summary, our model (1) requires a fixed and low memory footprint, (2) can handle out of vo

cabulary words and misspellings, (3) captures semantic and syntactic properties of words; (4) can be easily plugged to other NLP models and (5) can support training with data augmentation by perturbing characters of input words. To validate the performance of our approach, we conduct a qualitative analysis of the nearest neighbours in the learned representation spaces and a quantitative evaluation via similarity, language modeling and NLP tasks.

## 2 Neural Projection Model

We propose a novel model (NP-SG) to learn compact neural representations that combines the benefit of representation learning approaches like skipgram model with efficient LSH projections that can be computed on-the-fly.

### Vanilla Skip-Gram Model

In the skip-gram model Mikolov et al. (2013), we learn continuous distributed representations for words in a large fixed vocabulary, \(\mathbb{V}\) to predict the context words surrounding them in documents. We maintain an embedding look up table, \(v(w)\in\mathbb{R}^{d}\) for every word, \(w\in\mathbb{V}\).

For each word, \(w_{t}\) in the training corpus of size \(T\), the set of context words \(\mathbb{C}_{t}=\{w_{t-W_{t}},\ldots,w_{t-1},w_{t+1},\ldots,w_{t+W_{t}}\}\) includes \(W_{t}\) words to the left and right of \(w_{t}\) respectively. \(W_{t}\) is the window size randomly sampled from the set \(\{1,2,\dots,N\}\), where \(N\) is the maximum window size. Given a pair of words, \(\{w_{c},w_{t}\}\), the probability of \(w_{c}\) being within the context window of \(w_{t}\) is given by equation 1.

\[\begin{split}\mathrm{P}(w_{c}|w_{t})&=\sigma(v^{ \prime}(w_{c})^{\intercal}v(w_{t}))\\ &=\frac{1}{1+\exp(-v^{\prime}(w_{c})^{\intercal}v(w_{t}))}\end{split} \tag{1}\]

where \(v,v^{\prime}\) are input and context embedding look up tables.

### Neural Projection Skip-Gram (NP-SG)

In the neural projection approach, we replace the input embedding look up table, \(v(w)\) in equation 1 with a deep \(n\)-layer MLP over the binary projection, \(\mathbb{P}(w)\) as shown equation 2.

\[v_{\mathbb{P}}(w)=\mathbb{N}(f_{n}(\mathbb{P}(w))) \tag{2}\]

where \(v_{\mathbb{P}}(w)\in\mathbb{R}^{d}\), \(f_{n}\) is a \(n\)-layer deep neural network encoder with \(ReLU\) non-linear activations after each layer except for the last layer as shown in Figure 1. \(\mathbb{N}\) refers to a normalization applied to the final layer of \(f_{n}\). We experimented with Batch-normalization, L2-normalization and layer normalization; batch-normalization works the best.

The binary projection \(\mathbb{P}(w)\) is computed using locality-sensitive projection operations Ravi (2017) which can be performed on-the-fly (i.e., without any embedding look up) to yield a fixed, low-memory footprint binary vector. Unlike Ravi and Kozareva (2018) which uses _static_ projections to encode the entire input text and learn a classifier, NP-SG creates a _trainable_ deep projection representation for words using LSH projections over character-level features combined with contextual information learned via the skip-gram architecture.

### Training NP-SG Model

We follow a similar approach as Mikolov et al. (2013) and others for training our neural projection skip-gram model (NP-SG). We define the training objective to maximize the probability of predicting the context words given the current word. Formally, the model tries to learn the word embeddings by maximizing the objective, \(J(\theta)\) known as negative sampling (NEG), given by equation 3.

\[J(\theta)=\sum_{t=1}^{T}\sum_{w_{c}\in\mathbb{C}_{t}}J_{w_{t},w_{ c}}(\theta) \tag{3}\]

\[\begin{split} J_{w_{t},w_{c}}(\theta)&=\log( \mathrm{P}(w_{c}|w_{t}))\\ &\quad+\sum_{i=1,w_{i}\sim\mathrm{P}_{n}(w)}^{k}\log(1-\mathrm{ P}(w_{i}|w_{t}))\end{split} \tag{4}\]

where \(k\) is the number of randomly sampled words from the training corpus according to the noise distribution, \(\mathrm{P}_{n}(w)\propto U(w)^{3/4}\), where \(U(w)\) is the unigram distribution of the training corpus.

Figure 1: Neural Projection Skip-gram (NP-SG) model





**Model improvements:** Training an NP-SG model as is, though efficient, may not lead to highly discriminative representations. During training, we noticed that the word representations, \(v_{\mathbb{P}}(w)\) were getting projected in a narrow sub-space where the cosine similarities of all the words in the dataset were too close to \(1.0\). This made the convergence slower and led to poor generalization.

### Discriminative NP-SG Models

To encourage the word representations to be more spaced out in terms of the cosine similarities, we introduce an additional explicit regularizing L2-loss function. With the assumption that the words in each mini-batch are randomly sampled, we add a L2-loss over the cosine similarities between all the words within a mini-batch, as shown in equation 6.

\[Loss=J(\theta)+\mathrm{L}^{cs}_{2}(\textbf{w}_{mb}) \tag{5}\]

\[\mathrm{L}^{cs}_{2}(\textbf{w}_{mb})=\lambda\,\cdot\,\|\left\{\mathrm{CS}(w_{ i},w_{j})\,|\,i,j\in[0,mb)\right\}\|^{2}_{2} \tag{6}\]

where \(\mathrm{CS}(w_{i},w_{j})\) refers to the cosine similarity between \(w_{i}\) and \(w_{j}\), \(mb\) refers to the mini-batch size and \(\textbf{w}_{mb}\) refers to the words in the mini-batch. We enforce this using a simple outerproduct trick. We extract the cosine-similarities between all the words within a mini-batch in a single shot by computing the outer-product of the \(L_{2}\) row normalized word representations corresponding to each mini-batch \(\hat{v}_{\mathbb{P}}(\textbf{w}_{mb})\), as shown in equation 7.

\[\mathrm{L}^{cs}_{2}(\textbf{w}_{mb})=\frac{\lambda}{2}\,\cdot\,\|\,\mathrm{ Flatten}(\hat{v}_{\mathbb{T}}(\textbf{w}_{mb})\,\cdot\,\hat{v}_{\mathbb{T}}( \textbf{w}_{mb})^{\intercal})\,\|^{2}_{2} \tag{7}\]

### Improved NP-SG Training

Since the NP-SG model does not have a fixed vocabulary size, we can be flexible and leverage a lot more information during training compared to standard skip-gram models which require vocabulary pruning for feasibility.

To improve training for NP-SG model, we augment the dataset with inputs words after applying character level perturbations to them. The perturbations are such a way that they are commonly occurring misspellings in documents. We mainly experiment with three types of pertub operation APIs Gao et al. (2018).

* _insert(word, n)_ : We randomly choose \(n\) chars from the character vocabulary and insert them randomly into the input _word_. We ignore the locations of first and last character in the word for the _insert_ operation. Example transformation: \(sample\to sample\).
* _swap(word, n)_ : We randomly swap the location of two characters in the word \(n\) times. As with the _insert_ operation, we ignore the first and last character in the word for the _swap_ operation. Example transformation: \(sample\to sapmle\).
* _duplicate(word, n)_ : We randomly duplicate a character in the word by \(n\) times. Example transformation: \(sample\to saample\).

We would like to note that the perturbation operations listed above are not exhaustive and we plan to experiment with more operations in the future.

## 3 Training Setup

### Dataset

We train our skipgram models on the wikipedia data XML dump, _enwik9_1. We extract the normalized English text from the XML dump using the Matt Mahoneys pre-processing perl script2. We fix the vocabulary to the top \(100k\) frequently occurring words. We sub-sample words in the training corpus, dropping them with probability, \(\mathrm{P}(w)=1-\sqrt{t/freq(w)}\), where \(freq(w)\) is the frequency of occurrence of \(w\) in the corpus and we set the threshold, \(t\) to \(10^{-5}\). We perturb the input words with a probability of \(0.4\) using a randomly chosen perturbation described in Section 2.5.

Footnote 1: [http://mattmahoney.net/dc/enwik9.zip](http://mattmahoney.net/dc/enwik9.zip)

Footnote 2: [http://mattmahoney.net/dc/textdata](http://mattmahoney.net/dc/textdata)

### Implementation Details

We fix the number of random projections to \(80\) and the projection dimension to \(14\). We use a 2-layer MLP (sizes: \([2048,100]\)) regularized with dropout (with probability of \(0.65\)) and weight decay (regularization parameter of \(0.0005\)) to transform the binary random projections to continuous word representation. For the vanilla skipgram model, we fix the embedding size to \(100\). For both models, we use \(25\) negative samples for the NEG loss. We learn the parameters using the Adam optimizer Kingma and Ba (2014) with a default learning rate of \(0.001\), clipping the gradients which have a norm larger than \(5.0\). We initialize the weights of the MLP using Xavier initialization, and output embeddings

uniformly random in the range \([-1.0,1.0]\). We use a batch size of \(1024\) in all our experiments. We found that \(\lambda=0.01\) for the outerproduct loss to be working better after experimenting with other values. Training time for our model was around \(0.85\) times that of the skipgram model. Both the models were trained for \(10\) epochs.

## 4 Experiments

We show both qualitative and quantitative evaluation on multiple tasks for the NP-SG model.

### Qualitative Evaluation and Results

Table 2 shows the nearest neighbors produced by NP-SG for select words. Independent of whether it is an original or misspelled word, our NP-SG model accurately retrieves relevant and semantically similar words.

### Quantitative Evaluation and Results

We evaluate our NP-SG model on similarity, language modeling and text classification tasks. Similarity tests the ability to capture words, while language modeling and classification warrant the ability to transfer the neural projections.

#### 4.2.1 Similarity Task

We evaluate our NP-SG word representations on 4 different widely used benchmark datasets for measuring similarities.

**Dataset:**_MTurk-287_Radinsky et al. (2011) has 287 pairs of words and was constructed by crowdsourcing the human similarity ratings using Amazon Mechanical Turk. _WS353_Finkelstein et al. (2001) has 353 pairs of similar English words rated by humans and is further split into _WS353-SIM_. _WS353-REL_Agirre et al. (2009) captures different types of similarities and relatedness. _RWSTANFORD_Luong et al. (2013) has 2034 rare word pairs sampled from different frequency bins.

**Evaluation:** For all the datasets, we compute the Spearman s rank correlation coefficient between the rankings computed by skip-gram models (baseline SG and NP-SG) and the human rankings. We use cosine similarity metric to measure word similarity.

**Results:** Table 1 shows that NP-SG, with significantly smaller number of parameters comes close to the skip-gram model (SG) and even outperforms it with 2.5x-10x compression. NP-SG gets better representations even with misspellings which cannot be handled by vanilla SG.

It is interesting to note that the vanilla skip-gram model does well on _WS353-SIM_ compared to _WS353-REL_. This behavior is reversed in our NP-SG model, which indicates that it captures meronym-holonym relationships better than the vanilla skip-gram model. Although NP-SG handles out of vocabulary words in the form of misspellings, it needs further improvement for rare word similarity task. We plan to improve it by including context word n-gram features in the LSH projection function, allowing NP-SG to also leverage information from the context words in the case of rare words and provide word sense disambiguation.

#### 4.2.2 Language Modeling

We applied NP-SG to language modeling task on the Penn Treebank (PTB)Taylor et al. (2003) dataset. We consider a single layer LSTM with hidden size of 2048 for the language model task. With the input embedding size of 200, we observed a perplexity of \(\approx 120\) on the test set after training

\begin{table}
\begin{tabular}{c|c|c|c|c|c} \hline _Dataset_ & _SG (10M)_ & _NP-SG_ (w/oOP) & _NP-SG_ (1M) & _NP-SG_ (2M) & _NP-SG_ (4M) \\ \hline EN-MTurk-287 & 0.5409 & 0.0107 & **0.5629** & **0.5517** & **0.5494** \\ EN-WS-353-ALL & 0.5930 & 0.0710 & 0.4891 & 0.5215 & 0.5370 \\ EN-WS-353-REL & 0.5359 & 0.0203 & 0.4956 & **0.5746** & **0.5671** \\ EN-WS-353-SIM & 0.6242 & 0.1043 & 0.4994 & 0.5116 & 0.5111 \\ EN-RW-STANFORD & 0.1505 & 0.0401 & 0.0184 & 0.0375 & 0.0835 \\ EN-VERB-143 & 0.2452 & 0.0730 & 0.1333 & 0.1500 & 0.2108 \\ \hline \end{tabular}
\end{table}
Table 1: Similarity Tasks: # of params, \(100k\) vocabulary size for skipgram baseline, \(100\) embedding size.

\begin{table}
\begin{tabular}{l|l} \hline _Word_ & _Nearest neighbours_ \\ \hline king & reign, throne, kings, knights, vii, regent \\ kinge & vii, younger, peerage, iv, tiberius, frederick \\ \hline woman & man, young, girl, child, girls, women \\ wwoamn & man, herself, men, couple, herself, alive \\ \hline city & town, village, borough, township, county \\ city & town, village, borough, country, unorganized \\ \hline time & few, times, once, entire, prominence, since \\ tinme & times, once, takes, taken, another, only \\ \hline zero & two, three, seven, one, eight, four \\ zzero & two, three, five, six, seven, four \\ \hline \end{tabular}
\end{table}
Table 2: Sampled nearest neighbors for NP-SG.

 

for \(5\) epochs. We replace the input embeddings in the LSTM with transferable encoder layer of the NP-SG model. We train the LSTMs with and without pretrained initializations. Since we observed convergence issues with the single layer NP-SG representation, we considered 2-layer MLP with layer sizes (1024, 256) for the NP-SG representations. We found that while the model without pretrained NP-SG layer got stuck at a perplexity of around \(300\), the pretrained version converged to a perplexity of \(140\), comparable to the embedding based network. We leave the analysis of the impact of the deeper NP-SG layers to the future work.

#### 4.2.3 Text Classification

For the text classification evaluations, we used two different tasks and datasets. For the dialog act classification task, we used the MRDA dataset from the ICSI Meeting Recorder Dialog Act Corpus Adam et al. (2003). MRDA is a multiparty dialog annotated with 5 dialog act tags. For the question classification task, we used the TREC dataset Lin and Katz (2006). The task is given a question to predict the most relevant category.

We trained a single layer LSTM (hidden size: \(256\)) with and without the pretrained NP-SG layers. Overall, we observed accuracy improvements of \(+5.7\%\) and \(+3.75\%\) compared to baseline models without pretrained NP-SG initializations on TREC and MRDA respectively.

## 5 Conclusion

In this paper, we introduced a new neural architecture (NP-SG), which learns transferable word representations that can be efficiently and dynamically computed on device without any embedding look up. We proposed an unsupervised method to train the new architecture and learn more discriminative word representations. We compared the new model with a skip-gram approach and showed qualitative and quantitative comparisons on multiple language tasks. The evaluations show that our NP-SG model learns better representations even with misspellings and reaches competitive results with skip-gram on similarity tasks, even outperforming with 2.5x-10x fewer parameters.

## Acknowledgments

The authors would like to thank the Google Expander team for many helpful discussions.

## References

* Adam et al. (2003) Janin Adam, Don Baron, Jane Edwards, Dan Ellis, David Gelbart, Nelson Morgan, Barbara Peskin, Thilo Pfau, Elizabeth Shriberg, Andreas Stolcke, and Chuck Wooters. 2003. The icsi meeting corpus. In _Proceedings of the 5TH SIGdial Workshop on Discourse and Dialogue_, pages 364-367.
* June 5, 2009, Boulder, Colorado, USA_, pages 19-27.
* Charikar (2002) Moses S. Charikar. 2002. Similarity estimation techniques from rounding algorithms. In _Proceedings of the Thirty-fourth Annual ACM Symposium on Theory of Computing_, STOC '02, pages 380-388, New York, NY, USA. ACM.
* Finkelstein et al. (2001) Lev Finkelstein, Evgeniy Gabrilovich, Yossi Matias, Ehud Rivlin, Zach Solan, Gadi Wolfman, and Eytan Ruppin. 2001. Placing search in context: the concept revisited. In _Proceedings of the Tenth International World Wide Web Conference, WWW 10, Hong Kong, China, May 1-5, 2001_, pages 406-414.
* Gao et al. (2018) Ji Gao, Jack Lanchantin, Mary Lou Soffa, and Yanjun Qi. 2018. Black-box generation of adversarial text sequences to evade deep learning classifiers. In _2018 IEEE Security and Privacy Workshops, SP Workshops 2018, San Francisco, CA, USA, May 24, 2018_, pages 50-56.
* Kingma and Ba (2014) Diederik Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. _arXiv preprint arXiv:1412.6980_.
* Lin and Katz (2006) Jimmy J. Lin and Boris Katz. 2006. Building a reusable test collection for question answering. _JASIST_, 57(7):851-861.
* Luong et al. (2013) Thang Luong, Richard Socher, and Christopher D. Manning. 2013. Better word representations with recursive neural networks for morphology. In _Proceedings of the Seventeenth Conference on Computational Natural Language Learning, CoNLL 2013, Sofia, Bulgaria, August 8-9, 2013_, pages 104-113.
* Mikolov et al. (2013) Tomas Mikolov, Ilya Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. 2013. Distributed representations of words and phrases and their compositionality. In _Advances in Neural Information Processing Systems 26: 27th Annual Conference on Neural Information Processing Systems 2013. Proceedings of a meeting held December 5-8, 2013, Lake Tahoe, Nevada, United States._, pages 3111-3119.
* Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In _Proceedings of the 2014 Conference 

on Empirical Methods in Natural Language Processing (EMNLP)_, pages 1532-1543. Association for Computational Linguistics.
* Peters et al. [2018] Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_, pages 2227-2237. Association for Computational Linguistics.
* April 1, 2011_, pages 337-346.
* Ravi [2017] Sujith Ravi. 2017. Projectionnet: Learning efficient on-device deep networks using neural projections. _CoRR_, abs/1708.00630.
* November 4, 2018_, pages 804-810.
* Taylor et al. [2003] Ann Taylor, Mitchell Marcus, and Beatrice Santorini. 2003. The penn treebank: An overview.