Year: 2021

Abstract:
The performance of modern neural language models relies heavily on the diversity of the vocabularies. Unfortunately, the language models tend to cover more vocabularies, the embedding parameters in the language models such as multilingual models used to occupy more than a half of their entire learning parameters. To solve this problem, we aim to devise a novel embedding structure to lighten the network without considerably performance degradation. To reconstruct N embedding vectors, we initialize k bundles of M( N) k-sub-embeddings to apply Cartesian product. Furthermore, we assign k-sub-embedding using the contextual relationship between tokens from pretrained language models. We adjust our k-sub-embedding structure to masked language models to evaluate proposed structure on downstream tasks. Our experimental results show that over 99.9%+ compressed sub-embeddings for the language models performed comparably with the original embedding structure on GLUE and XNLI benchmarks.

heading: 1 INTRODUCTION
text: Embedding is useful for mapping high-dimensional vector to a low-dimensional space for various neural network models such as language models and graph neural networks. Especially in modern language models, a contextual embedding is represented respectively for each contextual unit. For instance, some language models like word2vec (Mikolov et al., 2013) which deal with a whole word as one embedding vector, suffer from the Out-of-Vocabulary(OOV) problem due to the variety of words. Later language models (Bojanowski et al., 2017) subdivide words into multiple segmented words. To deal with the subdividing, various tokenizers (Sennrich et al., 2016; Kudo & Richardson, 2018) are proposed to construct common tokens based on word corpuses. However, each token contains a different context, and a good criterion for dividing the token is not clear. There are some cases that the atomic-level tokens share notable common properties. For example, two tokens 〈A〉 and 〈a〉 are assigned to different embeddings, but they both are vowels and the same alphabet. In the case of combined characters(i.e., Hangul of Korean character) or ideograms(i.e., Chinese and Japanese characters), some tokens might include more common meanings than other languages. The tokens of similar meanings tend to locate closely in embedding space when the language model is trained with those tokens. In this paper, we propose a novel embedding structure which replaces a word embedding with several sub-embeddings. If a token consists of several contextual elements, a sub-embedding can be assigned to each element to constitute the original embedding. An embedding vector shares its learning parameters with other embedding vectors because of the nested allocated sub-embeddings. Various scattered sub-embedding structures can be generated depending on structural topology of the sub-embeddings. Firstly, we sequentially allocate sub-embeddings through module operation, and conduct Cartesian product with sub-embeddings to build embedding vectors(Figure 1). This could robustly map the sub-embeddings, but each sub-embedding does not reflect the context of each token. To address this problem, we modify the scattering algorithm by clustering the tokens in the embedding space using a pretrained network. Notable factors of this paper are as follows:
1. The number of learning parameters in embedding part is dramatically reduced by Cartesian product.
2. The language models are free from OOV problem through sub-embedding.
3. The proposed embedding structure can be applied to most existing language models easily by replacing only the input embedding part.
We evaluate our sub-embedding structure on English and multilingual downstream tasks. We borrow RoBERTa (Liu et al., 2019) structure for the standard English downstream tasks. In addition, XLMR (Conneau et al., 2020) is used to test the performance of sub-embedding in multilingual tasks. We show that the network replaced by the k-sub-embedding compresses the embedding over 99.9% while the accuracy on multilingual benchmark drops only 1.4%p. Finally, we demonstrate how the sub-embedding is trained by visualizing the distribution of k-sub-embeddings.

heading: 2 RELATED WORK
text: Distributed Representations of Word Embedding. Word embedding is commonly used in the language models to represent the word in the latent space. Distributed representation is mainly used instead of a one-hot vector for the efficiency. Despite the difficulty of learning distributed representation, word2vec (Mikolov et al., 2013) is able to map words to embedding space successfully and shows context analogies of each word embedding. In the case of word2vec, the vocabulary is composed with the words in the input corpus, it is easily suffered from OOV problem. To deal with this problem, some language models (Pennington et al., 2014; Bojanowski et al., 2017) split the words into subtokens and learn the co-occurrence of words. Furthermore, ELMo (Peters et al., 2018) gathers hidden states and the embeddings of the bidirectional language models to represent contextual embeddings. Around the same time, language models (Devlin et al., 2019; Liu et al., 2019; Lan et al., 2020) based on transformer (Vaswani et al., 2017) are proposed to learn the context of entire input sequence with the attention mechanism. In this paper, we split the contextual embeddings into sub-embeddings that contain common context of the embeddings.
Relationship Between Tokenizers and Language Models. Language Models usually split a sequence into tokens through tokenizers. Classical tokenizers are designed to divide a sequence into contextual units such as words, characters, or morphs. Those tokenizers are intuitive to implement,
but they are easily suffered from OOV problem and need special knowledge to split into morphs. To alleviate the problem mentioned above, some works propose that the tokenizers learn the input corpus to construct concrete vocabularies. Byte-Pair Encoding (Sennrich et al., 2016) iteratively merge tokens to a larger token, byte-level tokens are used to cover all input cases (Radford et al., 2019). On the other hand, Kudo (2018); Kudo & Richardson (2018) iteratively reduce meaningless tokens from a large token set. We can build up the vocabulary robustly using these tokenizers, but the contextual unit can be different to each token. Typical embedding scheme maps the token to each embedding vector, CharformerTay et al. (2021) pinpoints this inflexible problem. They propose the gradient-based method to figure out latent subword representation. In contrasts to other static tokenizers, the embedding of token is changed by subword block. In addition, Clark et al. (2021) alleviates the tokenization framework, the tokenizer is changed to character-level without human knowledge. Xue et al. (2021) proposes token-free models which encode input sequence with contextual unit, not tokens. In this paper, we propose the embedding structure regardless of tokenizers. Unlike token-free models (Xue et al., 2021), the our k-sub-embedding structure needs a tokenizer, but the networks do not suffered anymore from the number of tokens. Since the embeddings act a prominent role in the language models, a series of studies come up with simplifying the embeddings. One of the lightening methods is to compress the embedding weight with matrix factorization. Lan et al. (2020) conducts matrix factorization to the embedding part, and the high-dimensional embedding vector could be compressed suceessfully to 128-dimensional vector. Our proposed structure also reconstruct the embedding through lookup operation without additional computations.

heading: 3 PROPOSED METHODS
text: We investigate the requirements of each sub-embedding, and suggest (1) randomly scattered k-subembedding with naive approach; (2) clustered k-sub-embedding using contextual knowledge.

heading: 3.1 PRELIMINARIES
text: Modern language models suffer from massive parameters of embeddings because the number of embeddings is associated directly with the diversity of output tokens. Some studies tried to compress the embedding part; Lan et al. (2020) lightened the embedding weight via matrix factorization. We horizontally split embedding vectors into sub-embedding vectors which are shared with some other embedding vectors. In the proposed embedding structure, sub-embeddings tend to highly correlate. We suggest the following requirements to verify the modified embeddings perform like the original embedding.
1. (Uniqueness of divided embeddings) Let ei, ej be embedding vectors, and {eki }Kk=1, {ekj }Kk=1 are their sub-embeddings, ∀i, j ∈ {1, . . . , N} ∃k∗ ∈ {1 . . .K} s.t. ek ∗ i 6= ek ∗
j and i 6= j.
2. (Contextual mapped sub-embedding) If a pair of tokens have similar contextual meanings, their sub-embeddings share more parts than an arbitrary pair.
We try to find the function to map the origin embedding vectors into sub-embedding vectors. In this work, we suggest the bijective function F : N → M×· · · × M for converting the embedding to each sub-embedding where N = {1, . . . , N} ⊂ N is a set of the embedding index and M = {1, . . . ,M} ⊂ N is a set of each sub-embedding index. To expand the function to allocate sub-embedding in several ways, we generalize the mapping function using Cartesian product of functions:
F(n) = (f1 × f2 ×· · · × fk)(n, . . . , n︸ ︷︷ ︸ k ), (1)
where the function fk : N → M denotes k-th sub-embedding index. Since the cardinality of k-ary Cartesian product is Mk, the embeddings can be covered only N 1 k sub-embeddings. This makes the number of embedding parameters dramatically reduce with log scale. We suppose that the dimension of the embedding d is distributed equally to sub-embedding M = d/k. Then, the
number of the embedding parameters N × d is scaled down to k ×M × (d/k) = M × d. Finally, the representation of the embedding would be replaced by
en = K⊕ k=1 efk(n),
where ⊕
is a concatenate operator, and en, efk are the corresponding embedding vector of the embedding index. We suggest Algorithm 1, 2 by modifying fk that satisfies the requirements.
3.2 RANDOMLY SCATTERED k-SUB-EMBEDDING
We construct naively fk to combine sub-embeddings through a Cartesian product which can generate up to Mk embeddings. To ensure generated embeddings to be distinct, the number of each sub-embedding M should be larger than N1/k. Randomly allocated sub-embedding algorithm(Algorithm 1) shows that whole embeddings are generated by repeatedly applying the modulo operation. We set M = dN 1K e to extract the most compressed sub-embeddings. The modulo operation is derived with the perspective of M -base number. Converting to M -base number is bijective, and we can easily set the index of each sub-embedding as each digit of the radix.
Algorithm 1 Randomly allocated Sub-Embedding 1: Input: Number of the embeddings N , embedding dimension d, sub-embedding sets K 2: M ← dN 1K e . number of each sub-embedding 3: Initialize k-th M sub-embedding vectors {ekm ∈ R d k }Mm=1 for all k ∈ {1, . . . ,K}
4: for n = 1, 2, . . . , N do 5: for k = 1, 2, . . . ,K do 6: fk(n) ≡ (n/Mk−1) mod Mk, . k-th digit of the M -base number 7: end for
8: en = K⊕
k=1
ekfk(n)
9: end for 10: Output: The combined embedding vectors {en}Nn=1.
3.3 CLUSTERED k-SUB-EMBEDDING
The language models based on transformer learn the whole context of the input sequence, and each embedding vector of a token is mapped to the embedding space reflecting its context. Mikolov et al. (2013) recognizes that arbitrary two closely mapped word vectors tend to have similar context. If the context of each token is given, we can enhance the allocation heuristic using the contexts. In the case of tokens that have a similar meaning, we assume that the two tokens can be classified with smaller differences. In this case, we can allocate more sub-embeddings to be shared. To estimate the similarity of each pair of the tokens, a pretrained network is used to get each L2 distance between each embedding vector. Assuming that all sub-embeddings are independently initialized randomly, the tokens that share more sub-embedding are expected to have less L2 distance. The method of allocating sub-embedding to similar tokens is based on k-means (Arthur & Vassilvitskii, 2007) algorithm. Each embedding vector is treated as an instance of the k-means algorithm, thus the algorithm is adjusted recursively to each sub-embedding space. The recursive clustering algorithm aims to separate the instances which are allocated in some identical sub-embeddings. According to Algorithm 2, the mapped sub-embeddings can satisfy the second requirement due to the k-means algorithm based on L2 space.

heading: 4 EXPERIMENTS

heading: 4.1 EXPERIMENTAL SETUP
text: There are a few variant language modelings such as translation language modeling(TLM), casual language modeling(CLM), and MLM, that depends on the purpose of each modeling. We replace the
Algorithm 2 Allocating Clustered Sub-Embedding 1: Input: Number of embeddings N , number of sub-embeddings M , embedding dimension d,
number of sub-embedding sets K, embeddings of pretrained network P = {ρn}Nn=1 2: Initialize k-th M sub-embedding vectors {ekm ∈ Rd/K}Mm=1 for all k ∈ {1, . . . ,K} 3: fk(n)← 0, ∀k = 1, . . . ,K, n = 1, . . . , N . Initialize the labels to zero 4: for k = 1, 2, . . . ,K do 5: extract unique tuples from {F(n)}Nn=1 . Equation 1 6: for unique F(n∗) in {F(n)}Nn=1 do 7: if k 6= K then 8: PF(n∗) ← {ρn : F(n) = F(n∗)}Nn=1 9: Adjust k-means algorithm to PF(n∗)
10: Labeling the results to fk(n) where F(n) = F(n∗) 11: else 12: fk(n)← random number among M candidates where F(n) = F(n∗). 13: end if 14: end for 15: end for
16: Gather en = K⊕
k=1
ekfk(n) ∀n ∈ N
17: Output: The combined embedding vectors {en}Nn=1.
word embeddings of MLM with the sub-embedding to inspect the effect of our proposed embedding structure.

heading: 4.1.1 DATASETS
text: Common MLM is trained from a large plain text dataset and then fine-tuned in downstream tasks. The language models are trained primarily in monolingual dataset. Additionally, the multilingual dataset is trained to verify the k-sub-embedding also works within large vocabularies. BooksCorpus (Zhu et al., 2015) and English WIKIPEDIA corpuses are used for the monolingual set as following BERT (Devlin et al., 2019). In the case of the multilingual experiments, we extract COMMONCRAWL (Wenzek et al., 2020) corpuses written in 15 languages which defined in XNLI(Conneau et al., 2018). More details of the dataset are summarized in the appendix.

heading: 4.1.2 CONFIGURATIONS OF THE LANGUAGE MODELS
text: We modify RoBERTa (Liu et al., 2019), XLM-R (Conneau et al., 2020) implementations in HuggingFace (Wolf et al., 2020) framework. RoBERTa is followed BERT approach except for NSP prediction, and optimize hyperparameters such as momentum value and learning rate. We borrow the hyperparameters of RoBERTa, also we adapt MLM prediction which has a 0.15 probability of masking token. In order to simplify the network scheme, we set the base network to be composed of 8 transformer encoder layers and 512-d embeddings in contrast to BERTBASE. In similar fashion for the multilingual case, XLM-R is modified to 8 layers like predefined RoBERTa. The networks that we modified are denoted as RoBERTaMEDIUM, and XLM-RMEDIUM as shown in Table 1. We present randomly allocated case of k-sub-embeddings in the table. The number of embedding parameters in the modified networks is reduced remarkably from RoBERTaMEDIUM, and XLM-RMEDIUM. Other training settings(e.g., tokenizers) are followed as (Liu et al., 2019; Conneau et al., 2020).

heading: 4.1.3 OTHER EMBEDDINGS IN THE LANGUAGE MODELS
text: The language models (Devlin et al., 2019; Liu et al., 2019; Yang et al., 2019; Lan et al., 2020) are built on common transformer backbones, but their criterions are slightly different. In this study, we construct the network as MLM structure not using NSP. Token prediction models such as MLM needs a decoder from language models to predict token representation. Many language models tie the last output weights to input embedding weights for robustness, Chung et al. (2021) shows that not sharing between the embeddings and the decoder also can be performed better by regulating the number of features in the decoder. Empirically, the result of decoupling the embedding and
the decoder is not different from coupling weights, we trained the language models with coupling decoders. We replace the embedding part with the previous network into sub-embedding structure. There are additional embeddings to capture external information such as positional embeddings and token type embeddings, but we do not replace those embeddings to the sub-embedding.

heading: 4.1.4 EVALUATION BENCHMARKS
text: The modified neural language models are evaluated by GLUE (Wang et al., 2019) benchmarks which are consisted of single-sentence tasks, similarity and paraphrase tasks, and inference tasks.1 We also test multilingual language models with XNLI(Conneau et al., 2018) benchmark. XNLI is constituted with 15 monolingual corpuses for Natural Language Interface(NLI). Following Huang et al. (2019), we fine-tune the pretrained XLM-R in two ways, cross-lingual task and multi-language task.

heading: 4.2 COMPARING THE RANDOMLY ALLOCATED SUB-EMBEDDINGS
text: We construct the base network as in (Liu et al., 2019), and we replace the input embedding part to randomly scattered k-sub-embedding structure(Algorithm 1). Table 2 shows that the results on GLUE benchmarks of base network and k-sub-embedding. As the number of sub-embedding is defined by M = N 1 k , it needs only 4 sub-embeddings when k is 8 for the given N is 50627. The results on GLUE among k-sub-embedding networks are similar between each other. However, the results underperform compared to RoBERTaMEDIUM, it seems that randomly allocated sub-embeddings could not overcome the extremely entangled embedding part. The performance might decrease because we do not consider some special tokens(e.g. separate token, and padding token) when the sub-embeddings are allocated. To alleviate this problem, we introduce clustered allocating procedure based on the pretrained knowledge.

heading: Model k E SST-2 MNLI QNLI QQP RTE MRPC CoLA STS-B
text: 1single-sentence tasks contain CoLA (Warstadt et al., 2018), SST-2 Socher et al. (2013), similarity and paraphrase tasks contain MRPC (Dolan & Brockett, 2005), STS-B (Cer et al., 2017), QQP (Shankar et al., 2017), and inference tasks contain MNLI (Williams et al., 2018), QNLI (Rajpurkar et al., 2016), and RTE (Dagan et al., 2005)

heading: 4.3 ALLOCATING SUB-EMBEDDING FROM PRETRAINED KNOWLEDGE
text: Randomly scattered k-sub-embeddings compress successfully the original embeddings, but in some cases, the performance is struggled due to highly entangled sub-embeddings. Despite the tokens locate closely in the embedding space, randomly allocating algorithm cannot reflect the context between them. To solve this context mismatching problem, we extract pretrained RoBERTa (Liu et al., 2019) network from HuggingFace2. The 768-d embedding vectors of the pretrained network are clustered with Algorithm 2. We set the number of sub-embedding M to 50, 100, and 200 to allocate 3-sub-embeddings. Table 3 shows that advanced results on GLUE benchmarks. The original k-means algorithm deals with the instances regardless of cluster size, i.e. the cluster size can be different across k clusters. We perform both naive k-means and equally assigned k-means due to Algorithm 2. The results show that sub-embeddings with equal cluster size outperforms on small embedding set. Surprisingly, even clustered 3-sub-embedding has fewer parameters than 2- sub-embedding network, it outperforms every GLUE benchmark. The proposed method performs comparably with the original embedding structure, and outperforms on SST-2 benchmark.

heading: 4.4 TOKENS IN SUB-EMBEDDING SPACE
text: We also perform PCA to visualize k-sub-embeddings on the embedding space and derive the variance of the embeddings. The embedding weight E ∈ RN×d would be transfered to Ep ∈ RN×p where p < N is the number of main components according to PCA procedure. The explained variance ratio can be defined with rp = Σ p i=1σi/Σ d i=1σi where σi is i-th largest eigenvalue of E’s covariance matrix. It is used to measure the variance of factorized embedding space. We visualize 3-d mapped k-sub-embedding vectors and corresponding explained variance ratio in Figure 2. The sub-embedding vectors are easily factorized due to sharing parameters, so the instances are located in each isolated subspace.

heading: 4.5 COMPARING INSIDE OF THE EMBEDDING AND THE SUB-EMBEDDING
text: We found that k-sub-embedding is able to be performed comparably on some benchmarks, but it is still ambiguous that how the combined sub-embedding works like the embeddings. Inspired by (Mimno & Thompson, 2017; Ethayarajh, 2019; Cai et al., 2021), we calculate the inter-similarity and intra-similarity among the embedding vectors. Cai et al. (2021) captures all hidden states between each layer of the neural language model to address inter-type and intra-type cosine similarities. Likewise for early studies, we define the inter-similarity Sinter(l) = Ei 6=jcos(hli,hlj) and intra-similarity Sintra(l) = EiEj1 6=j2cos(hlj1 ,h l j2
) where hl∗ is one of the l-th hidden states. The cosine similarity indicates the isotropy of the embedding space. We report inter-similarity and intrasimilarity of each layer in Figure 3. The inter-similarities of each k-sub-embedding network are reported almost 1.0 at the end of the layer. This indicates that each hidden state of token is located in narrow cone in the latent space because the embeddings share their sub-embeddings.
2https://huggingface.co/roberta-base
Figure 4 shows more details of hidden states in each layer. We extract some commonly used tokens(e.g., 〈the〉, 〈to〉, 〈and〉, 〈of〉, and 〈a〉) to figure out where they locate in the hidden spaces. The embedding vectors tend to be clustered each other due to the shared sub-embedding as shown in Figure 2. Figure 4(a), 4(b) show that the networks try to distinguish the tokens by spreading out the tokens into the hidden spaces. The last hidden states congregate the tokens for the decoder of the language models.

heading: 4.6 EXPERIMENTS ON MULTILINGUAL DATASET
text: Training multilingual language models treats as much harder than monolingual case. Some multilingual language models (Lample et al., 2018; Lample & Conneau, 2019) focus on cross-linguity to train the pairs of translation dataset. XLM-R (Conneau et al., 2020) modifies RoBERTa to deal with multilingual embeddings, that leverages on extremely large dataset. We adapt the XLM-R network
to XLM-RMEDIUM which is described in Table 1. Following (Huang et al., 2019), We evaluate the multilingual networks in two ways; cross-lingual transfer and multilingual transfer(translate-train-all). Table 4 shows the accuracy on each of the 15 XNLI languages. The cross-lingual transfer results of Arabic and Urdu are underperformed due to lack of pretraining corpora. While randomly allocated sub-embedding cut down the number of embedding over 99.97%, the performance of each language is also diminished. We allocate subembeddings twice as much than the previous case to adjust clustering algorithm using the pretrained network3. The results on XNLI are improved 2%p on multilingual transfer task, while the embeddings are still compressed over 99.95%.

heading: Model E en fr es de el bg ru tr ar vi th zh hi sw ur Avg

heading: 5 CONCLUSIONS AND FUTURE WORK
text: In this work, we introduce the k-sub-embedding to replace the original embedding. We suggest two ways to allocate shared sub-embedding to the embedding vector; one is to assign sequentially by modulo operation(Algorithm 1), and the other is allocating scattered sub-embedding using contextual information from pretrained network(Algorithm 2). The number of parameters of the scattered sub-embedding is reduced over 99%, because the combined embedding can be generated exponentially as a result of Cartesian product. Although the embeddings are highly compressed, the results on GLUE, XNLI show that k-sub-embedding structure also performs similar with the base result. We conducted the experiments to replace the embeddings to k-sub-embedding through MLMs. Other language modelings such as TLM and CLM can be also trained with the replaced embeddings. Although TLM has additional decoder to translate the source language, we expect that the contextual embedding replaced to k-sub-embedding would be outperformed according to cross-lingual test results. We can further investigate about the relationship between the embedding dimension d and the number of sub-embedding space k. This will help to theoretically understand how the k-subembedding works.
3https://huggingface.co/xlm-roberta-base

heading: REPRODUCIBILITY
text: The embedding part of the language model is usually defined in Embedding layer. We design our k-sub-embedding layer to be easily replaced with the predefiend embedding layer. The implementations of k-sub-embedding are written in PyTorch framework, and it is also compatible with HuggingFace framework. Source code is accessible to the supplementary material.

heading: A APPENDIX

heading: A.1 PRETRAINING MONOLINGUAL LANGUAGE MODELS
text: We adapt RoBERTa case for training standard English corpora. The structure of RoBERTaBASE is based on BERTBASE which has 12 layers of transformer encoder. We cut down some layers to build 8 layers, the details of other hyperparameters are described in Table 5. The k-sub-embedding networks are also trained with the same environment of RoBERTaMEDIUM, and we accelerate training speed through half-precision floating point. The fine-tuning tasks are trained with the hyperparameters that are used in pretraining. Some values are modified to fit in GLUE benchmark as Table 6. We conduct the fine-tuning tasks and report the best accuracy among the candidates of hyperparameters. We also fine-tune RTE, STS-B, and MRPC tasks from the network checkpoint of MNLI downstream task.

heading: A.2 PRETRAINING MULTILINGUAL LANGUAGE MODELS
text: As we deal with the multilingual corpus with XLM-R framework, the pretraining procedure of multilingual case is not different from the monolingual case. To cover the tokens of multiple languages, XLM-R expands the vocabularies to 250k. We also adapt these tokenization method and other hyperparameters as described in Table 5. The hidden size and layers are reduced to 512 and 8 likewise for RoBERTaMEDIUM. We extract corpus of 15 languages from CommomCrawl. The details of each monolingual corpus are described in Table 7. Unlike the MNLI benchmark, we fine-tune XNLI task on fixed learning rate and batch size(Table 6).

heading: A.3 LAYERWISE ANALYSIS
text: In this section we analyze each hidden state of k-sub-embedding. The embeddings that are composed with the sub-embeddings tend to be highly correlated. The language models learn to distinguish the entangled embeddings. Figure 5, 6, 7, 8, 9, and 10 show that the tokens are spread out through the hidden space. The tokens in the last layer are located closely each other.

heading: ISO code Language Tokens(M) Size(GiB)

heading: Hyperparameters GLUE XNLI

heading: Hyperparameters RoBERTaMEDIUM XLM-RMEDIUM

Review summaries:
This paper proposes a method for reducing the memory footprint of word embeddings by representing each word not as its own unique embedding vector, but rather as a concatenation of K lower-dimensional sub-embeddings, where the sub-embeddings can be shared between different words. By properly choosing K, the number of unique sub-embeddings M, and the dimension of the sub-embeddings, the number of parameters can be reduced drastically. Two techniques are given to map the words to their respective sub-embeddings: one is essentially a uniform mapping which completely ignores any semantic meaning of the words, and the other approach uses k-means clustering to preserve some of the semantic structure present in the original word embeddings. These new word embeddings are then tested on a few tasks. As expected, the performance of the uniform mapping is much worse than that of the original word embeddings, since all semantic structure was lost. The performance of the k-means sub-embeddings approaches that of the original embeddings, while requiring far fewer parameters. 

This paper proposes a method for compressing the word embeddings. Though use cases of the compressing the (word) embeddings are much wider, this paper only focuses on the embeddings in pre-trained language models.  The main idea of the proposed method is to introduce the sub-embeddings shared among vocabulary. In detail, this paper proposes two variants for sharing the sub-embeddings, namely, random and cluster-based methods.  The experiments are conducted on GLUE and XNLI benchmarks. The results show that the proposed method provides to reduce the model size with the small performance degradation.

This paper aims to reduce the size of word embeddings in pretrained language models. The main idea is to replace the lookup table embeddings of size $N$ with only $k$ sub-embeddings of $N^{1/k}$ size and a mapping that allows to combine entries of the sub-embeddings to produce unique embeddings of the same size via concatenation. Two different types of mappings are explored, one random and one derived from clustering word embeddings from a pretrained model. For evaluation, the procedure involves pretraining a language model with the proposed embedding from scratch and then finetune it on downstream tasks. The  results show that the embedding can be reduced by 99.9% while maintaining performance within ~1.4% difference on a XNLI benchmark and within ~4.8% difference on GLUE benchmark.

This paper proposes an embedding compression method which replaces an $N$ x $d$ embedding table with $k$ distinct $M$ x $(d/k)$ embedding tables (where $M = N^{1/k}$). Specifically, the $d$-dimensional embedding for each token in the vocabulary is replaced by $k$ different sub-embedding vectors $(e_i)_{i=1}^k$, each of dimension $d/k$, where $e_i$ is drawn from the $i^{th}$ codebook. It proposes two ways of selecting these sub-embeddings: 1) arbitrarily assigning the sub-embeddings to each token (based on its index), 2) using k-means to do this assignment procedure.  It shows that very high compression rates (e.g., 99.4-99.9% compression) can be attained using this method, while typically only sacrificing 0%-4% absolute accuracy.