Reconstructing Word Embeddings via Scattered $k$-Sub-Embedding

Abstract: The performance of modern neural language models relies heavily on the diversity of the vocabularies. Unfortunately, the language models tend to cover more vocabularies, the embedding parameters in the language models such as multilingual models used to occupy more than a half of their entire learning parameters. To solve this problem, we aim to devise a novel embedding structure to lighten the network without considerably performance degradation. To reconstruct $N$ embedding vectors, we initialize $k$ bundles of $M (\ll N)$ $k$-sub-embeddings to apply Cartesian product. Furthermore, we assign $k$-sub-embedding using the contextual relationship between tokens from pretrained language models. We adjust our $k$-sub-embedding structure to masked language models to evaluate proposed structure on downstream tasks. Our experimental results show that over 99.9$\%+$ compressed sub-embeddings for the language models performed comparably with the original embedding structure on GLUE and XNLI benchmarks.

Reviews and author responses:

 I thank the authors for their thoughtful response to my review.

Regarding the comment: 
>"We think that the novelty of this work can be found in adjusting in modern language models. Unlike the previous studies we test our embedding structure in BERT based models."
- There is previous work that looks at embedding compression in the context of BERT models. For example, in [6] (Section 4.2, and Figure 8 of Appendix E.1), there are extensive experiments comparing the performance of various compression methods on the BERT WordPiece embeddings.

After considering the authors' response, I choose to keep my original recommendation for the paper unchanged.

 Thank you for your comments. To address your comments and questions:

1. OOV problem

The language models can easily suffer from the OOV problem because they are built on the tokenizers as you mentioned. We use tokenizers like other language models, but the tokenizer that we used can simply add new vocabularies by combining sub-embeddings. For this reason, our proposed model is free from the OOV problem.

2. Additional latency

Our proposed method has additional operations such as indexing sub-embedding vectors and concatenating sub-embeddings vectors during training steps. However the operations are defined in GPU and the latency was negligible regarding to other forward pass.

Q1) It is interesting to see that the sub-embeddings group the embedding vectors in a clear way due to parameter sharing. How this form of parameter sharing compares to other simpler ones, say linear factorization?

A1) The main difference between k-sub-embedding structure and linear factorization is the number of sharing parameters. Although linear factorization also compresses plenty of parameters with respect to hidden dimension, it needs hidden states for each  instance(vocabulary in the case of language models). 

Q2) What is the actual overhead of the clustering process required for the mapping?

A1) We did not mention the overhead of the clustering process, but it needs significant time to cluster the pretrained network. The typical language models have more than 50k+ vocabularies and 500+ hidden dimensions. Clustering trained embedding vectors needed 30 minutes ~ 1 hour in our experiments. 

 Thank you for your careful reviews and suggestions of related works. We would like to add some comments to each weakness.
1. Not comparing with or discussing baselines
- We compared the result of our proposed models with our baselines. It needs to compare with other previous studies [1, 4, 5, 8]. We will add additional comparison to follow-up study.
2. Lack of novelty
- The structure of the proposed method is almost same with product quantization [1], while we use different lengths of codebooks of each sub-embedding part. In addition the compression techniques are similar with other studies [2, 4, 8]. We think that the novelty of this work can be found in adjusting in modern language models. Unlike the previous studies we test our embedding structure in BERT based models. 

[1] Jegou, H., Douze, M., and Schmid, C. Product quantization for nearest neighbor search. IEEE transactions on pattern analysis and machine intelligence, 2010.

[2] Martin Andrews. Compressing word embeddings. In ICONIP, 2016.

[3] Joulin, A., Grave, E., Bojanowski, P., and Mikolov, T. Bag of tricks for efficient text classification. EACL, 2017.

[4] Raphael Shu, Hideki Nakayama. Compressing Word Embeddings via Deep Compositional Code Learning. ICLR, 2018.

[5] Chen, T., Min, M. R., and Sun, Y. Learning k-way ddimensional discrete codes for compact embedding representations. ICML 2018.

[6] May, A., Zhang, J., Dao, T., and Ré, C. On the downstream performance of compressed word embeddings. NeurIPS, 2019.

[7] Oleksii Hrinchuk, Valentin Khrulkov, Leyla Mirvakhabova, Elena Orlova, Ivan V. Oseledets. Tensorized Embedding Layers. EMNLP (Findings) 2020.

[8] Ting Chen, Lala Li, Yizhou Sun. Differentiable Product Quantization for End-to-End Embedding Compression. ICML 2020.

3. Significant drops in accuracy
- As you mentioned the drops in accuracy, some results on multilingual cases were significantly dropped from the baseline. It seems that the highly compressed sub-embedding structure is hard to overcome the original structure. We plan to have further experiments to show better results of the proposed method.
4. Lack of clarity in presentation (especially of Algorithm 2)
- Algorithm 2 uses a pretrained network to map each sub-embedding codebook via k-means algorithm. After mapping all sub-embeddings to the embeddings, all sub-embeddings are initialized randomly and trained with typical language modeling procedures. The codebooks are not changed during training steps. 

 Thank you for your detailed concerns. We would like to add some comments to each weakness.
1. Technical Novelty, Related Work
- We agree with the limitation of technical novelty. Some studies [1-3] deal with compositional embeddings like our proposed model. They optimize the sub-embedding by their own optimization techniques, and we should have referred to those ideas. We will add details about related works to follow-up study.

[1] Suzuki and Nagata, Learning Compact Neural Word Embeddings by Parameter Space Sharing, In Proc of IJCAI-2016, https://www.ijcai.org/Proceedings/16/Papers/292.pdf

[2] Jegou, H., Douze, M., and Schmid, C. Product quantization for nearest neighbor search. IEEE transactions on pattern analysis and machine intelligence, 2010.

[3] Raphael Shu, Hideki Nakayama. Compressing Word Embeddings via Deep Compositional Code Learning.

2.  Experiments
- Our experiments were conducted in different environments of related works. We cannot train the larger model because of the lack of hardware. We tried our best by training models in smaller environments, but still it needs more abbreviation studies. 

 Thank you for your careful review of the whole paper. We agree with most of your concerns, and we would like to add some comments to each weakness.
1. Some of the notations and descriptions can be confusing to understand due to the not well-defined proposed method. We tried to demonstrate the methods with examples, but it rather ruined the explanations.
2. There have been many studies to decrease the number of parameters such as pruning, quantization, etc, and some of those papers are listed in backgrounds. Although their shortcomings were not described, we indirectly pinpointed the previous studies with comparison between our proposed methods.
3. Thank you for the detailed suggestions. It was also confused with "k" in both the k-means algorithm and k-sub-embedding. The notations should be revised.
4. The randomly scattered sub-embeddings showed pool performances, but we focused on the topology of the scattered sub-embeddings structure. Despite the randomly scattered structure mess up the semantic information, some benchmarks(e.g. SST-2) showed quite similar with the base result.
5. Experiment results lack the comparison with other compression studies. We will add more experiment results to follow-up study.
6. We can easily add new vocabularies by combining sub-embeddings. For this reason, our proposed model is free from the OOV problem.

This paper proposes a method for reducing the memory footprint of word embeddings by representing each word not as its own unique embedding vector, but rather as a concatenation of K lower-dimensional sub-embeddings, where the sub-embeddings can be shared between different words. By properly choosing K, the number of unique sub-embeddings M, and the dimension of the sub-embeddings, the number of parameters can be reduced drastically. Two techniques are given to map the words to their respective sub-embeddings: one is essentially a uniform mapping which completely ignores any semantic meaning of the words, and the other approach uses k-means clustering to preserve some of the semantic structure present in the original word embeddings. These new word embeddings are then tested on a few tasks. As expected, the performance of the uniform mapping is much worse than that of the original word embeddings, since all semantic structure was lost. The performance of the k-means sub-embeddings approaches that of the original embeddings, while requiring far fewer parameters.  Strengths:
This paper gives a potentially interesting and novel method for reducing the parameters required for word embeddings. The idea of allowing different word embeddings to explicitly have identical blocks of entries seems potentially promising, as we want the embeddings of related words to be similar in some sense.

Weaknesses:
This paper suffers from a number of significant weaknesses. 
1. Language and editing: The paper needs much editing to improve the language and notation. In several spots, it is difficult to understand what the authors are trying to communicate. The notation is also often inconsistent or not properly defined. Ideas do not flow and overall it requires too much work for the reader to understand what is going on. 
2. Motivation: the main motivation of this work is to decrease the number of parameters needed for word embeddings. However, there are already many works in this direction, and the authors do not explain how their approach addresses any shortcomings of other methods.
3. Confusing explanations: It is difficult to understand exactly how the k-means clustered sub-embeddings work. There are notational issues in Algorithm 2 (mixing up lower-case k with capital K with k-means). The phrase "adjust k-means algorithm to..." is unclear. How are the k-sub embeddings initialized? I wouldn't know how to actually implement this idea given the description in the paper.
4. Unnecessary content: What's the point of the randomly scattered sub-embeddings? This idea clearly ignores the semantic structure of the words, and so it completely misses the point of word embeddings. This is reflected in the poor performance of this approach in the experiments. There is no need to discuss this method.
5. Experiments: this paper should compare the performance of their compressed word embeddings to other approaches that have been proposed in the literature. Since this isn't the first work to address the issue of compression, the experimental results given in this paper are not enough to justify the impact of its contribution.
6. Unsubstantiated claims: The paper claims that their method doesn't suffer from the OOV problem. This is unclear to me and is never explained.  This paper has only limited contributions and suffers from several significant issues, both technical and in the writing quality. It does not meet the standard for publication and therefore I recommend reject. 

This paper proposes a method for compressing the word embeddings.
Though use cases of the compressing the (word) embeddings are much wider, this paper only focuses on the embeddings in pre-trained language models.

The main idea of the proposed method is to introduce the sub-embeddings shared among vocabulary.
In detail, this paper proposes two variants for sharing the sub-embeddings, namely, random and cluster-based methods.

The experiments are conducted on GLUE and XNLI benchmarks.
The results show that the proposed method provides to reduce the model size with the small performance degradation. This current version of this paper has several concerns for publishing as a conference paper.

The following are the list of concerns that should be revised and reconsidered.

#### *** technical novelty
* 1, A paper [1] has already proposed a closely similar approach for splitting embeddings into K sub-embeddings and sharing them over vocabulary.
The motivation of reducing the embedding size is also identical.
Therefore, the idea of incorporating sub-embeddings is not novel, and the claim for the technical novelty is incorrect.
Similarly, the paper has already discussed the capacity of sub-embedding combinations (Cartesian products) that can drastically reduce the embedding parameters.
I think this paper has to cite the paper [1] and need to clearly state the similarity and difference between their method and the proposed method.

[1] Suzuki and Nagata, Learning Compact Neural Word Embeddings by Parameter Space Sharing, In Proc of IJCAI-2016, https://www.ijcai.org/Proceedings/16/Papers/292.pdf

#### *** Related work
* 2, There are so many previous papers that tackle reducing the memory requirements of word embeddings.
Of course, it is not necessary to cite all the papers related to the (word) embedding compression methods.
However, this paper mostly ignores such developments.
I feel that It is not acceptable for a scientific paper.

#### *** experiments
* 3, In the experiments, there are no baselines for reducing the embedding size. For example, quantization and pruning are typical baselines for the compression literature.

* 4, The experiments are only conducted on masked LMs. However, as this paper also pointed out, there are several different types of LMs, such as standard left-to-right (casual) LMs. Therefore, the experiments that support the effectiveness of the proposed method for reducing the memory requirements are limited. 
Additionally, I cannot find any reasons that the authors cannot do experiments on such LMs. I recommend doing experiments on several different LMs to enhance empirical evidence for the effectiveness of the proposed method.

* 5, The model sizes of LMs used in the experiments are relatively small comparing the recent trend of preferring larger LMs.
For example, 
BERT Base (12 layers, approx. 110 million parameters) and BERT Large (24 layers approx. 340 million parameters) are typical MLMs. Still, they are already 2 and 7 times larger than those used in the experiments (50M). 
This fact suggests that it should be better to use larger models to fit typical use cases.

* 6, The experimental results looks not so good; the proposed method often degrades the performance with wide margins.

 This paper has several concerns for publishing as a conference paper, such as low technical novelty because of the missing previous similar method, and weak empirical evaluation that cannot sufficiently support the authors' claims about contributions.

This paper aims to reduce the size of word embeddings in pretrained language models. The main idea is to replace the lookup table embeddings of size $N$ with only $k$ sub-embeddings of $N^{1/k}$ size and a mapping that allows to combine entries of the sub-embeddings to produce unique embeddings of the same size via concatenation. Two different types of mappings are explored, one random and one derived from clustering word embeddings from a pretrained model. For evaluation, the procedure involves pretraining a language model with the proposed embedding from scratch and then finetune it on downstream tasks. The  results show that the embedding can be reduced by 99.9% while maintaining performance within ~1.4% difference on a XNLI benchmark and within ~4.8% difference on GLUE benchmark. Strengths
* The idea of combining k sub-embeddings of small size to produce word embeddings of large size via concatenation is interesting. It allows to reduce the number of parameters and enables a form of parameter sharing that has not been explored before. 
* Simple and more complicated mappings are explored for assigning sub-embeddings to each vocabulary entry. The cluster-based mapping is well-motivated and takes advantage of the word embedding similarity from a pretrained model.
* Defining the word embedding in this way allows to massively reduce the number of parameters allocated for the word embedding in two pretrained models while maintaining the competitive performance on downstream tasks within 1.4% in one of them. 


Weaknesses
* While the idea bears some novelty, the framing with respect to existing alternatives and discussion of prior work is rather poor. There is no mention or comparison to viable alternatives such as linear factorization (a la ALBERT, adaptive inputs [1], or deep factorization [3]) or sparse code learning [2]. The latter study achieves similar levels of compression with a very small degradation (if at all) on sentiment analysis and machine translation. 
* In the introduction, it is mentioned that the language models that use the proposed embedding are free from OOV problem which does not seem right since the sub-embedding structure relies on a tokenizer (section 2). This issue is rather linked to the tokenizer and not the embedding.  
* The writing lacks clarity in several places especially with regards to the empirical claims and the evaluation procedure. Regarding the former, the paper claims comparable performance on both GLUE and XNLI but this is a bit misleading because on GLUE benchmark is far lower. Regarding the latter, it took me a while to figure out that this method requires pretraining the language model from scratch which is a limitation (this is made clear only in the supplementary). 
* Even though this work should be all about efficiency given the memory reduction, there is not a single experiment that reports throughput, latency or some other efficiency metric during training and inference. In particular, I would be curious to know if the mapping and concatenation operations that have to be performed are increasing the latency or reduce the throughput, especially when dealing with long text sequences. 
* There is lack of discussion about the fact that the language model needs to be pretrained from scratch and in addition to that a clustering procedure is required that is dependent on a pretrained language model. Reporting the corresponding training times would be helpful. 
 
Questions:
* It is interesting to see that the sub-embeddings group the embedding vectors in a clear way due to parameter sharing. How this form of parameter sharing compares to other simpler ones, say linear factorization? 
* What is the actual overhead of the clustering process required for the mapping? 

[1] https://openreview.net/pdf?id=ByxZX20qFQ

[2] https://openreview.net/pdf?id=BJRZzFlRb

[3] https://arxiv.org/pdf/1911.12385.pdf Overall, the paper proposes an interesting idea to share parameters across words and reduce the size of the embedding which hasn't been explored in the past with promising results on XNLI task. My main concern is the lack of clarity and issues with the framing, claims, and evaluation. There is no discussion or comparison to alternatives, some of the claims made are weakly or not supported, and evaluation lacks emphasis on efficiency aspects and relevant efficient word embedding baselines. The results are interesting but the claims about comparable performance hold only in one of the two models explored.  

This paper proposes an embedding compression method which replaces an $N$ x $d$ embedding table with $k$ distinct $M$ x $(d/k)$ embedding tables (where $M = N^{1/k}$). Specifically, the $d$-dimensional embedding for each token in the vocabulary is replaced by $k$ different sub-embedding vectors $(e_i)_{i=1}^k$, each of dimension $d/k$, where $e_i$ is drawn from the $i^{th}$ codebook. It proposes two ways of selecting these sub-embeddings:
1) arbitrarily assigning the sub-embeddings to each token (based on its index),
2) using k-means to do this assignment procedure.

It shows that very high compression rates (e.g., 99.4-99.9% compression) can be attained using this method, while typically only sacrificing 0%-4% absolute accuracy.
 Strengths
1. Compressing embedding matrices is an important problem. In additional to word/wordpiece embeddings used by NLP models, many other systems use large embedding tables (e.g., recommendation systems' product/user embeddings) which can benefit significantly from compression.
2. The proposed method attains very high compression ratios.

Weaknesses
1) Not comparing with or discussing baselines:
- The paper does not compare their method with (or even cite) a single embedding compression method in the literature, even though many embedding compression methods have been proposed (e.g., [1-8] below). Furthermore, the proposed algorithm is very similar to product quantization, a classic compression method from the source coding/information theory literature (discussed in [1]); nonetheless, product quantization is not mentioned in the paper. Similarly, the use of k-means in this paper is reminiscent of the Lloyd quantizer (also discussed in [1]), but this method is not discussed either.
2) Lack of novelty:
- The proposed method is not novel, as it appears to be very similar to product quantization.
3) Significant drops in accuracy:
- Although the proposed method attains high compression rates, it often incurs a significant drop in accuracy.
4) Lack of clarity in presentation (especially of Algorithm 2):
- The explanation and pseudocode for Algorithm 2 were very difficult for me to understand. I was unable to understand how this algorithm worked, and this algorithm appears to be one of the primary contributions of the paper. This made it difficult for me to assess the paper as a whole.
- It would be good to clarify whether the $k$ sub-embedding tables are trained end-to-end during pre-training and/or downstream task fine-tuning, or whether they are held fixed during training.

**Missing Citations**

[1] Jegou, H., Douze, M., and Schmid, C. Product quantization for nearest neighbor search. IEEE transactions on pattern analysis and machine intelligence, 2010.

[2] Martin Andrews. Compressing word embeddings. In ICONIP, 2016.

[3] Joulin, A., Grave, E., Bojanowski, P., and Mikolov, T. Bag of tricks for efficient text classification. EACL, 2017.

[4] Raphael Shu, Hideki Nakayama. Compressing Word Embeddings via Deep Compositional Code Learning. ICLR, 2018.

[5] Chen, T., Min, M. R., and Sun, Y. Learning k-way ddimensional discrete codes for compact embedding representations. ICML 2018.

[6] May, A., Zhang, J., Dao, T., and Ré, C. On the downstream performance of compressed word embeddings. NeurIPS, 2019.

[7] Oleksii Hrinchuk, Valentin Khrulkov, Leyla Mirvakhabova, Elena Orlova, Ivan V. Oseledets. Tensorized Embedding Layers. EMNLP (Findings) 2020.

[8] Ting Chen, Lala Li, Yizhou Sun. Differentiable Product Quantization for End-to-End Embedding Compression. ICML 2020.
 I propose rejecting this paper because it
1) does not compare with a single baseline compression method (most important reason for rejection),
2) lacks novelty (very similar to product quantization),
3) often attains significant drops in accuracy, and
4) lacks clarity (especially Algorithm 2).


Meta-review: Overall, the paper proposes an interesting idea to share parameters across words and reduce the size of the embedding which hasn't been explored in the past with promising results on XNLI task. However, all the reviewers agree that the novelty of this paper is not enough. In addition, the clarity and experiments are not sufficient enough too.

Decision: Reject