MorphTE: Injecting Morphology in Tensorized Embeddings
Authors: Guobing Gan
Peng Zhang
Sunzhu Li
Xiuqing Lu
Benyou Wang
Venue: NeurIPS 2022 Accept (2022)
Keywords: word embeddings
compression
morpheme
tensor product

Abstract: In the era of deep learning, word embeddings are essential when dealing with text tasks. However, storing and accessing these embeddings requires a large amount of space. This is not conducive to the deployment of these models on resource-limited devices. Combining the powerful compression capability of tensor products, we propose a word embedding compression method with morphological augmentation,  Morphologically-enhanced Tensorized Embeddings (MorphTE). A word consists of one or more morphemes, the smallest units that bear meaning or have a grammatical function. MorphTE represents a word embedding as an entangled form of its morpheme vectors via the tensor product, which injects prior semantic and grammatical knowledge into the learning of embeddings. Furthermore, the dimensionality of the morpheme vector and the number of morphemes are much smaller than those of words, which greatly reduces the parameters of the word embeddings. We conduct experiments on tasks such as machine translation and question answering. Experimental results on four translation datasets of different languages show that MorphTE can compress word embedding parameters by about $20$ times without performance loss and significantly outperforms related embedding compression methods.

MorphTE: Injecting Morphology in Tensorized Embeddings
Guobing Gan', Peng Zhang', Sunzhu Li', Xiuqing Lu', and Benyou Wang Cooleeee of Intelligence and Computing, Tianjin University, Tianjin, China Schooo of Data Science, The Chinese University of Hong Kong, Shenzhen, China
3 SShenzhen Research Institute of Big Data, The Chinese University of Hong Kong, Shenzhen, China {ganguobing, pzhang, 1 lisunzhu, (aanzhu,vxiuiinghll @tx edd...u....uu................... edu. wangbenyou@
Abstract
In the era of deep learning, word embeddings essential when dealing with text tasks. However, storing and accessing these embeddings requires large amount space. This is not conducive the deployment of these models devices. Combining the powerful compression capability of tensor products, we propose word embedding compression method with morphological augmenta- tion, Morphologically-enhaan eehhnnced Tensorized Embeddings (MorphTE). A word consists of one more morphemes, the smallest units that bear meaning have a grammatical function. MorphTE represents word embedding as an entangled form of its morpheme vectors via the tensor product, which injects prior semantic and grammatical knowledge into the learning of embeddings. Furthermore, the dimensionality of the morpheme vector and the number of morphemes are much smaller than those of words. which greatly reduces the parameters of the word embeddings. We conduct experiments on tasks such as machine translation and question answering. Experimental results on four translation datasets of different languages show that MorphTE can compress word embedding parameters by about 20 times without performance loss and significantly outperforms related embedding compression methods.
ssource-limited
Introduction
The word embedding layer is key component of the neural network models in natural language processing (NLP). It uses an embedding matrix to map each word into dense real-valued vector. However, when the vocabulary size and word embedding size (dimensionality) are large, the word embedding matrix requires a large number of parameters. For example, the One Billion Word task of language modeling 8 has a vocabulary size ((V)) of around 800K. Besides, the embedding size (d) can range from 300 to 1024 [32 II 23 Storing and accessing the VV| x d embedding matrix requires large amount of disk and memory space. This limits the deployment of these models on such devices having limited resources. To resolve this issue, there many studies compressing embedding layers [34 [5] 29 They can be roughly divided into two lines: product quantization- based and recomposition-based methods. The product quantization-bast based methods [34] 20 38 mainly utilize the compositional coding for constructing the word embeddings with fewer parameters, and it needs to introduce additional task to learn the compact code for each word.
The decomposition- based word embedding compression methods are mostly based on low-rank matrix factorization [18 9 and tensor decomposition [15 29 Utilizing low-rank matrix factorization,
"Corresponding Author "Code available at URL: htt.t//wwwggggggioooooo/ggggggag
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

un
Table 1: Phenomena in word formation.
unkindly
unkindness
Phenomenon
Example
cook+s, cook+ed, cook+ing cold+er, cold+est
Inflection
kind
un+like, un+like+ly im+posstible, im+posstibly
Derivation
ly
ness
police+man, post+man cuttle+fish, gold+fish
Compounding
Figure I: Morphemes of "unkindly" and "unkind- ness".
replaces the embedding matrix with the product of two small matrices. The tensor decomposition is widely used for parameter compression [5] 22 29 Inspired by quantum entanglement, Word2ket and Word2ketXS embeddings are proposed 29 Specifically, Word2ket represents a word embedding as an entangled tensor multiple small vectors (tensors) via the tensor product. The entangled form is essentially consistent with Canonical Polyadic decomposition [16]. Decomposition-be based methods approximate the original large word embedding matrix with multiple small matrices and tensors. However, these small matrices tensors have specific meaning and lack interpretation. and the approximate substitution with them often hurts the model performance in complicated NLP tasks such as machine translation [5
ALBERT
18
In this study, we focus on high-quality compressed word embeddings. To this end, we propose the Morphologically-enheee -echanccde Tensorized Embeddings (MorphTE), which injects morphological knowledge tensorized embeddings. Specifically, MorphTE models the embedding for word as the entangled form of their morpheme vectors via tensor products. Notably, the quality of word embeddings can be improved by fine-grained morphemes, which has been verified in literature
The benefits of introducing the morphology of morphemes in MorphTE can be summed up in two points. (1) word consists morphemes which are considered be the smallest soaniiggbeearing or grammatical units of a language [26]. As shown in Figure [ the root 'kind' determines the underlying meanings of 'unkindly' and 'unkindness" The affixes 'un', 'ly', and 'ness' grammatically refer to negations, adverbs, and nouns, respectively. In MorphTE, using these meaningful morphemes generate word embeddings explicitly injects prior semantic and grammatical knowledge into the learning of word embeddings. (2) As shown in Ta linguistic phenomena such as inflection and derivation in word formation make morphologically similar words often semantically related. In MorphTE, these similar words can be connected by sharing the same morpheme vector.
Mo only needs train and store morpheme vectors, which smaller in embedding size and vocabulary size than original word embeddings, leading to fewer parameters. For example, a word embedding of size 512 can be generated using three morpheme vectors size 8 via tensor products. In addition, since morphemes are the basic units of words, the size of the morpheme vocabulary smaller than the size of the word vocabulary. To sum up. MorphTE can learn high-quality and efffceenncc word embeddings, combining the prior knowledge of morphology and the compression ability of tensor products.
We conducted comparative experiments on machine translation, retrieval-basee question answering, and natural language inference tasks. Our proposed MorphTE achieves better model performance on these tasks compared to related word embedding compression methods. Compared with Word2ket, MorphTE achieves improvements of 0.7, 0.6, and 0.6 BLEU scores on De-En, En-It, and En-Ru datasets respectively. In addition, on 4 translation datasets in different languages, our method can maintain the original performance when compressing the number of parameters of word embeddings by more than 20 times and reducing the proportion of word embeddings to the total parameters approximately from 30% to 2%, while other compression methods hurt the performance.
The main contributions of our work can be summarized as follows:
We propose MorphTE, novel compression method for word embeddings using the form of entangled tensors with morphology. The combination of morpheme and tensor product can compress word embeddings in terms of both vocabulary and embedding size.

---

MorphTE introduces prior semantic knowledge in the learning of word embeddings from fine-grained morpheme perspective, and explicitly models the connections between words by sharing morpheme vectors. These enabled it to learn high-quality compressed embeddings. Experiments multiple languages and tasks show that MorphTE can compress word embedding parameters over 20 times without hurting the original performance.
Related Work
Embeddings. Related works 244 31 10 propose to improve the quality of word embeddings by integrating morphological information. Representing word embeddings as the sum of morpheme and surface form vectors has been employed in several studies [5] 21. Morphological RNNs [ learns word representations using morphemes as units of recursive neural networks [35. Our proposed MorphTTE also utilizes the information of morphemes and is a decomposition-balt based word embedding compression method, similar to Word2ket [29].
Morphologically-angneee
Decomposition- based Compression Embeddings. Decomposition based methods are either based on low-rank matrix factorization [T8
36 Based 18 simply approximates the embedding matrix by the product of two small matrices. GroupReduce 9 and DiseBlock [19] perform a fine une-grained matrix factorization. They first block the word embedding matrix according to the word frequency and then approximate each block. Notably, the method based on matrix factorization has low-rank bottleneck, and its expressive ability is limited under the condition of a high compression ratio [37]
1 21 19| or tensor decomposition [5
on low-rank matrix factorization, ALBERT
As for the tensor decomposition, TT embeddings [15 uses the Tensor Train decomposition 27 to approximate the embedding matrix with several 2-order and 3-order tensors. TT-Rec [44) improves TT embeddings in terms of implementation and initialization to fit the recommended scenario. Word2ket (29] represents a word embedding as an entangled tensor via multiple small vectors. It essentially exploits Canonical Polyadic decomposition [6. I7]. Word2ketXs [29] is similar to
Word2ket, but it compresses embeddings from the perspective of all words rather than individual
words. In addition, KroneckerBERT [36] uses Kronecker decomposition to compress the word embeddings, and the form of Kronecker Embeddings is consistent with Word2ket (29) with an order of 2. Unlike these compression methods, our MorphTE utilizes meaningful morphemes as basic units for generating word embeddings, rather than vectors or tensors with no specific meaning.
3 Preliminary
3.1 Tensor Product Space and Entangled Tensors
A tensor product space of two separable Hilbert spaces V and W is also a separable Hilbert space H. which denoted as H = V @ W. Suppose (,,,,..., vhy basis in V and W, respectively. The tensor product of the vector e = Σラ= eyrby E V and e =
and {di,..., ) are the orthonormal
Ehe ER E Win is defined as follow:
c
Cyeathy @ dx
(I)
j=1
) The set (1), @ Φk jk forms the orthonormal basis in H. and the dimensionality of is the product (gh) of dimensionalities of / and W. This tensor product operation can be simplified as the product of the corresponding coefficients as follow:
2 [c1, C,...., col @ [e1, e2,.,,,..,eel en]
Cieh, ..., Cyei, Cge2. Cgen)
The cumulative tensor product space of the following form is said to have a tensor order of n, and the dimensionality of cumulative tensor product space is the cumulative product of its subspace dimensionalities. See Appendix B for concrete examples of the cumulative tensor product of multiple vectors.
H = Hi ② H2... . Hn.
(3)
3

---

Considering the n-order tensor product space j=1 Mj. vectors of the form v = -j-1% where Uj E Hy, are called simple tensors. In addition, vectors need to be represented as the sum of multiple simple tensors are called entangled tensors. Tensor rank of a vector is the smallest number of simple tensors that sum up to e.
3.2 Tensorized Embeddings with Tensor Product
Embedding
Tensor products have been introduced to learn parameter- efficient word embeddings in KroneckerBERT 36 and Word2ket [2 As shown in Figure 2 Word2ket 29 rep-
OOOOOOOO
resents the embedding V E R of a word as an entangled tensor of rank r and order follow:
(4)
Vi
where Eyt E RV and E R9", Word2ket only needs to store and use these small vectors Fig to generate a large word embedding. If gf > d, the excess part of the generated embedding will be cut off. Therefore, setting q" = can avoid the waste of parameters caused by clipping, and the number of embedding parameters for a word is reduced from d to rn Vd. For example, when d 512, 4 n 3. and 2, the number of parameters of word embedding can be reduced from 512 to 48.
Figure 2: Word2ket embedding with rank
of r = 2 and an order of n 3.
Methodology: Morphologically-enhanee Tensorized Embeddings
In this section, we first discuss the rationale for introducing morphology in the embedding com- pression. Then, we propose MorphTE, a morphologically--enaanee word embedding compression method based on the tensor product. Finally, we show the detailed workflow of MorphTE.
4.1 Motivation to Introduce Morphology
To achieve compression, existing decomposition based word embedding compression methods [8 15] series of small vectors tensors to generate large word embeddings, as shown in Figure 2
These methods are not only uninterpretable as their small tensors do not have specific meaning [15 29], but also lack lexical knowledge. We argue that, in resource-limine limited scenarios like compression, knowledge injection is much more critical than common scenarios. Since with a significant amount of parameters in common scenarios it could to an easier extent learn implicitly such knowledge in data-driven way, which is also one of the objectives for neural networks. However, in compression, it is more beneficial inject explicit knowledge compensate for inferiority in parameter scales, therefore underscoring the importance of knowledge injection in compression.
From reductionism point of view, words might not be the smallest unit for some languages; for example unfeelingly could be separated into four meaningful parts [un, feel, ing, 1y]. a.k.a., morphemes + By using a limited number of morphemes, one could possibly exponentially extend a given core vocabulary by composing morphemes as new words according to the rules of word formation in Table The adoption of morphemes largely reduces the memory burden and therefore facilitates the learning of words for humans. We hypothesize that morphology also helps for word representation in neural networks, especially in resource-limited scenarios like compression.
4.2 Definition of MorphTE
Considering the above analysis, we propose to inject morphological knowledge to achieve high- quality and space-efficient word embeddings. Suppose word is segmented as / morphemes [mi, .., .., in the natural order. For example, a four-morpheme word unfeelingly is
"This also holds for logogram, written symbols of which represent words instead of sounds. For example. Chinese language has character components, a.k.a, radicals.

---

Aind
BISS
Un
M
kindness
unkindl
Morpheme Segmentation
Kind
Indexing
|
IVI
f
M
unleelingly
Teel Ingly
Kind
iness
Ingly
9"
uB
M
Feel
Vocabulary
Morpheme Index Matrix
Morpheme
Word
Embedding Matrices
Embedding matrix
Figure 3: The workflow of MorphTE. a the order (number of morphemes for word). q is the size of morpheme vectors. INI and IMI denote the size of word vocabulary and morpheme vocabulary.
segmented as [un, feel, ing, ly]. We refer fi()), f.().....fff N as different repre- sentation functions of morphemes selected from parametric family F {f - R*). The Morphologically-enhaare -nhannedd Tensorized Embedding (MorphTE in short) of word is defined as sum of cumulative tensor products morpheme vectors from different representation functions, namely,
N Rd
fi(m))
1=1 =1
Each representation function of morphemes fi could sometimes be considered as a subspace of morphemes. The number of subspaces (i.e., is called the rank, and the number of morphemes (i.e., n) of word called the order, similar to Word2ket. The sum of the outputs from different subspaces in MorphTE is similar to the multi-head mechanism in Transformer (39
Reasons use tensor product. Morph utilizes tensor products to aggregate several small vectors Interestingly, we find that there are some commonalities between tensor product and morpheme composition. (1) Both tensor product and morpheme composition are non-commutative (c.g., c e of @ and likewise "houseboat" *é "boathouse"), (2) Small vectors are the smallest units in Word2ket and morphemes are also the smallest meaning meaning-bearing units of words. Naturally, these small vectors in Word2ket can be assigned morpheme identities and shared among different words.
4.3 Workflow MorphTE
We describe the workflow of MorphTE in Figure
kind
ness
Morpheme segmentation Suppose there [V] individual words. We first segment each word as sequence of morphemes (a 1-morpheme word could be segmented facilitate processing in neural networks, we truncate morpheme sequences in fixed length n (e.g., 3 or 4). (1) For those words that have less than n morphemes, we pad them with some extra tokens; for example, word with n padding morphemes. (2) For those words that have more than morphemes, we concatenate the rest of
kind
ly
sequence as [m]...., mx) using polyglot [ To
BB
feel
Ingly
we pad
Figure 4: Segmentation
For example, such a function could be morpheme embeddings. each vector of which is a a q-sized vector. which are similar word embeddings.
'polyglot could be found in https : :tee:://iitttn githbb...t.....g com/abbSSamoor/polyylla Based on morfessor
polyglot provides trained morpheme segmentation models for variety of languages. Numerous studies 3 31 [2] also utilize morfessor for morpheme segmentation.

---

morphemes as the last one. The processing could be explained as follows:
[m1, 122. mu, pad pad, 1 <<
(6)
M2, concat M............................... mi]], [ n
where concat(-) is the operation to concatenate multiple morphemes into one morpheme. For example, a four- morphemep word unfeelingly, segmented as [un, feel, ing, ly), could be then truncated into (un, feel, ingly) when 3.
could get n-length morpheme sequence for each word; this results in matrix I E RIVIx" each row of which (denoted as IR" the morpheme sequence for a word (i.e., wj), see in Figure
After morpheme segmentation,
Rank-one MorphTE Assume that there are MM individual morphemes. We first define sized trainable morpheme embeddings f E TRMM|xy For a word wj with morpheme sequence [T 122.. ... j. its rank-one MorphTE embedding is tensor product between these q-sized embeddings of these morphemes, resulting in q" q"-sized vector. Namely, the formula for rank-one version of Morph as follows:
Embed(w,) f(4,)) @ f(42) e.... f(1,m))
If q" is greater than the given dimensionality of word embeddings (i.e., d), the excess part of the generated word embedding will be discarded.
General MorphTE Here, we propose a general version of MorphT';; the rank-one MorphTE is 1. We define copies of morpheme emheddings, namely,
special case of MorphTE when its rank f E [Rll[i with Xavier [13. The general MorphTE could be considered as sum of rank-one MorphTE embeddings, r is called a 'rank' of MorphTE. Technically, it is formalized as:
(my)
E((aa) 8... fol1sm))
(8)
4.4 Difference between MorphTE and Word2ket
[Viqur
[V|d=|V 1V
lgr
[M]
unddnely
U83
unkindhess
Idnd
I/1
Idnz
[M]
d
ly
9
WordZat
(F) MorphTT
Figure 5: The graphical representation and parameters Word2ket (A), and MorphTE (B). We show the procedure of how Word2ket and MorphTE generate word embeddings. IVI and IMI denote the size of word vocabulary and morpheme vocabulary. indicates the word embedding size and q indicates the size of the small vector in Word2ket and MorphTE. r is the rank and n is the order.
Although Eq. of MorphTE and Eq. of Word2ket have similar form, these two methods are fundamentally different. Compared with Word2ket, the innovations of MorphTE are as follows:
MorphTE is seapphooggiclll--nhaneeee MorphTE uses morpheme vectors with meaning gram- matical functions to generate word embeddings. Compared with vectors without specific meanings in Word2ket, this way introduces prior morphological knowledge and has better interpretability.
MorphTE captures commonness among words. MorphTE can explicitly add connections of morphologically similar words by sharing morpheme vectors. As shown in Figure 5 in MorphTE,

---

"unkindly" and "unkindness" share the vectors of "un" and "kind", and they are both semantically related to "unkind". However, Word2ket considers these two words to be completely unrelated.
Mo is more parameter-efficinnt Suppose MorphTE and Word2ket have the same rank (r 2 1), order (n 2 2), and dimensionality (g) of the subspace. (1) As shown in Figure[5] the number of trainable parameters required by MorphTE and Word2ket are |M [gr and |V |gner, respectively. (2) The number of parameters of Word2ket divided by that of Morph TE equals n n/Vi/|M) (3) As smaller units, the number of morphemes (lMM) is usually smaller than the number of words ((VD. Referring to Table [V/|M| 2.5. (4) Hence, MorphTE can save more than 5 times the parameters compared to Word2ket (n V M >> 2.5n 2 5).
5 Experiments
5.1 Experimental Setup
Baselines. (1) For word embedding compression, chose Matrix Factor. (low-rank matrix factorization [18), Tensor Train (15). Word2ket, and Word2ketXs embeddings (29] for comparison. (2) For morphological word embeddings, we reproduced the method (called as MorphSum) of representing a word embedding as the sum of its morpheme and surface form vectors [5] [1 2] In
addition, reproduced the method (called MorphLSTMM) of using the output of the word's last morpheme in the LSTM network [14] as the embedding of the word, referring to Morphological RNNs [5). Our MorphTE utilizes the same morpheme segmentation MorphSum and MorphLSTM. Besides, we call the original word embeddings as Original. Except for the embedding layer, all models have the same structure.
Tasks, Datasets, and Metrics. We conducted experiments on machine translation, question an- swering, and natural language inference (NLI) tasks. (1) For machine translation tasks, we chose IWSLT" German-to-Engee Engglissh (De-En) dataset [7]. English-to-ltalian (En-It), English-to-Spanign (En-Es), and English-to- Russsia (En-Ru) datasets of OPUS-100
45 The De-En dataset contains 160K sentence pairs and is processed by the BPE 33 of 10K tokens. The En-It, En-Es, and En-Ru datasets contain 1M sentence pairs and use joint source-target vocabulary processed by the BPE of 40K tokens. The performance is measured by case-sensitive tokenized BLEU (30) for all translation tasks. (2) For question answering tasks, we chose WikiQA 43 a retrieval- -basd question answering dataset. It contains 20.4K training pairs, and mean average precision (MAP), mean reciprocal rank (MRR) are used for evaluation. (3) For NLI tasks, we chose SNL/ (6) which consists of 570k annotated sentence pairs from an image captioning corpus. Accuracy is used as the evaluation metric.
Implementations. (1) For machine translation tasks, we chose Transformer [39] with the imple- mentation of Fairseq (28 For De- En dataset, the Transformer consists of 6-layer encoder and 6-layer decoder with 512 2 embedding size, 1024 feed-forward network (FFN) size. It is trained with a batch size of 4096 tokens
NVIDIA Tesla V100 GPU. For En-It, En-Es, and En-Ru tasks, the FFN size is increased to 2048. They are trained with a batch size of 32768 tokens on 2 NVIDIA Tesla V100 GPUs. (2) For question answering and NII tasks, we followed the implementation and setup of RE2 42). The word embedding size is set to 512, and we trained them for 30 epochs with the early stopping. (3) Notices: Unless otherwise specified, the hyperparameter order of MorphTE 3 in our experiments. For MorphTE and word embedding compression baselines, they are compared under a roughly equal number of parameters (compression ratio) by adjusting their hyperparameters of the rank. For more details on hyperparameters and training settings, refer to Appendix D
5.2 Main Results
MorphTE outperforms compression baselines. As shown in Table 2, we compared MorphTE and compression baselines at about 20x and 40x compression ratios on four translation datasets. (1) At the ratio of 20x, MorphTE can maintain the performance of the original embeddings on all datasets. Specifically, MorphTE achieves 0.4 and 0.3 BLEU score improvements on the De-En and En-Ru datasets, respectively. For other datasets, achieves the same BLEU scores. However, none of the other compression methods can maintain the performance of the original model on these datasets. (2) At the ratio of 40x, although almost all compression methods cannot maintain the original performance, MorphTE still achieves the best results compared to other compression basclines.
7

---

Table 2: Experimental results of different embedding compression methods on translation tasks.
BLEU scores and compression ratios of the embedding layers are reported, with the form of B(Cx). indicates that the method can not achieve approximately the same compression ratio as other methods. Refer to Appendix C for the analysis of the parameters of the methods.
Method
De-En
En-It
En-Es
En-Ru
Original
(1.0x
32.9 (1.0x)
39.1 (1.0x)
31.6
Matrix Factor, 32.7 (19x)
22.8 (40x)
31.0 (20x)
23.2 (42x)
38.0 (20x)
29.7 (42x)
28.9 (20x)
17.9 (40x)
Tensor Train
34,3 (20x)
33.4 (43x)
32.4 (21x)
32.1 (43x)
38.7 (21x)
38.5 (43x)
31.2 (21xx
30.9 (41xx)
Word2ketXs
34.3 (21x)
33.7 (42x)
32.6
31.5 (43x)
38.4 (21x)
38.0 (43x)
31.5 (21x)
(41x
Word2ket
34.2
32.3 (21x)
39.1 (21x)
31.3 (21x)
MorphTE
34,9 (21x) 34.1 (43x)
32,9 (21x)
32.1 (45x)
39.1 (21x)
38.7 (43x)
31.9 (21x) 31.5
Notably, the performance of the Matrix Factor. degrades significantly when a higher compression is conducted, and Word2ket cannot achieve more than 40x compression on these datasets.
MorphTE can handle different languages and tasks well. (1) We validated Mo on transla- tion datasets of De-En, En-lt, En-Es, and En-Ru. MorphTE has been shown to handle different lan- guages effectively. (2) For translation tasks, the De-En dataset uses separate source and target dic- tionaries, while other datasets use a shared source- target dictionary. MorphTE is shown to work in both ways. (3) Besides translation tasks, we also validated MorphTE on question answering (Wik- iQA) and natural language inference (SNL.)) tasks. The experimental results in Table 3 show that Mor- phTE still maintains the performance of the original model and outperforms other compression base- lines on these different tasks.
Table 3: Experimental results on WikiQA of question answering tasks and SNLI of natural language inference tasks. The ratio means the compression ratio of embedding layers.
WikiQA
SNLI
Method
ratio
MAP
MRR
ratio
Accuracy
Original
1x
0.6798
0.6970
1x
0.8492
Matrix Factor.
82x
0.5957
0.6121
38x
0.4166
show
Tensor Train
80x
0.6251
0.6440
37x
0.8473
Word2ketXs
80x
0.6686
0.6871
38x
0.8450
Word2ket
21x
0.6842
0.7025
21x
0.8487
MorphTF.
81x
0.6834 0.7051
38x
0.8497
Table 4: Experimental results for analyzing the effect of morphology. BLEU scores and compression
ratios of the embedding layers reported, with the form of B(Cx). Morph. indicates whether the method introduces morphological knowledge. Compr. indicates whether the method supports high compression. Word2ket+Rshnn Rshaar means the method of random sharing of small vectors in Word2ket.
Method
Morph. Compr.
De-Fn
En-It
En-Es
En-Ru
Original
x
34.5 (1.0x)
32.9 (1.0x)
39.1 (1.0x)
31.6(1.0))
MorphSum
34.9 (0.7x)
39.1(0.8x) (0.x))
31.7 (0.8%)
MorphLSTM
x
(1.60)
32.9 (2.8x)
39.7 (2.7x)
32.4
Word2ket
34.2
(21x)
39.1 (21x)
31.3
Word2ket<Rshare
<
(210)
(210)
38.3 (21x)
30.6 (21x)
MorphTE
34.9 (21x)
32,9 (21x)
39.1 (21x)
(219)
Morpheme-based morphology can enhance word embeddings. To study the impact of morphemes on MorphTE, we conducted following explorations. (1) As introduced in Section 5.1 MorphSum and MorphI.STM both introduce morphological knowledge based on morphemes. As shown in Table 4, although they cannot achieve high compression on word embeddings. they (especially MorphL achieve significant improvements compared with original embeddings. This shows that morphological knowledge based on morphemes is beneficial for word embeddings. (2) MorphTE assigns the identities of morphemes to the small vectors of Word2ket, and shares these small vectors based on morphemes. We consider the method (called Word2ket+Rshare) of random sharing of small vectors in Word2ket rather than morpheme-base based sharing. This method can be implemented in such a way that for each word, randomly assign several row vectors of the trainable matrix of the same shape as the morpheme embedding matrix. As shown in Table Word2ket+Rshann has lower BLEU scores than Word2ket on different translation datasets, and is significantly inferior to MorphTE. This also verifies that it makes sense to introduce morphemes in MorphTE.
8

---

Table 5: Statistics of translation tasks. |V| and |M are the size of the word vocabulary and morpheme vocabulary, respectively. #Struc and #Emb are the number of parameters of the model structure and that of the embedding layer, respectively. ratio is the compression ratio of the word embedding layer. is the proportion the embedding layer to the total parameters.
Original
MorphTE
Dataset
[V]
[M]
#Stuc
#Fmb
P
ratio
#Emb
BLEU
ratio
#Emh
A BLEU
De-En
15480
5757
31.54M
7.93M
20.1%
21x
0.37M
1.2%
+ 0.4
43x 0.18M
0.6%
En-It
41280
10818
44.14M
21.14M
32.4%
21x
0.99M
2.2%
+ +.00
45x 0.45M
1.0%
0.8
En-Es
41336
11377
44.14M
21.16M
32.4%
21x
1.03M
2.3%
+ +0.0
43x
0.49M
1.1%
0.4
En-Ru
42000
12423
44.14M
21.50M
32.8%
21x
1.02M
2.3%
+ 0.3
41x
0.52M
1.2%
0.1
Table 6: Experimental results for the ablation on the order of MorphTE on De-En. d q means the size of the word embedding or morpheme vector. IVI IMI means the size of the word or morpheme vocabulary. #Emb and ratio are the parameter number and com- pression ratio of the embedding layer, respectively.
0.87
1.4
112
0.85
1100
0.84
0.8
0066
Numberr
0.82
0.4
Method order rank d/g IVI IMI #Emb ratio BLEU
0.81
Accurccy Emb dParams
Orginal
512
15480
7.93M
34.5
10
15
20
25
2
7654
0.38M
34.4
Rank
MorphTE
3
5757
0.37M
Figure 6: Accuracy on SNLI and parameter num- bers of embeddings changes w.r.t. the rank.
4
5257
0.41M
19x
34.4
Marginal utility of higher compression ratios (e-g., > 40). We report the statistics of translation tasks in 3 The experimental results of this table are consistent with those in Table (1) At about 20x compression, MorphTE can maintain the performance of the original word embeddings. The number of embedding parameters is approximately reduced from 20M to 1M, and its proportion of total parameters is reduced from 20.1 13..8%
1.3%%..3%. (2) At about 40x compression, although MorphTEE outperforms other compression methods referring to Table 2 MorphTE's BLEU scores are significantly lower than the original model. Although the compression ratio is increased from 20x to 40x, the parameters of embeddings are only approximately reduced from 1M to 0.5M, and the proportion of embedding parameters to the total parameters is only reduced from 1.2%-2.3% to 0.6%-1.29% Considering the loss of model performance, the slight reduction in the number of parameters resulting from higher compression ratios does not make much sense.
5.3 Ablation Study
Sensitivity the order. (1) In previous experiments, we set the order of MorphTE to 3. In this section, we perform the statistical analysis and ablation experiments on the De-En dataset. Morpheme statisties show that more than 90% of the words have more than three morphemes, referring to
Appendix E (2) We set the order of MorphTE to 2, 3, and 4, that is, limiting the maximum number of morphemes 2, 3, and 4, respectively. The word embedding size set 512. We adjusted the rank of Mo so that the models corresponding to these three cases have a similar number of parameters. From Table G setting the order of MorphTE to 3 achieves the best results when compressing the word embedding parameters by about 20 times.
Sensitivity on the rank. We show the effect of rank on the number of parameters and performance of TE in Figure 6 The rank of MorphTE set from to 32. As can be seen, as the rank increases, the number of embedding parameters increases linearly, and the Accuracy first increases and then gradually stabilizes. This means that with the increase of embedding parameters, the model performance will gradually improve, but after the parameter amount exceeds certain threshold, will not drive the performance improvement. Therefore, the degree of compression and model performance can be balanced by adjusting the rank of MorphTE
Sensitivity on the Embedding Size. To verify that MorphTE can stably achieve compression under different word embedding dimensionalities, We conducted experiments with the dimensionalities

---

Table 7: Experimental results for the ablation on the embedding size IWSLT 14 De-En. d/g means the size the word embedding or morpheme vector. |V]/|M means the size the word or morpheme vocabulary. and are the order and rank of MorphTE. #Emb and #Stuc the parameter numbers the embedding layer and the model structure, respectively. ratio is the compression ratio of word embedding layers.
Method
d/9
[VI/IM]
#Emb
ratio
#Stuc
BLEU
Original
216
15480
3,34M
1x
8.71M
34.1
MorphTE
6
5757
0.32M
10x
8.71M
34,2
Original
512
15480
7.93M
31.54M
34.5
MorphTE
5757
0.41M
19x
31,54M
34,9
Original
1000
15480
15,48M
1x
96.72M
32.8
MorphTE
10
5757
0.51M
30x
96.72M
33.4
of 216, 512, and 1000. As can be seen in Table 7 when order n and the word embedding size / hidden size (d) is set to 216, 512, and 1000, the dimensionality of the morpheme vector (g) is set 6, and 10, respectively. that as the dimensionality of a word embedding increases significantly, the dimensionality of the morpheme vector does not increase significantly. This means that the larger the word embedding size, the more the compression potential of our method can be released. Specifically, when the word embedding dimensionality is 216, our method can compress the word embedding parameters by factor of 10 with comparable model performance. When the word embedding dimensionality is set to 512, our method is able compress the word embedding parameters by a factor of 19, with a 0.4 BLEU score higher than the uncompressed model. When the word embedding dimensionality is set to 1000, our method outperforms the original model by a 0.6 BLEU score when compressing the word embedding parameters by factor of 30. In conclusion, our method has stable compression ability and can maintain or slightly improve the model performance under different word embedding sizes.
addition, the rank is set to 8 for these three cases. It can be seen
Similar to other compression methods, our method introduces some computational overhead when generating word embeddings. However, this limitation is light according to the experimental analysis Appendix(C G
Conclusion
We propose MorphTE which combines the prior knowledge of morphology and the compression ability of tensor products to learn high-quality and space-efficient word embeddings. We validated method tasks such machine translation, text summarization, and retrieval-based question answering. Experimental results show that MorphTTE can handle different languages and tasks well. It outperforms related decomposition- baased word embedding compression methods such as Word2ket (29) and TT embeddings [15]. and can achieve dozens of times compression of word embedding parameters without hurting the performance of the original model.
Acknowledgements
This work is supported in part by Natural Science Foundation of China (grant No. fo.62276189 and No.61876129), the Beijing Academy of Artificial Intelligence(BAAAD) TJU-Wenge joint laboratory funding, and MindSpore
This work is also supported by Chinese Key-Area Research and Development Program of Guangdong Province (2020B0101 013500011) and the Guangdong Provincial Key Laboratory of Big Data Computing, The Chinese University of Hong Kong, Shenzhen.
https://wwx www.mmoow.ddooommmmmmm cn/
10

---

References
[1] Anish Acharya, Rahul Goel, Angeliki Metallinou, and Inderjit S. Dhillon. Online embedding compression for text classification using low rank matrix factorization. In The Thirty-Thiin Third AAAI Conference on Artificial Intelligence, pages 6196-6200 2019.
[2] Oded Avraham and Yoav Goldberg. The interplay of semantics and morphology in word embeddings. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2017, Valencia, Spain, April 3-7, 2017, Volume 2: Short Papers, pages 422 426. Association for Computational Linguistics, 2017.
[3] Tamali Banerjee and Pushpak Bhattacharyya. Meaningless yet meaningful: Morphology grounded subword-level NMT. In Proceedings of the Second Workshop on Subword(Character LEvel Models, pages 55-60, New Orleans, June 2018. Association for Computational Linguis- tics.
[4] Parminder Bhatia, Robert Guthrie, and Jacob Eisenstein. Morphological priors for probabilistic neural word embeddings. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNIP 2016, Austin, Texas, USA, November 1-4, 2016, pages 190-500. The Association for Computational Linguisties, 2016.
[5] Jan A. Botha and Phil Blunsom. Compositional morphology for word representations and language modelling. In Proceedings of the 31th International Conference Machine Learning, ICML 2014, Beijing, China, 21-26 June 2014, volume 32 of JMLR Workshop and Conference Proceedings, pages 1899-1907. JMLR.org, 2014.
[6] Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning, A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, EMNLP 2015, Lisbon, Portugal, September 17-21, 2015, pages 632-642, 2015.
[7] Mauro Cettolo, Jan Niehues, Sebastian Stüker, Luisa Bentivogli, and Marcello Federico. Report on the 11th IWSLT evaluation campaign. In Proceedings of the 11th International Workshop on Spoken Language Translation: Evaluation Campaign, pages 2-17, Lake Tahoe, California, December 4-5 2014.
[8] Ciprian Chelba, Tomás Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson. One billion word benchmark for measuring progress in statistical language modeling. In INTERSPEECH 2014, 15th Annual Conference of the International Speech Communication Association, Singapore, September 14-18, 2014, pages 2635-2639 ISCA, 2014.
[9] Patrick H. Chen, Si Si, Yang Li, Ciprian Chelba, and Cho-Jui Hsieh. Groupreduce: Block- wise low-rank approximation for neural language model shrinking. In Advances in Neural Information Processing Systems 31: Anmual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, pages 11011-11021, 2018.
[10] Ryan Cotterell and Hinrich Schütze. Morphological word embeddings. CoRR, abbs/190.000 907.02223. 2019.
[11] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. MMACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume (Long and Short Papers), pages 4171-4186. 2019.
[12] Nadir Durrani, Fahim Dalvi, Hassan Sajjad, Yonatan Belinkov, and Preslav Nakov. One size does not fit all: Comparing NMT representations of different granularities. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume I (Long and Short Papers), pages 1504-1516. Association for Computational Linguistics, 2019.
11

---

[13] Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, AISTATS 2010, Chia Laguna Resort, Sardinia, Italy, May 13-15, 2010, volume of JMLR Proceedings, pages 249-256. .orgR.rrg 2010.
[14] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Comput.. 98)::1335-1780 nov 1997.
[15] Oleksii Hrinchuk, Valentin Khrulkov, Leyla Mirvakhabova, Elena Orlova, and Ivan Oseledets. Tensorized embedding layers. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4847-4860, November 2020.
[16] Henk AL Kiers. Towards standardized notation and terminology in multiway analysis. Journal of Chemometrics: A Journal of the Chemometrics Society, 14(3):05--22,, 2000.
[17] Tamara G Kolda and Brett W Bader. Tensor decompositions and applications. SIAM review, 5 51(3):45--000 2009.
[18] Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. ALBERT: lite BERT for self-supervised learning of language representations. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020, 2020.
[19] Jong-Ryul Lee, Yong-Ju Lee, and Yong-Hyuk Moon. Block-wise word embedding compression revisited: Better weighting and structuring. In Findings of the Association for Computational Linguistics:
EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 16-20 November, 2021, pages 4379 4388, 2021.
[20] Zhongliang Li, Raymond Kulhanek, Shaojun Wang, Yunxin Zhao, and Shuang Wu. Slim embedding layers for recurrent neural language models. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence pages 5220-5228. AAAI Press, 2018
[21] Vasileios Lioutas, Ahmad Rashid, Krtin Kumar, Md. Akmal Haidar, and Mehdi Rezagholizadch. Improving word embedding factorization for compression using distilled nonlinear neural decomposition. In Findings of the Association for Computational Linguistics: EMNLP 2020, Online Event, 16-20 November 2020, volume EMNLP 2020 of Findings of ACL, pages 2774- 2784, 2020.
[22] Peiyu Liu, Ze-Feng Gao, Wayne Xin Zhao, Zhi- Yuan Xie, Zhong- Yi Lu, and Ji-Rong Wen. Enabling lightweight fine-tuning for pre-trained language model compression based on ma- trix product operators. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNPP 2021, (Volume I: Long Papers), Virtual Event, August 1-6, 2021, pages 5388-5398, 2021.
[23] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized BERT pretraining approach. CoRR, Miss/197.11 1692, 2019.
[24] Thang Luong, Richard Socher, and Christopher D. Manning. Better word representations with recursive neural networks for morphology. Proceedings of the Seventeenth Conference on Computational Natural Language Learning, CoNLL 2013, Sofia, Bulgaria, August 8-9, 2013, pages 104-113. ACL, 2013.
[25] Xindian Ma, Peng Zhang, Shuai Zhang, Nan Duan, Yuexian Hou, Ming Zhou, and Dawei Song. A tensorized transformer for language modeling. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019.
2019, December 8-14, 2019, Vancouver, BC, Canada, pages
[26] William O' Grady, Michael Dobrovolsky, and Francis Katamba. Contemporary linguistics. St. Martin's, 1997.
[27] I. Oseledets. Tensor-train train decomposition. SIAM J. Sci. Comput., 33:2295-2317,, 2011.
12

---

[28] Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACI.HLLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Demonstrations, pages 48-53,220 2019.
[29] Aliakbar Panahi, Seyran Sacedi, and Tomasz Arodz. word2ket: Space-efficient word em- beddings inspired by quantum entanglement. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020, 2020.
[30] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, July 6-12, 2002, Philadelphia, PA, USA, pages 311-318. ACL,
2002.
[31] Peyman Passban, Qun Liu, and Andy Way. Providing morphological information for SMT using neural networks. Prague Bull. Math. Linguistics,
2017.
[32] Jeffrey Pennington, Richard Socher, and Christopher D. Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, October 25-29, 2014, Doha, Qatar, A meeting of SIGDAT, Special Interest Group of the ACL, pages 1532 1543. ACL,, 2014.
[33] Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers, 2016.
[34] Raphael Shu and Hideki Nakayama. Compressing word embeddings via deep compositional code learning. In 6th International Conference on Learning Representations, ICLR 2018, Van- couver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018.
[35] Richard Socher, Cliff Chiung- Yu Lin, Andrew Y. Ng, and Christopher D. Manning. Parsing natural scenes and natural language with recursive neural networks. In Proceedings of the 28th International Conference on Muchine Learning, ICML 2011, Bellevue, Washington, USA, June 28 July 2, 201 1, pages 129-136. Omnipress, 2011.
[36] Marzieh S. Tahaei, Ella Charlaix, Vahid Partovi Nia, Ali Ghodsi, and Mehdi Rezagholizadeh. Kroneckerbert: Learning kronecker decomposition for pre-trained language models via knowl- edge distillation. CoRR, (ts///199.02433. 2021.
[37] Urmish Thakker, Jesse G. Beu, Dibakar Gope, Ganesh Dasika, and Matthew Mattina. Rank and run-time aware compression of NLP applications. CoRR, abs/2010.03193, 2020.
[38] Julien Tissier, Christophe Gravier, and Amaury Habrard. Near-lossless binarization of word embeddings. In The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27- February 1, 2019, pages 7104-7111. AAAI Press, 2019.
[39] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages 5998 6008, 2017.
[40] Sami Virpioja, Peter Smit, Stig-Ame Ame Grönroos, and Mikko Kurimo. Morfessor 2.0: Python implementation and extensions for morfessor bascline. In Aalto University publication series. Department of Signal Processing and Acoustics, Aalto University, 2013.
[41] Benyou Wang, Yuxin Ren, Lifeng Shang, Xin Jiang, and Qun Liu, Exploring extreme parameter compression for pre-trained language models. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022, 2022.
13

---

[42] Runqi Yang, Jianhai Zhang, Xing Gao, Feng Ji, and Haiqing Chen. Simple and effective text matching with richer alignment features. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACI. 2019, Florence, Ital,, July 28- August 2, 2019, Volume 1: Long Papers, pages 4699 4709, 2019.
[43] Yi Yang, Wen-tau Yih, and Christopher Meek. Wikiqa: challenge dataset for open- domain question answering. In Proceedings of the 2015 conference empirical methods in natural language processing, pages 2013-2018, 2015.
[44] Chunxing Yin, Bilge Acun, Xing Liu, and Carole-Jean Wu. Ti-rec: Tensor train compression for deep learning recommendation models. ArXiv, abs/2101.11714, 2021.
[45] Biao Zhang, Philip Williams, Ivan Titov, and Rico Sennrich. Improving massively multilingual neural machine translation and zero-shot translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages (628-1639) Association for Computational Linguistics, 2020.
14

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [No]
(c) Did you discuss any potential negative societal impacts of your work? [No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] See supplemen- tal material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section 5.1 and supplemental material.
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [Yes] See Section 5.1
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] See Section 5.1
(b) Did you mention the license of the assets? [No]
(c) Did you include any assets either in the supplemental material or URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're rre
using/curating? [No]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[No]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
15


Reviews:
[{"review_id": "zg26Ww3JSCZ", "reviewer": "Reviewer_mf8C", "summary": "This paper introduces a new word embedding compression method with morphological augmentation, which is called \"morphologically-enhanced tensorized embeddings (MorphTE)\". The general idea is that, similar to how each word's meaning is a combination of its morphemes, each word embedding is a the product of the word's individual morpheme embeddings. \nThe authors conduct experiments on machine translation, question answering, and natural language inference tasks and show that MorphTE can compress word embedding parameters by a lot without performance loss, significantly outperforming related embedding compression methods.", "questions": "- What are the limitations of your work?\n- How does the performance of your proposed approach depend on the performance of the morphological segmenter?", "limitations": "I don't see any direct negative societal impact of this work. The authors don't address limitations explicitly.", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The method is well motivated *and* performs well.\n- The paper is very clearly written.\n- The experimental setup is well done and the proposed approach is compared to relevant baselines.\n\nWeaknesses:\n- No big ones; one could consider using newer NLI datasets like MNLI, but this isn't really a weakness.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Ph5bm9-b3dR", "reviewer": "Reviewer_YXdx", "summary": "This paper proposes to incorporate morphemes while compressing word embeddings and achieve lossless compression while decreasing the number of parameters. The authors compare their method to several compression techniques and show that even with a 40x compression ratio, the proposed embeddings are on par with the baseline, and on 20x they can outperform previous baselines.", "questions": "- I did not quite understand how do you train morpheme embeddings? Is it a linear layer in the transformer? And on which step do you divide a word into morphemes?\n- Do you use words as inputs and split them into morphemes on the fly, or dataset is preprocessed with morphemes? Do you have any computational overhead, especially since your sequences are longer with morphemes representations?\n- line 233: <...>**bpe**<...> - when and for what do you use bpe? Is this only for **Original**? \n- do you still use positional embeddings? Does it somehow change compared to the original? \n", "limitations": "Limitations are discussed in analysis", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Well-written paper\n- Provide results on several tasks and multiple dataset for the task\n- Embeddings can be use on resource-limited devices while still keep the performance\n\nWeaknesses:\n- Usage on resource-limited devices is used as motivation, however there is no further discussion and speed/memory comparison\n- Ambiguity in terms which sometimes affects clarity: word/morpheme/bpe embeddings\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "7t-yDK7KSw", "reviewer": "Reviewer_uxZV", "summary": "This paper presents a model to determine word embeddings by composing morpheme embeddings. This is important research that mixes pre-existing knowledge and corpus-derived knowledge. \n", "questions": "How does the method deal with tensors of increasing dimensions?\n", "limitations": "\nThe paper fails to be clear with respect to the use of the tensor product. Indeed, the tensor product increases the dimensions of the tensors at each step. The dimension of the product of two tensors is the sum of dimensions of the two tensors, that is, if two matrices are the arguments of the tensor product, the results is a four-dimensional tensor. Hence, it is unclear how this model generalizes.\n\nThe authors should reduce the use of the bold and the italics.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pro:\n\n- the idea is sound\n- results are convincing\n\nCons:\n\n- it is unclear the use of the tensor product", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}]

Author rebuttal:

Decision: Accept