DeFINE: Deep Factorized Input Token Embeddings for Neural Sequence Modeling
Authors: Sachin Mehta
Rik Koncel-Kedziorski
Mohammad Rastegari
Hannaneh Hajishirzi
Venue:  (2020)
Keywords: sequence modeling
input representations
language modeling
word embedding

Abstract: For sequence models with large vocabularies, a majority of network parameters lie in the input and output layers. In this work, we describe a new method, DeFINE, for learning deep token representations efficiently. Our architecture uses a hierarchical structure with novel skip-connections which allows for the use of low dimensional input and output layers, reducing total parameters and training time while delivering similar or better performance versus existing methods. DeFINE can be incorporated easily in new or existing sequence models. Compared to state-of-the-art methods including adaptive input representations, this technique results in a 6% to 20% drop in perplexity. On WikiText-103, DeFINE reduces the total parameters of Transformer-XL by half with minimal impact on performance. On the Penn Treebank, DeFINE improves AWD-LSTM by 4 points with a 17% reduction in parameters, achieving comparable performance to state-of-the-art methods with fewer parameters. For machine translation, DeFINE improves the efficiency of the Transformer model by about 1.4 times while delivering similar performance.

Published as a conference paper at ICLR 2020
DEF INE: DEEP FACTORIZED INPUT TOKEN EMBEDDINGS FOR NEURAL SEQUENCE MODELING
Sachin Mchta', Rik
Mohammad Rastegari', and Hannanch Hajishirzi!
University of Washington
2 Allen Institute for AI
ABSTRACT
For sequence models with large vocabularies, majority of network parameters lie in the input and output layers. In this work, we describe a new method, DeFINE, for learning deep token representations efficiently. Our architecture uses a hierarchical structure with novel skip-connections which allows for the use of low dimensional imput and output layers. reducing total parameters and training time while delivering similar or better performance versus existing meth- ods. DeFINE can incorporated easily new or existing sequence models. Compared to state-of-the-ar methods including adaptive input representations, this technique results in 6% to 20% drop in perplexity. On WikiText- 103, DeFINE reduces the total parameters of Transformer-XL. by half with minimal impact on performance. On the Penn Treebank, DeF INE improves AWD-LSTM by 4 points with 17% reduction in parameters, achieving comparable perfor- mance to state-of-the-ar methods with fewer parameters. For machine transla- tion, DeFINE improves the efficiency of the Transformer model by about 1.4 times while delivering similar performance.
INTRODUCTION
Neural models for NLP tasks, such as language modeling and machine translation, require large vocabularies for generality (Chelba al., 2013] Bahdanau al., 2015 Luong al. [0155] Merity iet al.) (017)). These models often employ similar architecture: tokens (c.g., words, sub-words, or characters), represented as one-hot vectors, are mapped to a dense continuous space; they are then processed by context model; finally, the contextualized representations are mapped back to a vocabulary-sized vector for computing next-token probabilities. shown in Figure Ta The mapping in the first and last steps often uses shared learned look- up table, referred as an embedding layer, which takes every token the vocabulary to a fixed m-dimensional vector. One drawback of this approach is that the number of parameters in the em- bedding layer increases as the vocabulary size grows, limiting us to small values of m over large vocabularies. Researchers have sought improve the efficiency of the embedding layer by assign- ing lower frequency tokens smaller dimensional vectors, however, significant parameter reductions come at the cost of performance (Morin & Bengio 20055; Grave et al. 2017a Baevski Auli, 2019). In all these approaches, token embedding is approximated with a linear function from tokens vectors.
language modeling example
In this work, we introduce Deep Factorized INput token Embeddings (DeFINE) for neural se- quence modeling. DeFINE approximates the complicated token embedding function with far fewer parameters compared to standard methods. Der INE allows for lower- diver-dimensionall input and output mappings in sequence models, reducing their computational burden without reducing performance. The representations produced by DeF are more powerful than those of other factorization tech- niques and even standard embedding layers. To accomplish this, DeFINE leverages a hierarchical group transformation (HGT) that learns deep representations efficiently and effectively. HGT con- nects different subsets of the input using sparse and dense connections. To improve the flow of information, DeFINE introduces layer at every level of its hierarchy, allowing gradients to flow back directly to the input via multiple paths. DEFINE replaces standard embedding layers, leaving the rest of the model untouched, and
siip-connection that establishes direct link with the input

---

Published as a conference paper at ICLR 2020
W++1
Wc+1
Output
G1aes: Classiiicction flcstion Layer
aiasoiriation
R
Model
Context
Pransfozner-Xx
Transfomee--.
120
Trane Pransforne--h
DarAs
ginz
Issana Eorree DePTNR
Del INE
m >>
Input
Erboseeing Layer
frebweddiing Lagoz
Isput Oukkpt
We
Wr
(a) Transformer-X0 XL without and with DeF INE
Parameter distribution WikiText-103
Figure I: With DeFINE, Transformer-XL. learns input (embedding) and output (classification) rep- resentations in low n-dimensional space rather than high m-dimensionaa space, thus reducing pa- rameters significantly while having minimal impact on the performance.
so it can be used with a wide variety of sequence modeling architectures and token-types, includ- ing words and sub-words. Figure shows how incorporate DeFINE with Transformer-XL (Dai et al.) (019), a state-of-the Transformer-basee language model, and the resulting reduction in total parameters.
Our experiments show that both LSTM- and Transformer-bused sequence models benefit from the use of DeFINE. Furthermore, our experiments with word-level language modeling and sub-word level machine translation tasks show that DeF INE can be used with different token types. On the Wikitext-103 dataset, an LSTM-based language model with DeFINE provides point improve- ment over full capacity model while using half as many parameters. When combined with adaptive input (Baevski & Auli 2019) and output (Graee et al. 201 7a) representations, DeF INE improves the performance by about points across LSTM-
ssee Table Lap and Transformer-XL. -L--aae--bb (see Table language models with a minimal increase in training parameters. Computation time at infer- is unaffected [
Incorporating DeFINE into the popular AWD-LSTM language model (Merity) et al. 2018b) without finetuning results test perplexity of 54.2 on the Penn Treehank dataset, outperforming both the original and fine-tuned AWD-LSTM models as well Transformer-XX and MoS (Yang et al., 2018). For machine translation, DeF INF improves the efficiency of a Transformer model (Vaswani al.) (017) by 26% while maintaining translation quality, We provide substantive experiments which detail the impact of our architecture decisions and demonstrate the effectiveness of DeF INE across models of varying capacities.
RELATED WORK
Many sequence modeling tasks, including language modeling and machine translation, have large vocabulary. As consequence, the majority of model's parameters are located in the input (or embedding) and the output (or classification) layers. To reduce the computational load presented by these layers, Press & Wolf (001)) and Inan al. (2017) introduce an effective mechanism called weight-tying that enables learning input and output representations jointly while significantly reduc- ing the number of network parameters. To further reduce the computational load from these layers, factorization based methods, such as projective embeddings (Dai et al. 2019 grouped embeddings (Chen al.) 2018 Grave al.] 2017a| Goodman [000] Mnih Hinton 2009 Morin & Bengio)
2005), and slim embeddings (Li et al. 201 8), have been proposed. Projective embeddings approxi- mate large embedding matrix with two smaller matrices while grouped embeddings cluster input tokens by frequency and assign different capacities to different clusters using projective embedding methods. We note that projective embeddings is a special case of grouped embeddings when the
'Embeddings learned using DeFINE can be cached, so DeFINE does not increase the computational cost inference.
2

---

Published as a conference paper at ICLR 2020
number of clusters is one. The adaptive input method of Baevski & Auli (2019) generalizes pro- jective and grouped embedding methods and proposes a factorization method that allows for faster, memory-efficient end-to-end training while providing similar better benefits compared to exist- ing post-training methods which require a pretrained embedding matrix (Chen et al. 2018). Unlike projective and grouped embeddings, Li et al.] (2018) extends group transformation (Kuchaiev &
Ginsburg 2017: Mehta et al. 2018) with the shufffing algorithm of Fisherr&& & Yates ( (1933) to factor-
ize these layers. Other techniques such as codebook learning (Shu & Nakayamal 2017 Chen al.,
2016 Acharya et al. 2019) and quantization (Rastegari et al. 2016 et al. 2017) can be used
to further improve efficiency, especially in terms of storage requirements. DeF INE orthogonal
to these methods; our empirical results in Section 4 show improved performance compared to these methods alone.
Recent advances in sequence modeling, such Transformers and multi-layer RNNs, demonstrate the power of deep architectures in NLP (Jozefowicz et al., 20T6; Vaswani et al.) 2017 Merity al.) 201 8ab. But while significant attention has been given to modeling the interactions between tokens with deep architectures (e.g., ELMo (Peters et al., 2018) and BERT (Devlin et al. 2019)) context- free token representations are typically modeled with only corpus statistics (Pennington al. 2014) or a single linear transformation (Mikolov et al. 2013 McCann et al. 2017) Character-level models (Kim et al. 2016) also effect deep representations of words as convolution over characters, however these models often require more capacity to deliver performance comparable to word- or sub-word- level models (Baevski Auli 2019). Still. DeFINE can be used learn deep representations of a variety of token types, including words, characters, or sub-words (byte-pair encodings) (Sennrich al. 2015).
3 DEFINE
Token embedding is often treated as simple function of one-hot vector dense continuous space. The embedding layer can thus be thought of as a wide, shallow network consisting of a single linear transformation. At its heart, the function that this network approximates (call it f) takes a token from its orthographic form to a representation of those of its morphosyntactic and semantic properties which are relevant for modeling an arbitrary number of contexts in which the token can occur. Most NLP research assumes a simple embedding layer can sufficiently approximate the intractable function f. We hypothesize that, due the complexity of f, shallow network would require exceptional capacity to learn a good approximation. Time and data constraints prohibit learning such a high capacity shallow network. We propose, based on recent theoretical results of [iang & Srikant [2117] that a deeper network can approximate I with significantly fewer parameters than a shallow network. The validity of this assumption evidenced by our experimental results in Section 4
In this work, we introduce DEFINE, an effective way of learning deep token representations in high-dimensional space with a minimum of additional parameters. Our method is based on a Map- Expand-Reduse Reduce (MER) principle, described in Section 3.1 that first maps an input token to a low dimensional embedding vector, then transforms it
high-dimensional space using computa- tionally efficient hierarchical group transformation (HGT, Section (322). which is sketched in Figure 2c The resultant vector is then transformed to a low-dimensional space. Over the course of these transformations, we make use of a new connectivity pattern that establishes a direct link between the input and output layers (Figure 3), promoting feature reuse, and improving gradient flow (Section The output layer of DeF INE can then be used in place of traditional embedding as an input to sequence modeling tasks. We detail the various aspects of the architecture below.
3.3).
3.1 THE MAP-EXPAND EXPAND-REDUCE PRINCIPLE (MER)
The first step in MER, Map, is similar to standard sequence models. Every input token in the vocab- ulary V is mapped fixed dimensional vector e; E 1 However, in our case, the value of is small (say 64 or 128, compared to typical dimensions of 400 or more). The next step. Expand,
Liang & Srikant (0177) prove that, for a large class of functions, the number of neurons needed by shallow network to approximate function is exponentially larger than the corresponding number of neurons needed by deep network. We make the assumption that f in this class of functions.
3

---

Published as a conference paper at ICLR 2020
Layer
parameters
LT
GLT
n/ /g
Map
HGT
(a) ITT
(b) GLT
(c) HGT
(d)
Figure 2: Learning token representations using different transformation layers with N = (a) Linear Transform (b) Group linear transforms (GLT) (c) HGT (see text for details). Here, N is the total number of layers, n' and k are the input and output dimensions of 1-th layer, g is the number of groups in 1-th layer, and g is the fixed number of groups in group linear transforms.
takes e; as an input and applies a hierarchical group transformation (HGT) to produce very high- dimensional vector é; E Rx where k >> Unlike a stack of fully connected layers, HGT learns deep representations efficiently from different subsets of the input using sparse and dense connec- tions. The last step, Reduce, projects the vector é; to lower dimensional space to produce the final embedding vector e, R for given input token. The dimensions of e, can be matched to contextual representation models, such as LSTMs Transformers, allowing DeF INE serve input layer for these models.
3.2 HIERARCHICAL GROUP TRANSFORMATION (HGT)
We introduce hierarchical group transformation (HGT), sketched in Figure 2c to learn deep token representations efficiently. HGT comprises of a stack of N layers. At each layer, HGT uses a different number of groups that allows it learn representations from different subsets of input. HGT starts with Imar groups at the first layer and then subsequently decreases the number of groups by factor of 2 at each level. This hierarchical grouping mechanism sparsifies the connections in fully connected (or linear) layers and allows us to learn representations efficiently with fewer parameters. Similar stack of fully connected layers, the N-th layer in HGT has access every input element of the first layer through multiple paths, thereby, allowing it to learn effective representations. Group linear transformations (GLTT), originally introduced to improve the efficiency of the LSTM, also sparsify the connections in fully connected layers and significantly reduce computational costs (Kuchaiev & Ginsburg. 2017 Mchta et al. 2018). However, if we stack multiple GLT layers, the outputs a certain group are only derived from small fraction of the input, thus learning weak representations. The hierarchical grouping mechanism in HGT allows the N-th layer to obtain input data from multi- ple paths, enabling HGT to learn stronger representations. comparison of different transformations is given in Figure 2 We can see that HGT is both efficient and has better access to the input. Note that linear and group linear transforms are special cases of HGT when g' and gy' = g (fixed), respectively.
To transform e; gnxl é; E R HGT first samples the space between and k linearly to construct N intermediate layers of increasing dimensionality. Therefore, the output vector produced by 1+ 1-th layer will have higher dimensionality than the I-th layer. Assume that the linearly spaced vector dimensions are divisible by Imaz, we transform C; to é; as follows:
Fo W é
1=1
Fo (c (...) w')g), 1<lsN
1), 4 are the weights learned at 1-th layer, and Fa is group transfor- mation function defined in Mehta et al. (2018). Group transformation splits the input into g groups, each which is processed independently using a linear transformation. The output of these groups then concatenated to produce final output. See Section A.ction/A.Ife details.
where g' max (L Smpes
3.3
DEFINE UNIT
The DeFINE unit is composed of HGT transformations that are designed using the MER principle. Though HGT layers efficient approximation computationally expensive fully connected
4

---

Published as a conference paper at ICLR 2020
Layer (Q*=2)
Mixer
Split (g)
Layer (g'=4)
We
Figure 3: The DeFINE unit with N 2 that uses HGT learn input token representations efficiently and direct connection with the input to maximize the flow of information.
layers, they might impede training the depth N the DeF INE unit grows. Residual connections (He et al. 2016) have proved to be very effective at mitigating this issue, however, such connections are difficult to implement HGT because the input and output dimensions of each layer are different.
To maximize the flow of information and facilitate training with deeper DeF INE units, we introduce a simple new skip-connection that establishes a direct link between any layer HGT with the input Figure 3 visualizes the DeF INE unit with depth of two (N=2). To enable the sparse connec- tions in HGT to have access to the input and the output of the previous layer (é) ), we chunk the input and the output into g' groups using a split layer. The chunked input and output vectors are then mixed such that the first chunk of the input and the first chunk the 1-th layer's output are put together as the input for the first group transformation in the 1-th layer and so until g inputs have been constructed. The resultant vector is then fed to the /-th layer. This mechanism promotes input feature reuse efficiently. Additionally, it establishes direct link with the input e,, allowing gradients to flow back to the input via multiple paths and resulting in improved performance.
3.4 DEFINE FOR SEQUENCE MODELING
The DeFINE unit can be easily integrated with any new existing sequence models. Sequence models typically consist of a stack of an input layer (embedding or adaptive input layer), a contex- tual model (e.g., LSTM or Transformer). and a classification layer (a fully-connected or adaptive softmax). Since DeFINE learns deep token representations, we can easily stack it immediately after the input. An example is shown in Figure [ where DeF INE integrated with Transformer-Xl.. state-of-the-art language model. DeFINE enables the use of relatively lower dimensions in the input layer, thus reducing network parameters.
The input token representations, e,, é, and e., that neural model learns for each token are inde- pendent of other tokens. This allows us to create another independent look-up table (after training model) that caches the mapping between the input token and the output of the DeF INE unit (e,), resulting mechanism that allows to skip the computations of the DeF INE unit at inference time.
EXPERIMENTAL. RESULTS
We demonstrate the performance of DeFINE on two sequence modeling tasks: language modeling (Section 4.1 and machine translation (Section 4.2). We compare the performance of DeFINE with existing factorization and compression-be baseed methods in Section 4.3 We also provide ablations in Section 4.4 to show the effectiveness of design decisions. Throughout this section, the

---

Published as a conference paper at ICLR 2020
Configuration
Parameter Distribution (in millioons
Training
Perplexity
Row
#
Input--utpat
Layers
Depdh of DEFINE (N)
Dimension of
DeFINE
Context model
Input-Output (lied)
Total
Time (ms//btcc))
Val
Test
Standard
256
43.24
Adaptive
256
0.00
23.36
9:25
32,61
297
43.49
44.87
R3
Adaptive DeFIME
3
256
0.41
23.36
9.25
33.02
298
38.99
41.17
Adaptive DeFINE
7
384
1.83
24.73
13.90
40.46
36.95
38.01
Adaptive DeFINE
11
512
3,89
26.24
18.55
48.69
34.94
35.94
(a) LSTM-based language model (ours) on WT103. For this experiment, we use two GPUs.
Purameters millions)
Perplexity
Model
A Paraaeters
(in millions)
Perplexity (Test)
AWD-LSTM (Merity et 20186
M
61.2
58.8
20126 LSTM
48.7
AWD-LSIM Fineeree
M
58.8
56.5
LSTM
AWT-I-STM-Ma'S Yang et 2018
22, M
58.1
56.0
Minny
QRNN
AWD&LSTM-MM STM-MoS Finetne
22 M
56.5
54.4
1.TM DePINE (Qurs)
48.69 M
35.94
Transformer- XL. (Dii 2019
24M
54.5
AWD-LSIM SNPINE
56.5
54.2
(b) Comparison with existing works on WT-103
(c) Comparison with existing works on the PTB dataset
Table 1: Performance of RNN-bbsed language models on WT-103 and PTB dataset. In (a), standard refers standard (linear) embedding and classification layers while adaptive refers adaptive input and adaptive softmax for the input and the output layers, respectively.
following notation: n, k and m are dimensions of c; é; and respectively, and N represents depth of DeFINE.
4.1 LANGUAGE MODELING
In this section, we study the performance of our models with LSTM- and Transformer-based lan- guage models on two datasets: WikiText-113 (Merity et al. 2017) and the Penn Treebank (Marcus) jt al. 1994). On both datasets, we show that DeFINE is parameter efficient and improves the performance of existing language models.
4.1.1 WIKITEXT-103 (WT-103)
Data and models: The WikiText-103 dataset (Merity al. 2017) consists of 103M/217K/445K
tokens for training, validation, and test sets respectively and has a vocabulary size of about 260K. This dataset composed of Wikipedia articles and retains punctuation, numbers, and case. To evaluate the effectiveness of DeF INE, we study two different kinds of contextual models: LSTM, and Transformer (Transformer-Xi (Dai et al. 2019)). We measure the performance of these models in terms of perplexity, a standard metric for language modeling. Lower values of perplexity indicate better performance. Following recent works, including MealuddiinrMrttyyy et al. [2018s) Baevski & Auli (2019), and Dai et al. (2019), we adaptive inputs mapping function in DeF INE and adaptive softmax for classification with tied weights. See A.3 for more details.
Results of LSTM-based language models: Table T summarizes the results of L.STM-based lan- guage models. Though the adaptive input (Baevski & Auli 2019) and output (Grave al. 2017a) methods are effective and reduce the number of parameters significantly, our method further im- proves performance by about points while learning only 1. 1.25% (or 0.4 million) more parameters. It is important to note that the computational complexity of models in R2 and R3 is the same because our method allows caching outputs of DeFINE for use at inference (see Section 3.4).
When we scale the depth of DeF INE from 3 to 11 layers (Table lb the performance improves by a further points, delivering competitive performance to existing RNN-based methods with fewer parameters (e.g., 1 1/3 as many parameters as Merity et al. [201 8a)). The performance of our model better than existing methods such as Dauphin et al. (2017) and Bai et al. (2018).
Results of Transformer- model: Table 2 compares the performance of Transformer-XX a slate-ofttie the Transformer-bol based model, with and without DeF INE. Table 2a shows method is able to attain similar performance to Dai et al. (2019) while learning 10M fewer parameters.
We scale the input and the output dimensions to uniformly increase the network complexity.

---

Published as a conference paper at ICLR 2020
Input-Output
Dimension
Parameter Distrihution (in millions)
Training Time (mo/hatch)
Perplexity
Model
of
Context model
Input-Ouiput
Text
Layers
(n)
(tied)
Transformer-L
Standard
410
0.00
41.07
15L.1
894
24.03
Transformer XI.
Standard
384
0.00
36.25
103.08
139.13
855
26.10
27.06
Transfomer-XX
141.25
860
23.59
24.17
Transformer-X0 XL,
Projective
256
0.00
36.25
69.20
105.45
714
27.18
28.09
Transfomer-XL
DeFINE
256
1.92
36.25
69.20
107.37
721
24.81
25.72
Transfommer XL.
Projective
128
0.00
36.25
34.73
70.98
600
28.05
29.16
Iransforer--X XL
DeFINE
128
1.92
36.25
34.73
72.90
606
25.43
26.33
Transformer- XI.
Projective
64
0.00
36.25
17.50
53.75
550
32,94
33.74
Transforme--LL.
DeFINE
64
1.92
36.25
17,50
55.67
553
28.03
29.
(a) Comparison grouped by mapping layer e;-
Model
Parameters (in million)
Perplexity
Transformet-U (Standard)
139.33
Transformer- XI. (DeFINE)
72.90 M
26.33
Transformer- XL (Projective)
70.98 M
29.16
Transformer-X XI. (DaFINE)
55.67
29.10
(b) Comparison grouped by similar perplexity.
Table 2: Transformer-Xi performance on Wikitext- 103 dataset. We use DEFINE with N 3. k = 4096, and m = 384. For models without DeFINE, we use projective embeddings (Dai et al. 2019) that linearly projects the vector e; to a dimension of m 384. Except the row marked with that uses inner model dimension of 2100, all other rows uses an inner model dimension of 1920. Best number in each group Table 2a is highlighted in red while overall best numbers are marked in bold. Table 2a shows that adding DeF INE significantly improves results with low overhead; Table 2b shows the parameter reduction using DeF INE for similar performance.
interesting note that DeFINE enables us to reduce the computational burden from the input and output layers by a large amount with minimal impact on performance. With DeF INE, the performance of Transformer-XL drops only by about points while the number of parameters are reduced by 50%. For similar reduction in the number of parameters, the performance of original Transformer-XL XI. drops by points, suggesting the proposed method for learning word-level rep- resentations is effective. Table 2b highlights the fact that Transformer-XL with DeFINE is able to achieve comparable perplexity to a standard Transformer-Xx with projective embeddings while using significantly fewer parameters.
4.1.2 PENN TREEBANK (PTB)
Data and models:
The Penn Treebank dataset (Marcus et al. 1994) contains about 929K/74K/82K
tokens in its train, validation, and test sets respectively. It has a vocabulary size of about 10K. Following recent works, we the processed version provided by Mikolov et al. (2010). To evaluate the effectiveness of our model, we compare to AWD-LSTM (Merity et al. 20185). Our model replaces the embedding layer in AWD-LSTM with DeFINE unit with the following settings: 128, & = 1024. N = 7, and m = 400. We use the same hyper-parameters and PyTorch version as the original AWD-LSTM.
Results summarized in Table Ic The proposed method improves the performance of AWD-LSTM by 4 points while simultaneously reducing the number of parameters by million. Without any finetuning, AWD-LSTM DeFINE achieves comparable performance state-of-the- art methods, including Transformer-Xi with fewer parameters.
Results:
4.2 MACHINE TRA TRANSLATIOON
Data and models: We use the WMT 2014 English- Gerimnn (EN-DE) dataset (Luong et al. 2015)
for training. Following Vaswani et al. (017), encode sentences using byte-pair encoding (Britz jet al. 2017) and use newstest2014 and newstest2017 as validation and test sets, respectively. We integrate DeFINE with the state-of-the-ard Transformer model (Vaswani et al. (017)) with following
parameters: = 128, k = 1024, m = 512, and N = 3. We use the implementation in OpenNMT- py (Klein al. 2017 for training and evaluation with the recommended seseer-aaramtter,
7

---

Published as a conference paper at ICLR 2020
Checkpoint
Parameters (in millions)
BLEU (EN-DE)
Model
Averaging?
newstest2014
newstest2017
Transformer /Vaswani al. 2017
27.30
Transformer SRU Lei 2018
/
90 M
27.1
28.30
Transformer (OpenNMI impl) Klein et al. 2017
92 M
26.89
28.09
Transformer
X 92 M
25. 01.01
25.81
Transformer + DeFINE
X 68 M
27.01
28.25
Table 3: Results of Transformer- model (with and without DeF INE) on the task of neural machine translation. DeF DNFINE attains similar performance to checkpoint averaging, but with fewer parameters.
Sequence Model
Input-Output
Parameters (in millions)
Task
Performance
Layers
LSTM
Standard Adaptive DeFINE
92 M
44.12
(Table
a
Language Modeling
33 M
44.87
33 M
41.17
AWD-LSTM
Standard DeFINE
24 M
58.8
I Language Modeling
(Table Ic]
20
54.2
Transformer-X
Standard
M
27.06
(Table
Language Modeling
Projective
71
29.16
DeFINE
73 M
26.33
Transformer
Standard
92 M
25.81
Machine Translation
(Table
DeFINE
68 M
28.25
Table 4: Performance comparison of different sequence models with different factorization methods. Projective and adaptive factorization method refers to methods in Dai et al. (2019) and [aevski & Auli (2019), respectively. For language modeling, performance is measured by perplexity; for machine translation, BLEU used.
Table summarizes the results. DeFINE improves the performance of the Transformer model without checkpoint averaging by 2% while simultaneously reducing the total number of pa- rameters by 26%, suggesting that DeF INE effective.
Results:
4.3 COMPARISON WITH DIFFERENT METHODS
cectorization-basse methods: Table compares the performance of different factorization meth- ods for different sequence models. With DeFINE, the performance and efficiency of sequence models improves across different tasks. This is likely because the output of DeFINE more closely approximates the correlation pattern standard embedding layer compared to other embeddings (see Figure 4 and Appendix [B). Furthermore,
that strong correlations between dimensions in the mapping layer of DeFINE are reduced over the course of the expansion layers (see Figures and 10 in Appendix). Figure 11 in Appendix shows that groups within expansion layer of DeFINE are not correlated, suggesting these matrices are learning different representations of their input.
Impact of compression-based methods: Compression-based methods allow for efficiently dis- cretizing the continuous 32-bit full-precisin embedding vectors, thus reducing the memory foot- print of the input layer. With DeFINE, also learn continuous full precision 32-bit floating-point embedding vector (similar to Baevski & Auli Aulii (2019) and|Da et al. (201)).. Therefore, compression- based methods, such as (Shu Nakayama| 2017), can applied sequence models with DeFINE and other factorization methods. Table 5 shows that DeF INE embeddings can be compressed simi- larly to standard embeddings without loss of performance.
4.4 ABLATION STUDIES ON WIKITEXT- 103 DATASET
In this section, we provide an analysis of our design choices using an LSTMM-baee language model. In our ablations, choose L.TM- over Trasse language models because they less
8

---

Published as a conference paper at ICLR 2020
-00
(a) Standard (no factorization)
(b) Projective Daai ct al. 2019)
(c) DeF INE (Ours)
Figure 4: Correlation map (m x m) of different embedding layers used in Transformer-Xi with n 128 and m = 384 on the VikiText-103, DeFING is able to approximate the standard embedding matrix efficiently. More visualizations are included in Appendix B
Dimension
Input-Output
Compression Used?
Look-up Table Size (in MB)
Perplexity
Inference Time
Layers
(in ms/batch)
384
Standard
None
411
27.06
202
384
Standard
Yes
21
27.36
201
128
Projective
None
29.16
128
Projective
Yes
21
29.82
129
128
DeF INE
None
127
26.33
131
128
DeFINE
Yes
21
26.03
130
Table 5: The performance of Transformer-XI. with different factorization methods, with and without compression method of (Shu & Nakayama (2017). For compression, we used a 32 x 16 coding described in Shu & (2017
sensitive to hhper--prrnnen parameters and can be trained on a single GPU. We use the same hyper-parameters for training as described in Section 4.1.] specifically N 7, n 384, k 1024, and m 384.
results. HGT is effective lin- ear transformation while learning two million fewer parameters. Compared to group linear transform (GLT), HGT improves perplexity by about 5 points while learning similar number of parameters. Furthermore, when we establish a direct connection with the input (see Section 3.2 for details), the performance further improves by 2.9 points with minimal impact on number of parameters, suggesting that DeF INE learns good representations.
Impact of difffrrnt transformations: Table summarizee
Impact of scaling depth (N) and width (k)): Table summarizes the results of our scaling exper- iments. For the same value of the performance of the language model improves with the increase in the depth N. However, when we scale the width & for a fixed value of depth N, the performance does not improve. This is likely because, as we increase the size of more neurons are receiving their input from the same subset of dimensions and thus learning many redundant parameters.
DeF INE with different connections: Table 8a demonstrates the impact of residual connections DeFINE. In order to facilitate residual connections inside INNN,E fix the dimension each layer é in DeF INE to be k instead of linearly spanning from to k. We can clearly see that the proposed skip-connections are more effective.
Impact of reduce operation in MER: In the strategy (Section 3.1 we project the high- dimensional vector to low-dimensional space before feeding it to a contextual model, such as an LSTM. We empirically found that the performance with and without this reduction step is similar, however, model without the reduction step learns more parameters (Tbble(8b).
CONCLUSION
DeFINE deep, hierarchical, sparse network with new skip connections learn better token embeddings efficiently. Sequence models with DeF INE (c.g.. Transformer and LSTM) perform comparably or better with state-of-the-art methods with fewer parameters. Our experiments show

---

Published as a conference paper at ICLR 2020
Parameters
Perplexity
Layer
(in millions)
Val
Test
Layer
Parameters
Perplexity
Linear
(in millions)
Val
Test
GLT
39.69
44.28
45.63
39.79
40.92
GLT Shuffle
39.69
44.08
45.25
mixer)
BGZ
40.73
39.79
40.92
DeFINE
40.89
36.95
38.01
(a) Different transformations (see Figure
(b) HGT vs. DeFINE
Table 6: Comparison between different transformations on the WikiText- 103 dataset.
Depth of
Dimensions
Paraameters
Perplexity
DeFINE (N)
(nt)
(ri)
(k)
(in millions)
Val
Test
1024
33.02
39.99
41.17
256
256
1536
33.15
40.08
41
1024
40.73
36.95
38.01
2048
43.19
36.95
37.84
1024
34.94
35.94
11
512
512
1536
52.02
35.25
35.98
2048
55.02
35.00
35.92
Spocha
10
3.
(a) Depth (N) width (k)
(b) Validation perplexity vs, epochs
Table 7: Impact of scaling depth and width on WT-103.
Parameters (in millions)
Perplexity
Parameters (in millions)
Perplexity
val
Test
Test
DeFINE + residual comn.
41.63
38.96
40.03
MER
40,89
36.95
38.01
DeFING INE
40.89
36.95
38.01
Reduce
43.91
37.19
38.34
(a)
(b)
Table Different settings WT-103: (a) Impact of different skip-connections. See Figure 5b and
Figure 5c| in Section A.2 for block level diagrams. (b) Impact of reduce operation in MER (Sec-
tion
that the proposed architectural decisions each contribute to the effectiveness of the DeFINE unit. We believe neural sequence models with DeFINE can be further improved with extended hyper- parameter search, similar to Melis al. (011)). In future work, will apply DeFINE other sequence modeling tasks. For instance, we believe that pretrained language model architectures such ELMo and BERT can benefit from incorporating DeFINE improve efficiency and performance. Another direction is to use the components of DeFINE specifically MER, HGT, and mixing layers in neural architecture search processes. We have shown the promise of these components here, but a thorough architecture search may discover more optimal configurations in the large search space defined by the depth, grouping, and connectivity parameters.
LEKNOWTED
This research was supported by ONR N00014-18-1-2826 DARPA N66001-19-2-403, NSF (IIS- 1616112, IIS12528351 an Allen Distinguished Investigator Award, Samsung GRO and gifts from Allen Institute for AI, Google, and Amazon. Authors would also like thank members of the UW- NLP and the H2Lab at The University of Washington for their valuable feedback and comments.
10

---

Published as a conference paper at ICLR 2020
REFERENCES
Anish Acharya, Rahul Goel, Angeliki Metallinou, and Inderjit Dhillon. Online embedding com- pression for text classification using low rank matrix factorization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 6196-6203, 2019.
Alexei Baevski and Michael Auli. Adaptive input representations for neural language modeling. In International Conference on Learning Representations, 2019. htttt::///p/erreeee
net/Eorum?2 2tt
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning align and translate. In International Conference Learning Representations, 2015.
Shaojie Bai, J. Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv:1803.0127/ 2018.
Denny Britz, Anna Goldie, Minh-Thang Luong, and Quoc Le. Massive exploration of neural ma- chine translation architectures. arXiv preprint arXiv: 2017.
Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Kochn, and Tony Robinson. One billion word benchmark for measuring progress in statistical language modeling. arXiv preprint arXiv: 3312.3322 2013.
Patrick Chen, Si, Yang Li, Ciprian Chelba, and Cho-Jui Hsieh. Groupreduce: Block-wise low- rank approximation for neural language model shrinking. In Advances in Neural Information Processing Systems, pp. (0988-10995) 2018.
Yunchuan Chen, Lili Mou, Yan Xu, Ge Li, and Zhi Jin. Compressing neural language models by sparse word representations. arXiv preprint arXiv: 1610.03950, 2016.
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov. Transformer-Xe XL: Attentive language models beyond fixed-length context. Proceedings of the 57th Anmual Meeting of the Association for Computational Linguistics, 2019.
Yann N Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 933-941. JMLR. org, 2017.
Jacob Devlin, Ming-We Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), 2019.
Ronald A Fisher and Frank Yates. Statistical tables for biological, agricultural and medical re- search. Oliver and Boyd Ltd, London, 1943.
Goodman. Classes for fast maximum entropy training. In 2001 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings (Cat. No.01CH37221), 2001.
Édouard Grave, Armand Joulin, Moustapha Cissé, David Grangier Facebook AT Research, and Hervé Jégou. Efficient softmax approximation for gpus. In Proceedings of the 34th International Conference on Machine Learning Volume 70, ICML' 17, 2017a.
Edouard Grave, Armand Joulin, and Nicolas Usunier. Improving neural language models with continuous cache. In International Conference on Learning Representations, 2017b.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEFE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El- -anYv, and Yoshua Bengio. Quantized neural networks: Training neural networks with low precision weights and activations. The Jour- nal of Machine Learning Research, 18(1):6869-668 2017.

---

Published as a conference paper at ICLR 2020
Hakan Inan, Khashayar Khosravi, and Richard Socher. Tying word vectors and word classifiers: A loss framework for language modeling. In International Conference on Learning Representations, 2017.
Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. Exploring the limits of language modeling. arXiv preprint arXiv:/602.0244 2016.
Yoon Kim, Yacine Jemiite, David Sontag, and Alexander M Rush. Character aware neural language models. In Thirtieth AAAI Conference on Artificial Intelligence, 2016.
Guillaume Klein, Yoon Kim, Yuntian Deng, Jean Senellart, and Alexander Rush. OpenNMT: Open- source toolkit for neural machine translation. In Proceedings of ACL 2017, System Demonstra- fions, 2017.
Oleksii Kuchaiev and Boris Ginsburg. Factorization tricks for Istm networks. In International Conference on Learning Representations Workshops, 2017.
Tao Lei, Yu Zhang, Sida I. Wang, Hui Dai, and Yoav Artzi Simple recurrent units for highly paral- lelizable recurrence. In Empirical Methods in Natural Language Processing (EMNLP), 2018.
Zhongliang Li, Raymond Kulhanek, Shaojun Wang, Yunxin Zhao, and Shuang Wu. Slim embedding layers for recurrent neural language models. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
Shiyu Liang and Rayadurgam Srikant. Why deep neural networks for function approximation? ICLR, 2017.
Minh-Thaue Thang Luong, Hieu Pham, and Christopher D. Manning. Effective approaches to attention- based neural machine translation. In Empirical Methods Natural Language Processing (EMNLP), September 2015.
Mitchell Mareus, Grace Kim, Mary Ann Marcinkiewicz, Robert MacIntyre, Ann Bies, Mark Fer- guson, Karen Katz, and Britta Schasberger. The penn treebank: Annotating predicate argument structure. In Proceedings of the Workshop on Human Language Technology, HLT '94, 1994.
Bryan McCann, James Bradbury, Caiming Xiong, and Richard Socher. Learned in translation: Contextualized word vectors. In Advances in Neural Information Processing Systems, pp. 6294 6305, 2017.
Sachin Mehta, Rik Connel-Kedziorree Mohammad Rastegari, and Hannanch Hajishirzi. Pyrami- dal recurrent unit for language modeling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2018.
Gbor Melis, Chris Dyer, and Phil Blunsom. On the state of the art of evaluation in neural language models. In International Conference on Learning Representations, 2018.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In International Conference on Learning Representations, 2017.
Stephen Merity, Nitish Shirish Keskar, and Richard Socher. An analysis of neural language modeling at multiple scales. arXiv preprint
2018a
Stephen Merity, Nitish Shirish Keskar, and Richard Socher. Regularizing and optimizing LSTM language models. In International Conference on Learning Representations, 2018b.
Tomáx Mikolov, Martin Karafiát, Lukáš Burget, Jan Černocký, and Sanjeev Khudanpur. Recurrent neural network based language model. In Eleventh annual conference the international speech communication association, 2010.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed represen- tations words and phrases and their compositionality. In Advances neural information pro- cessing systems, pp. 311 111111199, 19, 2013.
12

---

Published as a conference paper at ICLR 2020
Andriy Mnih and Geoffrey E Hinton. A scalable hierarchical distributed language model. In Ad- vances in neural information processing systems, pp. 1081-1088, 2009.
Frederic Morin and Yoshua Bengio. Hierarchical probabilistic neural network language model. In Aistats, volume pp. 246-252. Citeseer, 2005.
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. Glove: Global vectors for word
representation. In Empirical Methods in Natural Language Processing (EMNLP), 1532-1543,
2014. htt.: www.::.wwwwwl..... http////..r.r.aaaoooooo////1-11-
Matthew E. Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In Proc. of NAACL, 2018.
Ofir Press and Lior Wolf. Using the output embedding improve language models. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, 2017.
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. In European Conference on Computer Vision, pp. 525-542. Springer, 2016.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.077 07790999807999,.. 2015.
Raphael Shu and Hideki Nakayama. Compressing word embeddings via deep compositional code learning. arXiv preprint 1rX1iv1711.000 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances Neural Infor- mation Processing Systems 30, pp. 5998-6008. 2017.
Zhilin Yang, Zihang Dai, Ruslan Salakhutdinov, and William W. Cohen. Breaking the softmax bottleneck: A high-rank RNN language model. In International Conference on Learning Repre- sentations, 2018.
13

---

Published as a conference paper at ICLR 2020
A
APPENDIX
A.1 TRANSFORMATION FUNCTION FG
To produce an output y E R=nx from imput x E R and weight matrix W E R R chunks the input into 9 groups and then concatenates the chunked parts produce R 3, then multiplied with weight matrix W to produce ý W R The resultant vector is then flattened to produce When 9 = 1, we obtain the linear transform.
Fe first
A.2 BLOCK LEVEL DIAGRAMS OF DIFFERENT SKIP-CONNECTIONS IN DEFINE
Block level diagrams of different variants of DeF INE are given Figure
Figure 5a stacks trans-
formation layer FG (Eq. 1 and is the same as HGT in Figure 2c Figure adds a residual connection
Figure 5a Figure 5c same as Figure 3 while Figure 5d the same as Figure but without split and mixer functionality.
e Trans formatioo Layer
e
e Fransformation Layer
e
Fransformation Layer
Transformation Layer
Split
Concat
Pransformation Layer
Pransformation Layer
Transformation Layer
Transformation Layer
Split and
Fransformat Layer
Praneformation Layer
Transformation Layer
Transformation Layer
ei
ei
e
e
(a) HGT
(b) HGT+ Residual
(c) DeFINE
(d) DeFINE (w/o mixer)
Figure 5: Different ways of stacking transformation layer FG (Sec.
A.1 for learning deep token
representations.
A.3 HYPER-PARE PARAMETERS FOR TRAINING LANGUAGE MODELS
For training LSTM-based language models, use single NVIDIA GTX 1080 GPU with I1 GB GPU memory while for training Transformer-XL, we used four GeForce RTX 2080 Ti GPUs, each with 11 GB of GPU memory (as recommended by authors). Following recent works, including Merity et al. (2018a) and Baevski & Auli (2019), we use adaptive inputs as mapping function in DeFINE and adaptive softmax for classification for our experiments with RNN-based sequence models. We also tie weights between the adaptive inputs and outputs. For Transformer-XI. Dai et al. (209), we use projective embeddings (as done by authors). We train our models using Py Torch (v1.2), For LSTM-based language models, we similar which are summarized Section g
as Merity et al. (2018) 8a)
A.4 PERFORMANCE OF TRANSFORMER -LL ON WIKITEXT-103
Figure 6 plots the validation perplexity of Transformer-Xi the WikiText-103 as function of training steps. We can see that DeF INE enables Transformer-X XL to deliver similar performance with fewer parameters.
14

---

Published as a conference paper at ICLR 2020
WikiText-103
# of GPUs
Weight decay
Optimizer
SGD
LR
20
BPTT Length
140
Batch size
Epochs
20
LR reduction (factor, steps)
10, [15]
LSTM Hidden Dimension
1024
# of LSTM Layers
Max. dimension of é; (k) Dropout
1024
Same as Merity al. 20188
Table for training word-level LSTM-based language model on WikiText-103. These settings similar to Merity et al. (018a).
105
Params 142M
Params 107M
Perplexity
Par ams 73M
75
60
30
15
4
28
52
76
100
124
148
172
196
Steps
(x1000)
Figure 6: Transformer-X XL performance on Wikitext-103 dataset with DeF INE.
B CORRELATION MAP VISUALIZATION FOR TRANSFORMER XL ON WIKITEXT-103
Computing correlation map: Let us say that we have an arbitrary look-up table E E gVxm that maps every token in vocabulary V to mn-dimensional vector space. We compute the correlation map M as: M = E E E R.m xm If the correlation map is identity, then suggests that the m- dimensions in E are independent. To encode better contextual representations among tokens using context models such as LSTMs and Transformers, embedding dimensions should be independent.
Can DeF INE approximate the standard embedding layer? Figure visualizes the correlation maps of embeddings learned using a standard embedding layer (top row), projective embeddings (Acharya al., 2019 Dai al., 2019) (middle row). and DeFINE embeddings (bottom row) at different values of where
the dimension of mapping layer in DeFINE. Compared to pro- jective embeddings, DeFINE is able to approximate the standard embedding layer efficiently and effectively (see Table 2 for efficiency and performance comparison).
Furthermore, we provide layer-wise comparison for DeFINE at different values of n in Figures g 9 and 10 The mapping layer in DeFINE is in low-dimensional space and has correlations. As we learn deeper representations using DeF INE, these correlations are reduced and we obtain a cor- relation matrix similar to standard embedding layer. This suggests that DeFINE is effective in approximating the standard embedding layer. Importantly, the groups at different expansion lay- in DNFINE independent (see Figure suggesting these matrices learning different representations of their input.
*Corrllaton maps normalized between and 1.
15

---

Published as a conference paper at ICLR 2020
Standard embeddings
66
22
8.0
Projective embeddings
14
10
-1
24
-04
-02
02
-02
64
71 128
n1 256
INE
16
00
-04
24
-04
-02
02
-02
64
Y2 128
n 256
Figure 7: Standard embedding matrix approximations using projective embeddings and DeFINE for Transformer- XL different values of n.
16

---

Published as a conference paper at ICLR 2020
Map layer in DeF INE
Expansion layer in DeFINE
Expansion layer in Def
Reduction layer DeFIM
Figure 8: Layer-wise visualization of correlation maps of DeFINE embeddings when 64.
17

---

Published as a conference paper at ICLR 2020
Map layer in DeF INE
Expansion layer in DeFINE
Expansion layer in DeFINE
Reduction layer DeFIM ONFINE
Figure 9: Layer-wise visualization of correlation maps of DeFINE embeddings when 128
18

---

Published as a conference paper at ICLR 2020
Map layer DeF
Expansion layer in DeFINE
32
Expansion layer in DeF INE
Reduction layer in DeF INE
Figure 10: Layer-wiss visualization of correlation maps of DeF INE embeddings when n 256
Expansion layer 1 in DeFINE with groups
64
128
n 256
Expansion layer 2 in DeF INE with groups
64
T1
128
V1
256
Figure 11: Groups in expansion layers of DeFINE are orthogonal, suggesting these matrices learning different representations of their input.
19


Reviews:
[{"review_id": "Syl00P_TFH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper describes a new method for learning deep word-level representations efficiently. The architecture uses a hierarchical structure with skip-connections which allows for the use of low dimensional input and output layers, reducing total parameters and training time while delivering similar or better performance versus existing methods.\n\n1. From table 1a or table 2, the training time of the proposed method is not reduced compared with existing methods. \n\n2. It seems the number of parameters in DeFINE still depends directly on vocabulary size. Methods proposed in [1] and [2] do not depend directly on the vocabulary size.  For dataset that has very large vocabulary size, [1] and [2] could potentially have larger compression rate.\n\n3. The experiments are detailed, and includes ABLATION studies.  \n\n[1] Variani, Ehsan, Ananda Theertha Suresh, and Mitchel Weintraub. \"WEST: Word Encoded Sequence Transducers.\" ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2019.\n[2] Li, Z., Kulhanek, R., Wang, S., Zhao, Y., & Wu, S. (2018, April). Slim embedding layers for recurrent neural language models. In Thirty-Second AAAI Conference on Artificial Intelligence.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "SJeIxKeYqr", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper describes an approach to learn word embedding functions more efficiently and with fewer parameters. This is done by replacing the embedding lookup function which is typical in NLP tasks such as language modeling and machine translation with a hierarchical embedding model. This allows for a low dimensional embedding layer, reducing total parameters and training time. A novel skip-connections architecture is introduced as a part of the \"embedding generation model\". Experiments are conducted for language modeling and machine translation tasks and performance improvements are observed with a reduction in parameters and lesser training time.\n\nThe direction of this work is nice, the problem that is being tackled is indeed important.\nThe obtained results are nice (though this can be improved) and there is indeed some potential value in this work.\nHowever, I have the following concern. The paper completely ignores a lot of previous and concurrent work in reducing the size of the embedding layer. These works are in most cases not even cited and no empirical comparisons are provided. For example, please see below works in matrix factorization approaches, sparse word representation learning, codebook learning and other quantization approaches for compressing word embeddings:\n\nhttps://www.aclweb.org/anthology/P16-1022/\nhttps://aaai.org/ojs/index.php/AAAI/article/download/4578/4456\nhttp://web.cs.ucla.edu/~chohsieh/papers/Mulcode_Compressor.pdf\nhttps://www.aaai.org/ocs/index.php/AAAI/AAAI18/paper/viewFile/17042/16071\nhttps://storage.googleapis.com/pub-tools-public-publication-data/pdf/f158f7c81ed8e985fd51a20d193103ce427cad51.pdf\nhttps://arxiv.org/pdf/1711.01068.pdf\nhttps://arxiv.org/abs/1510.00149\n\nI would appreciate if comparisons with some of these approaches is provided in the next iteration of this work.\n\nOther suggestions:\n\n1. I think the paper would benefit from some analysis of the differences in the word embeddings learnt by a general lookup table learning model in comparison with the word embeddings learnt by this model. How are the embeddings compressed? How do the decompressed embeddings compare to the embeddings learnt by the lookup approach? More insights in the machinery via some visualizations would help.\n\n2. How do the gains of this method change as more or less training data is provided. For example, are the gains lesser on Gigaword? This would be interesting to know.\n\n3. GLT is mentioned twice in this paper. Perhaps a slightly more detailed explanation of the same would help improve the readability of this paper.\n\nBased on the presentation improvements and new experiments added to the paper in the rebuttal time period, I am updating my evaluation of this work.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HJgUu7or9B", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a novel sparse network architecture to learn word embeddings more effectively. \n\nI am not an expert in the area of machine translation, so I am able to sanity-check the results and the reasoning and motivation given in the paper. \n\nGenerally I failed to find motivation as to why this specific architecture was chosen out of many others. I also do not understand the purpose of doing aggressive embedding expansion before another contraction. Why would this allows to learn a more efficient low dimensional embedding than the original one? This may happen to be the case, but why?\n\nOverall, the results seem to be a bit inconclusive.\nTable 1: b) DeFINE uses less parameters but also gives worse results. This does not allow me to conclude anything.\nTable 1: c) DeFINE seems to give better perplexity results, while using less parameters. This is good.\nTable 2: DeFINE uses more parameters and gives better perplexity results. I do not know what to conclude, as ideally I would like to see how would DeFINE do with the same number of parameters.\nTable 3: \"our implementation\" seems to provide much lower scores than the ones found in the literature and thus can not be used as an fair baseline. Once this baseline is discarded, DeFINE seems to be producing worse results while using less parameters. Is this good? I do no know. But certainly this is inconclusive. I do not see how this table allows to conclude the following: \"DeFINE improves the performance by 2% while simultaneously reducing the total number of parameters by 26%, suggesting that DeFINE is effective\".\n\nA few other comments:\n\nFigure 1: if m >> n, why is the bottom (green) of DeDINE network wider than the top (tellow)?\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors design a deep model architecture for learning word embeddings with better performance and/or more efficient use of parameters.  Results on language modeling and machine translation are promising.  Pros:  Interesting idea and nice results.  New model may have some independent value beyond NLP.  Cons:  Empirical comparisons could be more thorough.  For example, it is not clear (to me at least) what would be the benefits of this approach applied to whole words versus a competitor using subword units.
Decision: Accept (Poster)