The Importance of the Current Input in Sequence Modeling
Authors: Christian Oliva
Luis F. Lago-Fernandez
Venue: ICLR 2022 Submitted (2022)

Abstract: The last advances in sequence modeling are mainly based on deep learning approaches. The current state of the art involves the use of variations of the standard LSTM architecture, combined with several tricks that improve the final prediction rates of the trained neural networks. However, in some cases, these adaptations might be too much tuned to the particular problems being addressed. In this article, we show that a very simple idea, to add a direct connection between the input and the output, skipping the recurrent module, leads to an increase of the prediction accuracy in sequence modeling problems related to natural language processing. Experiments carried out on different problems show that the addition of this kind of connection to a recurrent network always improves the results, regardless of the architecture and training-specific details. When this idea is introduced into the models that lead the field, the resulting networks achieve a new state-of-the-art perplexity in language modeling problems.

Under review as a conference paper at ICLR 2022
THE IMPORTANCE OF THE CURRENT INPUT IN SEQUENCE MODELING
Anonymous authors Paper under double-blind review
ABSTRACT
The last advances in sequence modeling are mainly based on deep learning ap- proaches. The current state of the art involves the use of variations of the standard LSTM architecture, combined with several tricks that improve the final prediction rates of the trained neural networks. However, in some cases, these adaptations might be too much tuned to the particular problems being addressed. In this ar- ticle, we show that a very simple idea, to add a direct connection between the input and the output, skipping the recurrent module, leads to an increase of the prediction accuracy in sequence modeling problems related to natural language processing. Experiments carried out on different problems show that the addi- tion of this kind of connection to a recurrent network always improves the results, regardless of the architecture and training-specific details. When this idea is in- troduced into the models that lead the field, the resulting networks achieve a new Mate-of-theear perplexity in language modeling problems.
INTRODUCTION
Deep learning models constitute the current state of the art in most artificial intelligence applica- tions, from computer vision robotics or medicine. When dealing with sequential data, Recurrent Neural Networks (RNNs), specially those architectures with gating mechanisms such as the LSTM (Hochreiter Schmidhuber 1997). the GRU (Cho et al.] 2014) and other variants, are usually the default choice. One of the most interesting applications of RNNs is related to the field of Natural Language Processing, where most tasks, such as machine translation, document summarization or language modeling, involve the manipulation of sequences of textual data. Of these, language mod- cling has been extensively used to test different innovations in recurrent architectures, mainly due to the ease of obtaining very large datasets that be used to train neural networks with millions of parameters.
Sequence modeling consists of predicting the next element in sequence given the past history. In language modeling, the sequence text, and hence the task is to predict the next word or the next character. In this context, some of the best performing architectures include the Mogrifier LSTM (Melis et al. (222)) and different variations of the AWD-LSTM (Merity et al. 2018). usually combined with dynamic evaluation and mixture of sofmaxes (MoS) (Wang al. 2019 Gong al.) 2018). These models obtain the best state-of-the-ar performance with moderate size datasets, such as the Penn Treebank (Mikolov al. 2010) or the Wikitext-2 (Merity et al. 2017 corpora, when no additional data are used during training. When larger datasets are considered, or when external data are used to pre-train the networks, attention- based architectures usually outperform other models [Radford et al. 2019 Brown al. 2020).
In this work we use moderate-scale language modeling datasets to explore the effect of mechanism recently proposed by Oliva & Lago-FFenández) (202I), when combined with different LSTMd models in the language modeling context. The idea consists modifying a recurrent architecture by introducing direct connection between the input and the output of the recurrent module. This has been shown to improve both the model's generalization results and its readability in simple tasks related to the recognition of regular languages.
In a standard RNN, the output depends only on the network's 's hidden state, h, which in turn depends on both the input, 2t, and the recent past, h,-1- - But there is no explicit dependence of the network's

---

Under review as a conference paper at ICLR 2022
output on its input. In some cases this could be shortcoming, since the transformation of æ needed to compute the network's 's internal state not necessarily the most appropriate to compute the output. However, explicit dependence of the output æt be forced by adding dual connection that skips the recurrent layers. We claim that this strategy may be of general application in RNN models.
To test our hypothesis perform thorough comparison of several state-of-the-ee RNN architec- tures, with and without the dual connection, on the Penn Treebank and the Wikitext-2 datasets. Our results show that, under all experimental conditions, the dual architectures outperform their non- dual counterparts. In addition, the Mogrifier-LSTT LSTM enhanced with a dual connection establishes a new state-of-the the-art word-level perplexity for the Penn Treebank dataset when additional data are used to train the models.
The remainder of the article is organized as follows. First, in section we present the different models we have used and the two possible architectures, the standard recurrent architecture and the dual architecture. In section B] we describe the datasets and the experimental setup. section we present our results. And finally, in section [ we extract some conclusions and discuss further lines of research.
2 MODELS
We start by presenting the standard recurrent architecture which is common to all the models. In absence of dual connection, the basic architecture involves embedding layer, recurrent layer and fully-connected layer with otmaz activation:
W Et. h REC(e,,
(1)
(2)
y: softmar(W + b²),
(3)
where W and b" are weight matrices and biases, respectively, and æ, is the input vector at time t. The REC module represents an arbitrary recurrent layer, with S,-1 being a set of vectors describing its internal state at the previous time step. In the most general case, this module will simply be an LSTM cell, but we consider other possibilities as well, as described below.
The dual architecture introduces an additional layer, with ReLU activation, which is fed with both the output of the embedding layer and the output of the recurrent module:
e Wen xx
(4)
h REC(e,,
(5)
d, = ReLU e, W h, + bd)
(6)
y: = softmaz Wid de + b²).
(7)
This way the network's input can reach the softmax layer following two different paths, through the recurrent layer and through the dual connection. In the following we consider different forms for the recurrent module in equations and
2.1 THE LSTM MODULE
In the simplest approach the recurrent module consists of an LSTM cell, where the internal state includes both the output and the memory, S, (h+ ct), which computed follows:
2

---

Under review as a conference paper at ICLR 2022
ft o ( + W W h: + b()
(8)
& o (W e + W 1 b¹)
(9)
or o e, W - + b²)
(10)
tanh(W"* + W h + b²)
(11)
ct f. 0-0 2 z1
(12)
h or O tanh(c:).
(13)
where, as before, W are weight matrices and b' are bias vectors. The @ operator denotes an element-wise product, and a is the logistic sigmoid function. For convenience, we summarize the
joint effect of equations 13 as:
h, LSTM(e:; (h--1;0-13).
(14)
In the literature it is quite common to stack several LSTT layers. Here we consider a double-layer LSTM, where the output h, of the recurrent module is obtained by the concatenated application of two LSTM layers:
h1 LSTMi(e; (
(15)
h; LST (1-11:0-1))
(16)
We refer to this double LSTM module as dLSTM:
h: dLSTM (h
(17)
LSTMG(LSTM(e,, (h/_1 d-13), (-1))
(18)
2.2 THE MOGRIFIER-LSTY MODULE
The Mogrifier-LSTM Meeis et al. 2020) is one of the state-of-the-art variations of the standard LSTM architecture achieving the lowest perplexity scores language modeling tasks. It basically consists of standard LSTM block, but the input e, and the hidden state h are transformed The mogrifier transformation involves several steps where e, and
before entering equations S 13
h modulate each other:
20 ei
1 C 4-2
for odd i E (1,2, r) for even E 21, 2.......?. r):
(19)
h RR (-1 o h 2-2
(20)
where Q' and R' weight matrices and we have e, e, and h h4-1- The linear transfor- mations Q' Q' and R'e:-1 can also include the addition of a bias vector, which has been omitted for the sake of clarity. The constant r is hyperparameter whose value defines the number of rounds of the transformation. We refer this recurrent module, including the mogrifier transformation and the subsequent application of the LSTM layer, as:
h mLSTM (fi-1:i--1 = LS) (hi-1:
(21)
where ef and h; are the highest indexed et and h in equations 19 and 20 O recovers the standard LSTM model.
Note that the choice
Melis et al. (2020) also used a double-layer LSTM enhanced with the mogrifier transformation. This strategy be summarized follows:

---

Under review as a conference paper at ICLR 2022
h = mdLSTM(er: (tw.ss/MMeettthiiooo------o tb
(22)
= MLSTMM(LLSTMiee (h (11-)1)),,
(23)
3 EXPERIMENTS
3.1 DATASETS
We perform experiments on two datasets: the Penn Treebank corpus (Marcus et al. 1993). as pre- processed by Mikolov et al. (2000), and the WikiText-2 dataset (Merity al. 2017) In both cases, the data are used without any additional preprocessing.
The Penn Treebank (PTB) dataset has been widely used in the literature experiment with language modeling. The standard data preprocessing is due to Mikolov et al. (2010), and includes transforma- tion of all letters lower case, elimination of punctuation symbols, and replacement of all numbers with a special token. The vocabulary is limited to the 10,000 most frequent words. The data is split into a training set which contains almost 930,000 tokens, and validation and test sets with around 80,000 words each.
The WikiText-2 (WT2) dataset, introduced by Merity et al. (2017), more realistic benchmark for language modeling tasks. It consists of more than 2 million words extracted from Wikipedia articles. The training, validation and test sets contain around 2, 2,025,000, 220,000, and 250,000 words, respectively. The vocabulary includes over 30,000 words, and the data retain capitalization, punctuation, and numbers.
3.2 EXPERIMENTAL SETUP
All the considered models follow one of the two architectures discussed in section 2 either the Embeediing-Recurret--SS -eee--eet--tottutmnneee tions 7 In either case, the recurrent module can be any of LSTM, dLSTM, or mdLSTM. Weight tying (Inan et al. 2017] Press & Wolf 2017) is used to couple the weight matrices of the embedding and the output layers. This reduces the number of parameters and prevents the model from learning a one-to-one correspondence between the input and the output (Merity et al., 2018).
(ERS) architecture (equations 13) or the dual architecture (equa-
We run two different sets of experiments. First, we analyze the effect of the dual connection by comparing the performances of the two architectures (ERS vs Dual), using each of the recurrent tuned for
modules, both the PTB and the WT2 datasets. In this setting the hyperparameters the ERS architecture, and then transferred to the dual case. Second, we search for the best hyper- parameters for the dual architecture using the mdI.STM recurrence, and compare the perplexity score with current state-of-the-an values. All the experiments have been performed using the Keras library (Chollet et al. 2015), and the implementation available in public Github repository!
The networks are trained using the Nadam optimizer (Dozat, 2016), a variation of Adam (Kingmal & Ba 2015) where Nesterov momentum is applied. The number of training epochs is different for each experimental condition. On one hand, when the objective is to perform a pairwise comparison between dual and non-dual architectures, we train the models for 100 epochs. On the other hand, when the goal is to compare the dual network with state of the art approaches, we let the models run for 300 epochs. We use batch sizes of 32 and 128 for the PTB and the WT2 problems, respectively, and set the sequence length to 25 in all cases. The remaining hyperparameters are searched in the ranges described in table
Finally, all the models are run twice, both with and without dynamic evaluation (Krause et al., 2018) Dynamic evaluation
standard method commonly used to adapt the model parameters, learned during training, using also the validation data. This allows the networks to get adapted to the new evaluation conditions, which in general improves their performance. In order to keep the models as simple as possible, no additional modifications have been considered.
The repository is yet available to preserve author anonymity. It will he relcased after the review process.
4

---

Under review as a conference paper at ICLR 2022
Table 1: List of all the hyperparameters and the search range associated with each of them. Those marked with an asterisk (*) refer to the dual architectures only.
Name
Description
Values
Num epochs
Number of training epochs.
{100, 300]
Learning rate
Learning rate.
[10 6 10
Batch size
Batch size.
(32, 128)
Seq len
Sequence length.
{ [10, 25, 50}
Embedding units
Size of the embedding layer.
(400, 850)
Recurrent units
Size of the recurrent layers.
4000 850, 1150} (1, 3} 400, 850}
LSTM layers
Number of recurrent layers.
Dual units"
Size of the dual layer.
Embedding 1.2reg
1.2 regularization applied to the Embedding and output layers.
(0, 10-6, 10-5)
Rec. imput L2reg
L. regularization applied to the input weights of the recurrent layer.
(0, 10-6: 10 10-6)
Rec. L2veg
L2 regularization applied to the recurrent weights of the recurrent layer.
{0, 10-10.1,100 1000,0100 10
Activation L2reg
L2 regularization applied to the recurrent layers output.
80,10-8,
Dual L2reg"
L2 regularization applied to dual layer.
{0. 10-6, 10.10 10
Rec. imput Dropout
Dropout before the first recurrent layer. Dropout for the linear transformation of the recurrent state.
[0.0, 0.5] [0.0, 0.5]
Rec. Dropout
Rec. internal Dropout
Dropout between the recurrent layers. Dropout after the last recurrent layer. Dropout before the dual layer. Dropout after the dual layer.
[0.0, 0.5]
Rec. output Dropout
[0.0, 0.5)
Dual input Dropout"
[0.0, 0.5]
Dual output Dropout*
[0.0, 0.5]
Mogrifier deep Mogrifier L2reg
Mogrifier rounds.
(0, 2, 3, 6,
L2 regularization applied to Mogrifier weights. Weight factorization. Q' <R QiQ; with Qi E m Q R Dropout between the Mogrifier weights
{0, 10 1]
Mogrifier rank
(0, 50, 100, 200)
Mogrifier Dropout
[0.0, 0.2]
Learning rate eval Seq len eval Clipnorm eval
Learning when Dynamic evaluation. Sequence length when Dynamic evaluation. Gradients clipping maximum norm.
[10-6. 10-3] [5, 50] [0.0, 1.0]
4 RESULTS
We first show the results of the comparative analysis ERS vs Dual, then we focus on the search of the optimal hyperparameters for the dual architecture with the mdL.TM recurrence.
4.1 DUAL VS NON-DUAL ARCHITECTURES
Table displays the validation and test perplexity scores obtained for each of the experimental configurations the PTB and the WT2 problems, both with and without dynamic evaluation. To facilitate the comparison, each pair of rows contain the results for one of the recurrent modules (LSTM, dLSTM M mdLSTM. M) using the two architectures ERS and Dual, with the best values shown in bold. each case, the hyperparameters are tuned for the standard ERS architecture and then used within the dual networks without any additional adaptation. The exceptions are hyper-
5

---

Under review as a conference paper at ICLR 2022
Table 2: Validation and test word-level perplexity obtained for each of the experimental configura- tions on the PTB (top) and the WT2 (bottom) datasets.
Penn Treebank Dataset
No Dyneval
Dyneval
MODEL
No. PARAMS
Val.
Test
Val.
Test
LSTM
8.88 M
67.37
64.91
62.31 55.26
61.17
Dual LSTM
9.60 M
61.22
59.39
54.69
dLSTM
13.62 M
63.44
61.03
57.18
56.01
Dual dLSTM
13.94
60.99
59.56
56.11
54.87
mdLSTM
21,43 M
57.42
55.48
51.16
50.27
Dual mdLSTM
22.88 M
56.08
54.12
48.82
48.00
MdM
22.16
57.77
56.29
50.42
49.83
WikiText-2 Dataset
No Dyneval
Dyneval
MODEL
No. PARAMS
Val.
Test
Val.
Test
LSTM
20.23 M
92.84
88.28
74.98
69.42
Dual LSTM
20.95
85.88
82.48
61.94
57.61
dLSTM
29.60
78.65
75.60
63.26
59.42
Dual dLSTM
30.32 M
77.01
73.90
61.10
57.10
mdLSTM
37.51
72.05
69.06
57.42
53.93
Dual mdLSTM
38.95 M
71.78
70.83
53.48
50.71
parameters, such the dual dropout, which do not exist in the ERS configuration (those marked with an asterisk in table []. To give measure of the model complexity, table 2 contains also the approximate number of trainable parameters for each configuration.
As expected, dynamic evaluation improves the results regardless of the model or the dataset. The main observation, however, is that networks enhanced with the dual connection display lower per- plexity scores for almost all the training conditions on both the PTB and the WT2 datasets. The advantage of the Dual vs the ERS architecture larger for less complex models, and narrows as the model complexity increases. Nevertheless, even for networks with mdL.ST recurrence, the dual architectures outperform their non-dual counterparts more than 2 perplexity points on the test set, when dynamic evaluation is used.
In order to test that this improvement is due to the dual connection and not to the presence of an extra processing layer, we performed an additional experiment with Dual mdLSTM model, but removing the term Wde e, from equation 6 The results for the PTB dataset are shown in table 2 as mdI.STM+. Note that, in spite of slightly improving the baseline, this enhanced mogrifier model still well below the result obtained with the full dual architecture.
Finally, it is worth noting that all the results presented correspond to our own implementation of the models, and that in most cases are not including some of the several training validation adaptations frequently used in the literature (such as AWD or MoS, for example). This can explain the difference with respect to the results reported by Melis al. (2020) for the Mogrifier-LSTM model. We would expect a further improvement of the results were these additional mechanisms implemented.

---

Under review as a conference paper at ICLR 2022
Table 3: Best validation and test word-level perplexity scores reported in the literature for the Penn Treebank dataset, with and without dynamic evaluation. Missing values in the last two columns correspond to works where the dynamic evaluation approach was not considered. The last row in the table displays the results obtained with our Dual mdLSTM network.
No Dyneval Test
Dyneval Test
MODEL
Val.
Val.
AWD-LSTM (Merity et al.
2018
24 M
60.00
57.30
AWD-LSTM-DOC (Taaase et
2018
23 M
54.12
52.38
AWD-LSTM (Krause et al. 2018
24 M
59.80
57.70
51.60
51.10
AWD-LSTM +PDR (Brahma 2019
24 M
57.90
55.60
50.10
49.30
AWD-LSTM +.MoS Yang et al. 2018
22 M
56.54
54.44
48.33
47.69
AWD-LSTM +MMSS +PDR BBahma
2019
22 M 56.20
53.80
48.00
47.30
AWD-LSTM-DOC x5 (Takase et al. 2018
185 M
48.63
47.17
AWD-LSTM +MoS +FRAGE Gong et al.
2018
24 M
55.52
53.51
47.38
46.54
AWD-LSTM +MoS +Adv (Wang al. (0019)
22 M 54.98
52.87
47.15
46.52
AWD-LSTM +MoS +Adv +PS Wang et al.
2019
22 M 54.10
52.20
46.63
46.01
Mogrifier- LSTM (Melis et al. 2020)
24 M 51.40
50.10
44.90
44.80
Dual MSL ours
23 M
52.87
51.19
45.13
44.61
4.2 DUAL MOGRIFIER FINE TUNING
The second part of the experiments consists of searching for the best hyperparameters in the con- figuration that provided the smallest perplexity in the previous setup, that is the Dual mdLSTM architecture. We carry out this experiment with the FTB problem. After an extensive search (see table the best performance is obtained with model with 850 units in the embedding layer, 850 units in each the mogrifier LSTM layers, and 850 units also in the dual layer. The input, recurrent, internal, and output dropout rates are all set to 0.5, the dual input and output dropout rates set to 0.5 and 0.4, respectively, and the mogrifier dropout rate is set to 0.15. Both the embedding and the dual 1.2 regularization parameters set to 10-5 The mogrifier number of rounds is set and the rank to 100. All the remaining hyperparameters are set to 0.
After the training phase, continue with fine tuning of some additional hyperparameters, using the validation data. First, we look for the best sequence length in the range [5, 70). and then we fine- tune the softmax temperature in the range [0.9, 1.3). When using dynamic evaluation, we also look for the best gradient clipping value (in the range [0.0, 1.0)) and, following Melis et al. (2020), we repeat the whole procedure with the 31 parameter of the Nadam optimizer set 0, which resembles the RMSProp optimizer without momentum. The results are shown in table 3 together with the top perplexity scores reported in the literature for the same problem.
The state-of-the-art is dominated by several variations of the AWD-LSTM network (Merity et al., 2018), the most common being the inclusion Mixture of Softmaxes (MoS) (Yang et al. 2018). Other add-ons include Direct Output Connection (DOC) (Takase et al., 2018). which is a general- ization of MoS, Frequency Agnostic word Embedding (FRAGE) (Gong et al. 2018), Past Decode Regularization (PDR) (Brahmal 2019). or Partial Shuffling (PS) with Adversarial Training (Adv) (Wang et al.] 2019). The mogrificr-LSTM described section 2.2 combines many of these ideas with mutual gating between the input and the hidden state vectors obtain the best results reported in the literature for the PTB problem, among those obtained by networks that do not use additional data during the training phase. Compared with all these models, our current approach leads the ranking with a perplexity score of 44.61, even though most of the aforementioned tricks have not been considered.
DISCUSSION
In this work, have presented network design for the Language Modeling task based on the dual network proposed by Oliva & Lago-Fernández (202I). This network adds direct connection between the input and the output, skipping the recurrent module, and can be adapted to any of the
7

---

Under review as a conference paper at ICLR 2022
traditional Embedding- Sebeeeddinn-Reuureen---e Recurrent-Softmat (ERS) models, opening the way this task. We have based our work on the Penn Treebank (Mikolov et al. 2010) and the WikiText-2 (Merity et al. 2017) datasets, comparing the ERS approach and its dual alternative. Regardless of the configuration, the dual version performs always better, even though it faces a slight disadvantage, since most of the hyperparameters performance if the complete set of hyperparameters is properly tuned for the dual network.
approaches for
tuned using the ERS model. We can expect much better
This is in fact the case for the second experiment, where Dual mdLSTM, which includes simpli- fied version of the mogrifier LSTM (Melis et al. 2020) within a dual architecture, is fine tuned for the Penn Treebank dataset. After a thorough search of the hyperparameters space, we have found a network configuration that establishes new pate-oo--hee-an score for this problem. Interestingly, this new record has been obtained spite of leaving aside many the standard features used in most maae-offtheear approaches, such as the Averaged SGD Weight-Drop (AWD) (Merity et al.) 2018) or the Mixture of Softmaxes (MoS) (Yang et al. 2018). The incorporation of these features into the dual architecture can be expected to further increase the model performance.
The dual architecture was firstly proposed as an alternative that reduces the computational load on the recurrent layer, letting it concentrate on modeling the temporal dependencies only. From more abstract point view, has been argued that the dual architecture can be understood as a sort of Mealy machine, where the output explicitly depends on both the hidden state and the input (Oliva & Lago-Fernández) 2021). Our results show that this explicit dependence on the input can indeed lead to better performance on language modeling tasks. This emphasizes the importance of the current input in RNN models.
Finally, although the new approach has not been tested with large-scale language corpora, expect that our results scale well to larger datasets. Work in progress contemplates this extension. The dual architecture also needs further research concerning the deepness of the specific variations of Language Modeling and other families of problems not necessarily related to Natural Language Processing. This work opens a new line of research be considered when processing any sequence or time series. The utility of this approach in more general problems will be addressed in future work.
ACKNOWLEDGMENTS
Omitted to preserve anonymity.
REFERENCES
Siddhartha Brahma. Improved language modeling by decoding the past. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguisties, pp. 1468-1476, Florence, Italy, July 2019, Association for Computational Linguistics.
Tom Brown, Benjamin Mann, Nick Ryder. Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Prucessing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., 2020.
Kyunghyun Cho, Bart van Merrienboer, Çaglar Gülçehre, Dzmitry Bahdanau, Fethi Bougares, Hol- ger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In Alessandro Moschitti, Bo Pang, and Walter Daelemans (eds.). Proceedings of the 2014 Conference Empirical Methods in Natural Language Process- ing, EMNLP 2014, October 25-29, 2014, Dohu, Qatar, A meeting of SIGDAT, a Special Interest Group of the ACL, pp. 1724-1734. ACL, 2014.
François Chollet et al. Keras. https://keras:: 10. 2015.
Timothy Dozat. Incorporating nesterov momentum into adam. In Proceedings of 4th International Conference Learning Representations, Workshop Track, 2016.
8

---

Under review as a conference paper at ICLR 2022
Chengyue Gong, Di He, Xu Tan, Tao Qin, Liwei Wang, and Tie- Yan Liu. Frage: Frequency-agnose word representation. In Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa aasa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 9(8): 1735-1780, November 1997. ISSN 0899-7667.
Hakan Inan, Khashayar Khosravi, and Richard Socher. Tying word vectors and word classifiers: A loss framework for language modeling. In Proceedings of the 5th International Conference on Learning Representations, 2017.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
Ben Krause, Emmanuel Kahembwe, lain Murray, and Steve Renals. Dynamic evaluation of neural sequence models. In Jennifer Dy and Andreas Krause ), Proceedings of the 35th International Conference on Machine Learning. volume 80 of Proceedings of Machine Learning Research, pp- 2766-277 2775. PMLR, 10-15 Jul 2018.
Mitchell P. Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. Building a large annotated corpus of english: The penn treebank. Comput. Linguist., 19(2):13-330, June 1993. ISSN 0891-2017.
Gábor Melis, Tomás Kočiský, and Phil Blunsom. Mogrifier LSTM. In International Conference on Learning Representations, 2020.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In 5th International Conference Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017.
Stephen Merity, Nitish Shirish Keskar, and Richard Socher. Regularizing and optimizing LSTM language models. In International Conference on Learning Representations, 2018.
Tomas Mikolov, Martin Karafiát, Lukás Burget, Jan Cermocky, and Sanjeev Khudanpur. Recur- rent neural network based language model. In Takao Kobayashi, Keikichi Hirose, and Satoshi Nakamura (eds.), INTERSPEECH, pp. 1045-1048. ISCA, 2010.
Christian Oliva and Luis Lago-Fernández. Separation of memory and processing dual recurrent neural networks. In Igor Farkaš, Paolo Masulli, Sebastian Otte, and Stefan Wermter (eds.), Artifi- cial Neural Networks and Machine Learning - ICANN 2021, pp. 360-371, Cham, 2021. Springer International Publishing. ISBN 108-3-000086380--
Ofir Press and Lior Wolf. Using the output embedding to improve language models. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pp. 157-163, Valencia, Spain, April 2017. Association for Computational Linguisties.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
Sho Takase, Jun Suzuki, and Masaaki Nagata. Direct output connection for high-rank language model. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Pro- cessing, pp. 4599-4609, Brussels, Belgium, October-November 2018. Association for Computa- tional Linguistics.
Dilin Wang, Chengyue Gong, and Qiang Liu. Improving neural language modeling via adversar- ial training. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 6555 6565. PMLR, 09-15 Jun 2019.
Zhilin Yang, Zihang Dai, Ruslan Salakhutdinov, and William W. Cohen. Breaking the softmax bottleneck: A high-rank RNN language model. In International Conference on Learning Repre- sentations, 2018.


Reviews:
[{"review_id": "_x3ZIhZbtQK", "reviewer": "Reviewer_6pt8", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I found the paper to be clearly written and easy to follow. This is an experimental paper, and the authors have taken care to describe the setup in sufficient detail. I also appreciated the use of two separate sets of experiments, one with hyperparameters for the baselines transferred over to the \"dual\" models, and another with hyperparameters tuned for the \"dual\" models.\n\nThere are a few concerns that make it difficult for me to accept this paper in its current form:\n\n1. It is unclear where the improvement is coming from. Is the added dual connection improving ease of optimization, model capacity, or regularization (providing a useful inductive bias)? It's possible that this will be much clearer if the training and validation loss curves are compared for models in Table 2. Results for additional baseline models with slightly larger sizes might also help: those models should have slightly more capacity in principle, so how do their training and validation loss curves compare to the \"dual\" models.\n2. Related to the above question: it appears that Melis et al. [A] use skip connections in their LSTMs to improve optimization, but this paper doesn't appear to use them. This appears to be very relevant here if the improvement from the proposed dual layer is coming from improved optimization. Would simpler skip connections provide a similar benefit? This can be checked by modifying the baseline to use additive skip connections.\n3. Overall the experimental results are not strongly in favor of the proposed model. In Table 2, we find that dual layer leads to worse results without dynamic evaluation when added to \"mdLSTM\", but better results with dynamic evaluation. We don't know why this should be the case, and it casts doubt on a universal benefit of this design. We see a similar situation in Table 3 too, where the results are worse without dynamic eval, as well as with dynamic eval on the val set. Only on the test do we see a small improvement. It is very difficult to say from these results that the proposed design can be expected to improve performance in general. On this point, I don't have concrete suggestions to offer, but perhaps the authors can address this by adding experiments on more datasets, and/or clarify the reason for these mixed results.\n\nIf 1 & 2 are satisfactorily addressed, I'm willing to raise my score up to 6. If 3 can also be addressed, I will increase the score further.\n\n[A] Melis, G., Dyer, C., & Blunsom, P. (2017). On the state of the art of evaluation in neural language models. arXiv preprint arXiv:1707.05589.\n\n————\nUpdate:\nI am thankful to the authors for providing additional results and analysis. My concerns are partially addressed so I am increasing my score. However, I believe more work is needed to make sure that the increased capacity provided by the _dual_ connection can actually be useful in practice. What I mean by this is that many results in Table 2, while useful from a comparative perspective, feel less relevant from a practical perspective because they show that the _dual_ connection improves performance of models _underpowered for the task/dataset_ at hand. The most relevant results are when we get close to the “best” test set performance for the task. It is here the evidence gets fuzzy, and the use of only 2 datasets makes the issue difficult to ignore. \nCan I take a setup that’s otherwise well tuned for a task, add this connection to improve capacity, and expect improvement? If not, why not, and under which conditions should I expect it? I don’t have sufficient confidence to answer these questions based on the results so far. The experiments suggested by the authors in the responses and in the paper’s Discussion section may be necessary to provide that confidence.", "summary_of_the_paper": "This paper proposes a simple improvement to recurrent network architectures for language modeling. The idea is to insert a single additional layer into a network with one or more recurrent layers, just before the output layer (Eq. 6). This is termed a \"dual connection\" and it combines the output of the last recurrent layer directly with the input to the network at the current step. The paper presents the results of adding this modification to both LSTM and Mogrifier-LSTM networks on the Penn Treebank and WikiText-2 datasets.\n", "summary_of_the_review": "The paper proposes a simple modification to recurrent language models that can potentially improve performance, but additional analysis and results are needed to establish the utility of the approach.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "YK2CsfuhOvM", "reviewer": "Reviewer_m9iZ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n+ The technique is very simple and is applicable to any embedding-recurrent-softmax (ERS) architecture.\n+ The presentation of the method is very clear. Sufficient details were given on the experiment setup and hyperparameter search.\n\nWeaknesses:\n+ The proposed technique, while simple, is not very novel. A very similar technique has been examined in a previous paper (https://ieeexplore.ieee.org/document/9207238), and similar experiments on PTB have also been conducted.\n+ It's hard to justify whether the performance gain in the Table 1 actually due to the residual connections or because of the extra input projections introduced. The authors could have examined this by simply doing a version of the residual connection by directly adding the input embedding $x_t$ to the hidden state $h_t$, which will incur no extra parameters at all.\n+ After hyperparameter tuning, the performance gain brought by the architectural change becomes very marginal.\n\nI also have a question for the author: why is the parameter number for Dual mdLSTM smaller than Mogrifier-LSTM in Table 3?\n\n=== UPDATE ===\n\nI have read the authors response and the other reviewers' reviews and do not think any change in my overall evaluation is justified. See my detailed comment below.", "summary_of_the_paper": "This paper proposes to add an extra residual connection between a transformed word embedding and the final output layer, which easily generalizes over different recurrent architectures. The language model experiments show that their proposed model performs better than the same recurrent model without the residual connection, but the performance gain becomes marginal after optimal hyperparameter setups were used.", "summary_of_the_review": "The idea is simple and generally applicable, but the lack of novelty and the marginal result improvement greatly limit the impact of the paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "XPnfz9pwiNf", "reviewer": "Reviewer_zNdy", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n\nIt's good to revisit the LSTM recurrent structure.\n\nHyper-parameter tuning is reported in detail.\n\nI got a bit surprised by the improvement from the dual structure.\n\nWeakness:\n\nThe used datasets like PTB or WT2 are small-scale. I'd like to see something like wiki-103. \n\nWhy there's no comparison with transformers?\n\nNot much analysis or explanation about why the dual structure is better is given.\n\nBelow are some minor comments:\n\nI sometimes got confused by \"dual\" and \"double\" in the writing.\n\nSome possible typos:\n\nSec1: \"for the Penn Treebank problem \"-> PTB \"dataset\"?\n\nSec2.2: \"achieving the highest perplexity scores\" -> lowest PPL?", "summary_of_the_paper": "This work revisits the LSTM architecture. They propose to modify a recurrent architecture by adding a direct connection between the input and the output of the recurrent module, called \"dual\". They also consider a double-layer LSTM, where the output of the recurrent module is obtained by the concatenated application of two LSTM layers.\nIn experiments, the dual modification can get consistent improvement.", "summary_of_the_review": "While I think this revisit to LSTM is interesting, the scope of the experiments are still limited. Comparison with transformers is not conducted, and the datasets are small scale.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper considers augmenting LSTM language models with a form of residual connection that adds and additional feed forward layer before the softmax that integrates the output of the recurrent cell with the input embedding. This architectural variation is evaluated on the standard Penn Treebank and Wikitext-2 language modelling tasks and shown to lead to lower perplexities on the test sets, particularly when dynamic evaluation is used. 

The reviewers agree that the proposed addition is well motivated, however they also observe that there has been substantial work in language modelling on various forms of residual and skip connections and it is not clear how this work relates to that body of work. The authors have provided some additional comparisons during the discussion, however the reviewers feel that further evaluation and analysis is needed. There was also some additional confusion about the varying hyperparamter tuning protocols employed in the different evaluations. The author’s have clarified this in their response so that it is clearer how the different results were obtained.

Overall this paper presents an promising initial result, but it would benefit from more complete evaluation, analysis, and hyperparameter tuning. This could include ablation studies and analysis to shed more light on what the proposed architectural addition is contributing, how this relates to other varieties of residual connection, and it’s positive interaction with dynamic evaluation. It would also be useful to include a tuned model with a comparison to previously reported Wikitext-2 results.
Decision: Reject