Autoencoder-based Initialization for Recurrent Neural Networks with a Linear Memory
Authors: Antonio Carta
Alessandro Sperduti
Davide Bacciu
Venue:  (2020)
Keywords: recurrent neural networks
autoencoders
orthogonal RNNs

Abstract: Orthogonal recurrent neural networks address the vanishing gradient problem by parameterizing the recurrent connections using an orthogonal matrix. This class of models is particularly effective to solve tasks that require the memorization of long sequences. We propose an alternative solution based on explicit memorization using linear autoencoders for sequences. We show how a recently proposed recurrent architecture, the Linear Memory Network, composed of a nonlinear feedforward layer and a separate linear recurrence, can be used to solve hard memorization tasks. We propose an initialization schema that sets the weights of a recurrent architecture to approximate a linear autoencoder of the input sequences, which can be found with a closed-form solution. The initialization schema can be easily adapted to any recurrent architecture.
    We argue that this approach is superior to a random orthogonal initialization due to the autoencoder, which allows the memorization of long sequences even before training. The empirical analysis show that our approach achieves competitive results against alternative orthogonal models, and the LSTM, on sequential MNIST, permuted MNIST and TIMIT.

Under review as a conference paper at ICLR 2020
AUTOENCODER-BASED INITIALIZATION FOR RECUR- RENT NEURAL NETWORKS WITH A LINEAR MEMORY
Anonymous authors Paper under double-blind review
ABSTRACT
Orthogonal recurrent neural networks address the vanishing gradient problem by parameterizing the recurrent connections using an orthogonal matrix. This class models is particularly effective to solve tasks that require the memorization of long sequences. We propose an alternative solution based on explicit memorization using linear autoencoders for sequences. We show how a recently proposed recurrent architecture, the Linear Memory Network, composed of a nonlinear feedforward layer and a separate linear recurrence, can be used solve hard memorization tasks. We propose an initialization schema that sets the weights of a recurrent architecture approximate linear autoencoder of the input sequences, which can be found with closed-form solution. The initialization schema can be easily adapted to any recurrent architecture. We argue that this approach is superior to a random orthogonal initialization due to the autoencoder, which allows the memorization long sequences even before training. The empirical analysis shows that our approach achieves competitive results against alternative orthogonal models, and the LSTM, on sequential MNIST, permuted MNIST and TIMIT.
INTRODUCTION
Several sequential problems require the memorization of long sequences of patterns. As an example, a generative model for music should be able to memorize long sequences of notes and be able to repeat them, as it is typically done in musical pieces. RNNs and LSTMs struggle to solve even simple memorization tasks (Arjovsky et al. 2013 Graves et al. 2014). Therefore, it is important to study alternative solutions this problem.
Orthogonal RNNs are a class of recurrent architectures that solve the vanishing gradient problem by constraining the recurrent connections to be orthogonal unitary matrix (Arjovsky et al.) (015). They are particularly effective to solve long memorization tasks Henaff et al. (2016).
In this paper, we address the problem of memorization with orthogonal RNNs and linear autoen- coders. Our objective is to find a solution to the problem of memorization of long sequences. The memorization of long sequences with orthogonal models can require large number of hidden units, increasing the hidden state size and the cost in time and space.
If we assume that the input sequences in the training data lay in low-dimensional manifold, as is typically believed for real-world data, then we can train an autoencoder with a small number of hidden units, sufficient to encode the entire sequence.
If we restrict ourselves to the linear case, we can compute the optimal autoencoder with a closed- form solution (Sperduti) 2013). This result can be exploited initialize recurrent architectures to approximate the linear autoencoder for the input sequences. In our experiments we use the RNN (Elman 1990) and the Linear Memory Network (L.NN) (Bacciu et al. 2019). The LMN with the autoencoder initialization is recurrent architecture equivalent to the Elman RNN but able to solve the vanishing gradient problem and memorize input sequences with a minimal hidden state.
We test our approach classic benchmarks for orthogonal RNNs, showing that our proposed ap- proach behaves similarly to orthogonal architectures on pure memorization tasks, and even improving the performance real-wort wolld datasets. Finally, we show that the model can also be used in situa- tions where a strict orthogonal parameterization struggles (Vorontsov et al., 2017), like the TIMIT benchmark (Garofolo et al. 1993).

---

Under review as a conference paper at ICLR 2020
Work orthogonal models typically focuses on the properties of orthogonal parameterization the backward step, to address the vanishing gradient. In our work instead, we focus on the forward step. by investigating the effect of orthogonal parameterization against autoencoder rrooencoerr-bsedd based solution. For these reasons, we are not particularly interested enforcing exact orthogonality constraints, but in the study of the effectiveness of autoencoder-based memorization mechanism. Our proposed approach requires the memorization of the entire sequence within the hidden state activations. It necessary note that this approach can be quite inefficient whenever the underlying task does not require complete memorization of the input sequences. In this situation, the hidden state size necessary to encode the entire sequence could be much larger than the minimal hidden state size necessary to solve the problem. Therefore, it is fundamental to allow the model to diverge from the orthogonality or the autoencoder solution by only imposing soft orthogonality constraints. Nonetheless, we believe that even when complete full memorization is not necessary, the autoencoder initialization can help to speed up training convergence by allowing the model to remember long sequences, which can then be gradually forgotten during training if deemed uninformative.
In summary, the main contributions the paper are:
the proposal of novel initialization schema designed for the explicit memorization of long sequences;
highlighting the connection between orthogonal models and linear autoencoders for se- quences:
an empirical analysis that shows the effectiveness of our proposed initialization schema to solve tasks that require long memorization.
LINEAR MEMORY NETWORKS
The Linear Memory Network (LMN) (Bacciu al. 2019) is recurrent neural network that computes a hidden state h' and separate memory state m The hidden state he is computed with fully connected layer from the input and the previous state followed by an activation function. The memory state m' is computed with a linear transformation of the current hidden state and the previous memory state. The equations for the model update are the following:
ht = ooWaaa' + W Wmimn'
m = Wamh' h' + W m
where W W mh, Whm. W mm the model's parameters and is non-linear activation function (tanh in our experiments), and m' is the final output the layer. The linear recurrence can be used to control the memorization properties of the model, as will show in the subsequent sections.
The linearity of the memory update can be exploited to guarantee a constant propagation of the gradient by constraining W.. to be an orthogonal matrix. This property allows the LMN to avoid the vanishing gradient problem. Notice that to guarantee the constant gradient propagation
ak* it is also necessary to truncate the gradient between m and h by setting Bm--T 0. This approach is similar the truncated backpropagation proposed originally for the LSTM (Hochreiter Schmidhuber (197), which is necessary to guarantee that an LSTM without forget gate propagates the gradients without vanishing effects. The modification to the training algorithm trivial to implement with the current automatic differentiation packages, done in (Kanuparthi et al., 2018)) However, in this work, we focus on the effect of the orthogonality on the forward propagation and therefore we only use the full gradient in our experimental results.
2.1 EQUIVALENCE LMN-RNN
The linearity of the memory update of the LMN might seem major limitation of the architecture. However, is easy to show that an RNN can always be rewritten as an equivalent LMN. Given an RNN with parameters V and U such that h h (V + Uh)-1) can define equivalent
2

---

Under review as a conference paper at ICLR 2020
LMN such that m' hr Vt by setting the parameters in the following way: Wah V, Wmb U, hm W mm
Therefore, the linearity does not limit the expressivity of the LMN. Differences between the two architectures however become important during training. The linear update of the LMN without proper regularization
be affected by the exploding gradient. This problem is less frequent in RNNs due to the non-linearity (which often favors the vanishing gradient). preliminary experiments, we found that training LMNs without any kind of regularization can lead instability during the training. Fortunately, the gradient propagation can be controlled in the linear recurrence by tuning the spectral radius p of the recurrent matrix W wm. for example, by using an orthogonal parameterization. In our experiments we use soft orthogonality constraints by adding regularization term \/ W W-1 - as in Vorontsov et al. (2017). Alternatively, is possible build fading memory system by imposing Tino et al. 2004).
Another possible cause of instability is the uncontrolled growth of the norm of the memory state m'. The LSTM solves this problem by using forget gate (Cummins al.) 2000) to reset the memory cell activations. This is not an issue in RNNs with bounded nonlinear activation. We overçome
this problem by adding to the loss function a regularization term 3 Σ penalizes the norm of the hidden activations as in Krueger & Memisevic (2015).
+== (11lll2 || )² that
INITIALIZATION WITH A LINEAR AUTOENCODER FOR SEQUENCES
The linear recurrence the LMN allows to exploit results from the field linear autoencoder for sequences (LAES) (Sperduti) 2013 2015 Pasa & Sperduti 2014).
In this section, we show how to initialize the LMN and RNN models using linear autoencoder for sequences trained on the input samples. This initialization allows the resulting model to encode the input sequences into a minimal hidden state.
3.1 LINEAR AUTOENCODER FOR SEQUENCES
A linear autoencoder for sequences (LAES) (Sperduti 2013) linear model composed of linear autoencoder, which is a linear dynamical system, and a linear decoder. The autoencoder takes as input sequence x= ...., x² and computes internal state mn' with the following equations:
m'
Ax' Bm
2t m
= Cm2,
where A and B are the encoder parameters, C' is the decoding matrix and a' and vn !-1 are the reconstructions of the current input and the previous memory state.
SSerrduti (013)) provides closed- solution for the optimal linear autoencoder. ing decoder matrix C can be reconstructed from the autoencoder parameters as C
The correspond-
B
3.2 LAES INITIALIZATION
By means of the linear autoencoder for sequences, we can construct a simple linear recurrent model which uses the autoencoder to encode the input sequences within a single vector, the memory state of the autoencoder. To predict the desired target, can train linear layer that takes input the states of the autoencoder to predict the target. The equations for the model are as follows:
m Ax Bm
y W.m'
(2)
where and are the parameters of the autoencoder trained to reconstruct the input sequences and W, are the parameters of the readout trained to the predict the target. Figure la shows schematic view of the architecture.
Pasa & Sperduti 2014) propose initialize RNN with the linear RNN defined in Eq. T [nnd]
2
3

---

Under review as a conference paper at ICLR 2020
y¹
y¹
y¹
W
W
W
m
B
h
h
m'
B
0
A
A
x²
x²
x
(a) Linear RNN.
(b) RNN with LAES initialization.
(c) LMN with LAES initialization.
Figure 1: Linear RNN and initialization schemes for RNN and LMN. Notice how the nonlinearity only affects the imput in the LMN with LAES initialization. Blue nodes highlight the LAES memory state.
h' = tanh(Aa +Bh y' W.h
(3)
(4)
This initialization approximates the linear RNN. The quality of the approximation depends on the values of the hidden activations m'' With small values close to zero, the tanh activation function is approximately linear, and therefore obtain good approximation of the autoencoder. However, for large values the correspondence between the autoencoder activations and the initialized RNN degrades quickly since the tanh activation function enters into the nonlinear part and cannot represent values > or < -1. In practice, we find that the correspondence between the linear autoencoder and the hidden states of the initialized RNN with tanh activation function tends degrade quickly due to the accumulation of the error through time.
To solve this problem, propose adopt similar initialization scheme initialize the weights of an LMN using the linear RNN defined in Eq. [ and The LMN model is initialized as follows:
h' 0 Ax
Omn"
m 1 Ih' Bm !-1
y'
W.m
where the parameters of the LMN are initialized to approximate the linear RNN: W., = A, Wm 0, hm = I, Wmm = B. The initialization only an approximation of the linear RNN but since the nonlinearity only affects the input transformation Ax and not the recurrent part By the approximation is closer to the original linear RNN. Figures Tb and show a schematic view of the architectures. Experimental results in Section 5 provide quantitative comparison between the two approaches. These results show that the accuracy of the initialized LMN is the same as the linear RNN.
DECODING WITH THE UNROLLED MODEL
To give an intuition on the operations enabled by the use of an autoencoder-based memorization mechanism, we discuss how it is possible to use the decoder to build explicit model that is able to reconstruct the entire sequence, and to process the unrolled sequence with a feedforward layer that combines weights for each specific time delay. Let assume to have the optimal autoencoder with parameters A and B trained on the input sequence x! r!..... xT. autoencoder has enough hidden units have exactly zero reconstruction error. The output of the
We further assume that the
4

---

Under review as a conference paper at ICLR 2020
entire model can be computed with linear layer follows:
y' W'm',
(5)
where y" is the output vector time and W the parameters of the output layer. Given the hidden state m at time t of the autoencoder, we reconstruct the input at time - k using the decoder
C
In fact, after applications of the decoder we find:
B
x オーー A (B m'
(6)
which shows how to reconstruct any input vector x', = l....., from the autoencoder memory m'. The output layer can reconstruct the entire input sequence using the decoder. This can be made explicit with a decomposition of W as:
W
W
W
A (B (BT
where W is the matrix applied to the input at delay i. Using this decomposition, the output y' can be expressed as:
y' T
W
This resulting model requires explicit parameters for each time delay, with a total number of param- eters that grows linearly with the length T of the sequence (or the maximum dependency length < T), and it is expensive to train. However, using the autoencoder, these two operations are combined within single matrix multiplicatiin.
Notice that in general W will not be decomposed with this factorization. However, any model of this form can be potentialy learned by gradient descent (given suitable loss function). Therefore it is in principle possible for an autoencoder- based model to reconstruct the entire sequence and process each element separately.
The unrolled model shows the expressiveness of the autoencoder. By comparison, an orthogonal parameterization contains the activations of the entire sequence but it is not possible, in principle, to separate them and reconstruct the original input. Furthermore, since the encoding is not explicitly optimized, memorization with an orthogonal parameterization might become inefficient and be prone to require larger hidden state size.
EXPERIMENTS
In this section, we evaluate the performance the orthogonal LMN and the proposed initialization scheme synthetic tasks and real-world datasets: the copy task (Arjovsky et al. 2015). digit classification with sequential and permuted MNIST (Le et al. 2015). and framewise classification with TIMIT (Garofolo al., 1993). These datasets standard benchmarks for the assessment of orthogonal recurrent neural networks, and they offer the opportunity to evaluate the proposed approach in different settings. While for permuted MNIST orthogonal models reach state-of-the-ar results, Vorontsov et al.) (017)) and Jose et al. (2018) showed that orthogonality constraints can reduce the performance of the trained model on TIMIT. We also compare the LAES initialization for the RNN and the LMN, showing that the LMN is better approximation in practical scenarios.
Each model trained with Adam Kingma Ba 2014), with a learning rate in {10-3 10-4, 10-53 chosen by selecting the best model on separate validation set. A soft orthogonality constraint A|| W W 1|2 is added to the cost function as in Vorontsov et al. (017)) with À chosen
{0, 10 10 10
10 10-21.
5.1 COPY TASK
The copy task is a synthetic benchmark often used to test the ability of a model to capture long-ten term dependencies. We the same setup in Jose al. 2018). The objective is the memorization
5

---

Under review as a conference paper at ICLR 2020
of sequence of S elements, which must be repeated sequentially after T timesteps. The input is sequence of 2S + T timesteps where the first S elements are randomly generated from a set of K elements, followed by S - 1 blank elements, an output delimiter, and S blank elements. The target sequence contains 5 + 7 blank elements followed by the first S elements of the input sequence. To solve this task, the network must be able to memorize S elements into its hidden state and remember them for I" timesteps. Each model is a single layer recurrent architectures with 100 hidden units and has been trained with 10 000 batches containing 64 samples each.
The copy task can be easily solved with linear orthogonal models. Henaff et al. (016)) showed a handcrafted solution for the problem. However, adding nonlinearity makes the task difficult to solve even for orthogonal models (Henaff et al., 2016). The LMN solves this limitation by separating the nonlinearity used to compute the hidden activations from the linear memory update. As shown in Table [ the LMN can solve the task even using a saturating nonlinearity (tanh in our experiments). In more general terms, the LMN is able to combine nonlinear activations with the memorization properties of orthogonal linear models.
Table shows the results on the copy task for T = 100 and = 500, with S = 10, and K = 8. We compare the results to a memoryless baseline that outputs the blank element for S - T timesteps and random elements for the last elements. The LMN and the linear RNN solve the task perfectly in both cases. The RNN with tanh nonlinearity fails to beat the baseline even with T 100. The LSTM beats the baseline in both settings but does not solve the task optimally.
These results confirm expectations that nonlinearity even for tasks that require learning long-term dependencies.
the LMN architecture with saturating
Table 1: Test accuracy on the Copy task.
T=100
T=500
RNN (lanh)
FAIL
RNN (linear orthogonal)
100
100
LMN
100
100
LSTM
99.7
99.4
5.2 SEQUENTIAL AND PERMUTED MNIST
Sequential and permuted MNIST (Le et al. 2015) are two standard benchmarks to test the ability of recurrent neural networks to learn long-term dependencies. They consist of sequences extracted from the MNIST dataset (LeCun) 1998) by scanning each image one pixel time, in order (sequential MNIST) or with a random fixed permutation (permuted MNIST). We use these datasets to compare the LAES initialization a random orthogonal initialization. A validation set is extracted by sampling 10 000 random images from the training set and separating them from the training data to use them to perform the model selection.
These datasets provide an ideal setting to test a pure memorization approach like the LAES initializa- tion since the output layer can use the final memory state m' to reconstruct the entire sequence of hidden activations.
Table shows the results on the two MNNIST benchmarks. We compare the RNN and LMN with the LAES initialization and random orthogonal initialization with several orthogonal models. We also show the results of the linear RNN used to initialize the models with the LAES initialization. All the models have 128 hidden units and single hidden layer. Each model has been trained for 100 epochs with a batch size of 64.
The best results achieved by the AntisymmetricRNN (Chang et al., 2019). while the LMN with the LAES initialization obtains slightly lower results, but still improves compared to the results the other orthogonal models.

---

Under review as a conference paper at ICLR 2020
Table 2: Test accuracy on sequential MNIST and permuted MNIST.
Sequential MNIST
Permuted MNIST
74
Valid.
Test
Valid.
Test
FC uRNN Wisdom et al.
2016
512
97.5
96.9
94.7
94.1
EURNN Jing et al. 2016
1024
94.0
93.7
KRU et al.
512
96.6
96.4
94.7
94.5
LSTM(Jose el al. 2018)
128
98.1
97.8
91.7
91.3
ASRNN(Chang et
2019
128
98.0
95,8
ASRNN( (Chang et al.
2019)
128
98.8
93.1
Linear RNN
128
87.1
86.6
84.6
84.2
RNN (ortho init.)
128
79.2
77.2
85.0
84.9
RNN (LAES init.)
128
97.3
97.0
92.4
91.8
LMN (ortho init.)
128
92.0
92.6
89.1
88.7
LMN (LAES init.)
128
98.7
98.5
96.3
95.4
The strength the LAES initialization the result of the ability of the LAES to efficiently compress the entire sequence and the high performance that can be obtained on MNIST using linear mode and the linear RNN.
5.3 TIMIT FRAMEWISE CLASSIFICATION
TIMIT is speech recognition corpus Garofolo et al. 1993). In our experiments follow the setup Jose et al. 2018) for the framewise classification problem (Graves & Schmidhuber (005). Differently from MNIST, this dataset is more difficult for orthogonal models than for non-orthogonal models. Vorontsov et al. (2017): Jose et al. (2018) have shown that orthogonality constraints can be detrimental on this dataset. Therefore,
ideal study the performance of the LMN and the LAES initialization in a more challenging setup. All the models have been trained for 50 epochs with batch size of We compare models with single layer and varying number of hidden units to keep the number of parameters approximately constant across the models (about 400k).
Table shows the results of the experiments.
Table Test accuracy on TIMIT, Results for the RNN, LSTM and KRU-LSTM are taken from Jose al. (2018)
7/h
Valid.
Test
RNN et al. 2018
600
65.84
64.53
LSTM al KRU-LSTM al.
2018
300
65.99
64.56
2018
2048
66.54
64.81
LMN-ortho
420
67.4
66.1
LMN-laes
420
65.0
63.4
We found that without any regularization the norm of the memory state m tended to grow indefinitely, which degraded the final performance. We did not find the same problem the MNIST dataset. penalization on the memory state norm Krueger & Memisevic (2015) was sufficient to solve this issue. We selected by cross-validation on this benchmark the best result is achieved by the LMN, the best model does not use the LAES initialization. The best configuration for the LMN an orthogonal initialization but does use soft orthogonality constraints since the best configuration is obtained when the hyperparameter for the soft orthogonality constraints is À 0. This confirms the other results in the literature and shows that the architecture is general enough to model problems that do not require explicit memorization.
{0, 10, 100). It important note that, while
results for linear classifier found http::/yann. www...ccm//xxd///mm..m
7

---

Under review as a conference paper at ICLR 2020
5.4 LAES INITIALIZATION
In Section 3 we remarked that LAES initialization can incur in large errors due to the nonlinearity that affects the hidden state of the autoencoder. To verify if these problems happen in practical scenarios, we trained linear autoencoder with 128 hidden units on sequences from sequential and permuted MNIST and initialized an LMN and an RNN with it. Table shows the results.
Table 4: Comparison between the LAES initialization sequential and permuted MNIST for the RNN and the LMN using the test accuracy. The LMN is a better approximation and therefore reaches the same performance, while the RNN has much lower accuracy.
Sequential MNIST TRAIN
Permuted MNIST
VALID
TRAIN
VALID
linear-RNN
85.5
87.2
82.9
84.6
RNN (after init.)
11.7
11.3
27.3
27.5
LMN (after init.)
85.5
87.2
83.0
84.7
The results show that the error of the RNN initialization is large enough to cause a significant drop in the accuracy, from 85.5 of the linear RNN to 11.7 of the initialized model. Conversely, the initialized LMN reaches the same accuracy of the linear RNN. Therefore, it seems that the linear recurrence is the model is necessary to achieve good approximation of the autoencoder. The results on TIMIT show a similar trend.
RELATED WORK
Orthogonal RNNs solve the vanishing gradient problem by parameterizing the recurrent connections with an orthogonal or unitary matrix Arjovsky et al. (2015). Some orthogonal models exploit specific parameterization
factorization of the matrix (Mhammedi al., 2017 Jose al. 2018 Jing et al. 2016) to guarantee the orthogonality. Other approaches constrain the parameters with soft or hard orthogonality constraints (Vorontsov al.) 2017; Zhang et al. 2018 Wisdom
2016: Lezcano-Casado & Martinez-Rubin 2019). Vorontsov et al. (2017) have shown that hard orthogonality constraints can hinder training speed and final performance.
Linear autoencoders for sequences can be trained optimally with closed-form solution (Sperduti)
2013). They have been used to pretrain RNNs (Pasa & Sperduti 2014). The LMN (Bacciu et al. 2019 is recurrent neural network with separate linear connection.
The memorization properties of untrained models are studied the field of echo state echo net- works (Jaeger & Haas 2004), recurrent model with untrained recurrent parameters. Tino et al. (2004) showed that untrained RNNs with a random weight initialization have a Markovian bias, which results in the clustering of input with similar suffixes in the hidden state space. Tiioo & Rodan (2013) and White et al. (2004) study the short-term memory properties of linear and orthogonal memories.
7 CONCLUSION
In this work, we studied the problem of building an autoencoder-based memorization mechanism. This system has the ability to encode and decode the input sequences to compute the desired target. Using results for the linear autoencoder for sequences, we showed how to initialize a recurrent neural network by approximating a linear autoencoder the input sequences. The architecture exploits a linear recurrence to obtain better approximation of the autoencoder. The results show that an autoencoder- based initialization can be effective for learning memorization tasks. the future, we plan to extend this work by studying the effect of the autoencoder during training, possibly by enforcing the encoding of the input sequence even during the learning phase. Another possible avenue of research is the study of better optimization algorithms for the parameters of the linear component, where the linearity could exploited to speed up the training process through dedicated learning algorithms.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Martin Arjovsky, Amar Shah, and Yoshua Bengio. Unitary Evolution Recurrent Neural Networks. 11 2015. (tRt:ttp://axxiii org/ abs 1511. 06464
Davide Bacciu, Antomio Carta, and Alessandro Sperduti. Linear Memory Networks. In ICANN, 2019.
URL http: http::/aaxxv.../g// abs /111l/1111.... 03356
Bo Chang, Minmin Chen, Eldad Haber, and Ed H. Chi. AntisymmetticRNN:: A Dynamical System View on Recurrent Neural Networks. 2 2019.
wttp://wwww...ar/////90000000000 09689
Fred Cummins, Felix A. Gers, and Jurgen Schmidhuber. Learning to Forget: Continual Prediction with LSTM. Neural Computation, 2(June 2016):850-855, 2000. ISSN 0899-7667. doi: 10.1197/
jam77.M2577.
Jeffrey L Elman. Finding structure in time. Cognitive science, 14(2):179 -21, 1990.
John Garofolo, William M Fisher, Jonathan Fiscus, David Pallett, and Nancy L Dahlgren. DARPA TIMIT: acoustic-phonetic continuous speech corpus CD-ROM, NIST speech disc 1-1.1. 1993.
Alex Graves and Jürgen Schmidhuber. Framewise phoneme classification with bidirectional LSTM and other neural network architectures. Neural Networks, 18(5-6):602-610) 2005.
Alex Graves, Greg Wayne, and Ivo Danihelka. Neural Turing Machines. CoRR, ab/114105::111 2014. ISSN 2041-1723. doi: 10.3389/neuro.12.000
URL htt.httt://arxxw.orrtt
1410. 5401
Mikael Henaff, Arthur Szlam, and Yann Lecun. Recurrent orthogonal networks and long-memory tasks. 33rd International Conference on Machine Learning, ICMI. 2016, volume 5, pp. 2978- 2986, 2016. ISBN 9781510829008
06662
Sepp: Hochreiter and Jurgen; Schmidhuber. Long Short-Term Memory. Neural Computation, 9(8): 1-32, 1997. ISSN 0435-4052. doi: 10.1144/GL.. 199.MMEE...999999909 1999.018.01.02
Herbert Jaeger and Harald Haas. Harnessing nonlinearity: Predicting chaotic systems and saving energy in wireless communication. science, 304(5667):78-88) 2004.
Li Jing, Yichen Shen, Tena Dubček, John Peurifoy, Scott Skirlo. Yann LeCun, Max Tegmark, and Marin Soljačić. Tunable Efficient Unitary Neural Networks (EUNN) and their application RNNs. In ICML, pp. 1733-1741, 12 2016. URL
abs /taa////11... 05231.
Cijo Jose, Moustpaha Cisse, and Francois Fleuret. Kronecker Recurrent Units. In ICML, pp. 2018. URL L.L 10142
Bhargav Kanuparthi, Devansh Arpit, Giancarlo Kerg, Nan Rosemary Ke, Ioannis Mitliagkas, and Yoshua Bengio. h- DETACCH: MODIFYING THE LSTM GRADIENT TOWARDS BETTER OP.
httttt:s:///peeeeeviiii TIMIZATION, pp. 1-21, 2018. CtLhttps::/ hwtp///eeeeeeeeeeeeee/eeeeeeeeee net net./ppdFPlo--1177599a
Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. arXiv preprint
1412.66412.9988 pp. 1-15, 2014. ISSN 09252312, doi: http://doi.acm.oog wttp://o...................l..ba ezwroxy.lib.uf..duee 1145/1830483. 1835003. URL http:/tttttpaiiiee
abs 1100..1442... 6980
David Krueger and Roland Memisevie, Regularizing RNNs by Stabilizing Activations. 11 2015.
URL http:://x/a/ wwwprxivv..ogg..eooooooo abs/ 08400
Quoc V Le, Navdeep Jaitly, and Geoffrey E. Hinton. A simple way to initialize recurrent networks of rectified linear units. arXiv preprint arXiv: 1504:0094494 2015.
Yann LeCun. The MNIST database of handwritten digits. htpp://aann. lecun. com/exdd/mnist// 1998.
Mario Lezcano-Casade and David Martínez- Rubine Cheap Orthogonal Constraints in Neural Networks: A Simple Parametrization the Orthogonal and Unitary Group. 2019. URL
http://arxiis http://wwwww...oa/////////11111

---

Under review as a conference paper at ICLR 2020
Zakaria Mhammedi, Andrew Hellicar, Ashfaqur Rahman, and James Bailey. Efficient Orthogonal Parametrisation of Recurrent Neural Networks Using Houscholder Reflections. In JCML, pp. 2401-2409, 12 2017. BRLlhttp://axxt www.aaiwww.w..aaaaaoooooooooeete abs 1612. 00188
Luca Pasa and Alessandro Sperduti. Pre-training of Recurrent Neural Networks via
Linear Autoencoders. Advances in Neural Information Processing Systems 27, pp.
3572-3580, 2014. ISSN 10495258. URL http::/ wwwpppppprr...s.cccooo..p..... cc/paperrrcppp
pdf
Alessandro Sperduti. Linear autoencoder networks for structured data. In International Workshop on Neural-Symbolic Learning and Reasoning, 2013.
Alessandro Sperduti. Equivalence results between feedforward and recurrent neural networks for sequences. IJCAI International Joint Conference on Artificial Intelligence, 2015-Janua(jcai):: 3827-3833, 2015, ISSN 10450823.
Tino, M. Cernansky, and L. Benuskova. Markovian Architectural Bias of Recurrent Neural Networks. IEEE Transactions on Neural Networks, 15(1):6-15, 2004. ISSN 1045-9227. doi: 00.1109/NNN2203.800833) URL ltttp://ieeexploree www.eeeeee..g............. org//ouuut//mmmmtttt
12635747
P Tiño and A Rodan. Short term memory in input-driven linear dynamical sys- tems. Neurocomputing, 112:58-63 63, 2013. ISSN 09252312. doi: 10.1016//.meucom.
2012.12.041. URL http:/ /wwp:://wwww.e scopus ottp://wwop...o..o/aooooooooooo.
0 [aw.apaeeaaaaaannnnooooeeeeeeee.
Eugene Vorontsov, Chiheb Trabelsi, Samuel Kadoury, and Chris Pal. On orthogonality and learning recurrent networks with long term dependencies. In ICML, pp.
2017. URL
http://arxix: org/orgggoo......
wtrggabs////eeo. abs 702 00071
Olivia L. White, Daniel D. Lee, and Haim Sompolinsky. Short-term memory in orthogonal neural networks. Physical Review Letters, 92(14):0-3, 2004. ISSN 00319007. doi: 10.1 10.11000 10.Phhyyeese....t. 92.148102.
Scott Wisdom, Thomas Powers, John R. Hershey, Jonathan Le Roux, and Les Atlas. Full-Capacity Unitary Recurrent Neural Networks. In NIPS, pp. 4880-4888, 10 2016. URL http://axxiv org/abs/1611 /lll:. 00035
Jiong Zhang, Qi Lei, and Inderjit Dhillon. Stabilizing Gradients for Deep Neural Networks via Effi- cient SVD Parameterization. In ICML, pp. 5801-5809, 7 2018. URL http:t/tppppppcedi.ng
mlr wwtpppssss/vssaggil.1p html
10


Reviews:
[{"review_id": "H1llbY2sdH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThe paper proposes an autoencoder-based initialization for RNNs with linear memory. The proposed initialization is aimed at helping to maintain longer-term memory and instability during training such as  exploding gradients (due to linearity).\n\nPros:\n\n1. The paper is well written, the motivation and methods are clearly described.\n\nCons.\n\n1. The authors claimed the proposed method could help with exploding gradient  in training the linear memories. It would be helpful to include some experiments indicating that this was the case (for the baseline) and that this method does indeed help with this problem.\n\n2. The experiments on the copy task only showed results for length upto 500, which almost all baseline models are able to solve. I am not too sure how the proposed initialization helps in this case.\n\n3. TIMNIT is a relatively small speech recognition dataset. The task/ dataset does not require long-term memorization. It is nice to see that the initialization helps in this case. However, it is still a little how this experiment corresponds to the messsage that the authors are attempting to deliver at the end of the introduction.\n\n4. In general, it seems that the experiments could be more carefully designed to reflect the contributions of the proposed method. Some suggestions for future edits are, more analysis on gradients, maybe more experiments on the stability of training such as gradients could help.\n\nMinor:\n\n1. There are some confusions, on P2 \"we can construct a simple linear recurrent model which uses the autoencoder to encode the input sequences within a single vector\", I think the authors meant encode the input sequences into a sequence of vectors? Equation 1 and 2 suggest that there is a vector m^t per timestep (as oppose to having 1 for the entire sequence).\n\n2. Although the copy task was used in ((Arjovsky et al., 2015), I believe the original task was proposed in the following paper and hence this citation should properly be the correct one to cite here,\n\nHochreiter, Sepp and Schmidhuber, Jürgen. Long short-term memory. Neural computation, 9(8):\n1735–1780, 1997.\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "BJxS8t0k9r", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an initialization scheme for the recently introduced linear memory network (LMN) (Bacciu et al., 2019) and the authors claim that this initialization scheme can help improving the model performance of long-term sequential learning problems.\n\nMy concerns lie with the novelty of the proposed model and the insufficiency of the experiments. First, the LMN seems to be a simpler version of LSTM and it has no significant advantages compared with other recurrent structures introduced in the past several years.\nSecond, the autoencoder-based init scheme (Pasa&Sperduti, 2014) is not new while the only technical contribution of this paper is a minor change of this scheme so that it works for the LMN. In my opinion, combining these two (LMN and init scheme) can hardly be considered as a solid novelty contribution.\nFor the experiment part, the first two tasks are a bit toyish in 2019 and I have not seen any significant improvement gained from the proposed model. Even for the TIMIT dataset, the results are a bit far from state-of-the-art which makes the paper's claim less convincing.\n\nOverall I think the novelty contribution is marginal and I suggest the authors to test their models on larger-scale real problems.\n\nThe writing is clear and easy to follow.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "B1gi6tditH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper proposes a new initialization method for recurrent neural networks. They first obtain the weight from a linear optimal autoencoder. And then they use the weight to initialize the Lieanr Memory Networks(LMN). Basically, this paper is a combination of [1] and [2].\n\nStrength:\nThe method of initializing LMN using a linear RNN is natural and simple. (section 3.2)\nThe proposed initialization outperforms the baselines on the MNIST dataset.\n\nWeakness:\nWhat do you mean by the \"optimal autoencoder\"?\nThe performance on TIMIT is worse than the baseline methods.\nThe scale of the experiments is too small. Do you have any experiment results on any large dataset? e.g. Penn Treebank.\n \n\nReference:\n[1] Pre-training of Recurrent Neural Networks via Linear Autoencoders\n[2] Linear Memory Networks", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper explores an initialization scheme for the recently introduced linear memory network (LMN) (Bacciu et al., 2019) that is better than random initialization and the approach is tested on various MNIST and TIMIT data sets with positive results.

Reviewer 3 raised concerns about the breadth of experiments and novelty. Reviewer 2 recognized that the model performs well on its MNIST baselines and had concerns about applicability to larger settings. Reviewer 1 acknowledges a very well written paper, but again raises concerns about the thoroughness of the experiments. The authors responded to all three reviewers, responding that the tasks were chosen to match existing work and that the approach is complementary to LSTMs to solve different tasks. Overall the reviewers did not re-adjust their ratings.

There remains questions on scalability and generality, which makes the paper not yet ready for acceptance. We hope that the reviews support the authors further research.
Decision: Reject