Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper proposes an initialization scheme for the recently introduced linear memory network (LMN) (Bacciu et al., 2019) and the authors claim that this initialization scheme can help improving the model performance of long-term sequential learning problems.

My concerns lie with the novelty of the proposed model and the insufficiency of the experiments. First, the LMN seems to be a simpler version of LSTM and it has no significant advantages compared with other recurrent structures introduced in the past several years.
Second, the autoencoder-based init scheme (Pasa&Sperduti, 2014) is not new while the only technical contribution of this paper is a minor change of this scheme so that it works for the LMN. In my opinion, combining these two (LMN and init scheme) can hardly be considered as a solid novelty contribution.
For the experiment part, the first two tasks are a bit toyish in 2019 and I have not seen any significant improvement gained from the proposed model. Even for the TIMIT dataset, the results are a bit far from state-of-the-art which makes the paper's claim less convincing.

Overall I think the novelty contribution is marginal and I suggest the authors to test their models on larger-scale real problems.

The writing is clear and easy to follow.
review_title: Official Blind Review #3

final_score: rating: 1: Reject
review_content: Summary:
This paper proposes a new initialization method for recurrent neural networks. They first obtain the weight from a linear optimal autoencoder. And then they use the weight to initialize the Lieanr Memory Networks(LMN). Basically, this paper is a combination of [1] and [2].

Strength:
The method of initializing LMN using a linear RNN is natural and simple. (section 3.2)
The proposed initialization outperforms the baselines on the MNIST dataset.

Weakness:
What do you mean by the "optimal autoencoder"?
The performance on TIMIT is worse than the baseline methods.
The scale of the experiments is too small. Do you have any experiment results on any large dataset? e.g. Penn Treebank.
 

Reference:
[1] Pre-training of Recurrent Neural Networks via Linear Autoencoders
[2] Linear Memory Networks
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: Summary:

The paper proposes an autoencoder-based initialization for RNNs with linear memory. The proposed initialization is aimed at helping to maintain longer-term memory and instability during training such as  exploding gradients (due to linearity).

Pros:

1. The paper is well written, the motivation and methods are clearly described.

Cons.

1. The authors claimed the proposed method could help with exploding gradient  in training the linear memories. It would be helpful to include some experiments indicating that this was the case (for the baseline) and that this method does indeed help with this problem.

2. The experiments on the copy task only showed results for length upto 500, which almost all baseline models are able to solve. I am not too sure how the proposed initialization helps in this case.

3. TIMNIT is a relatively small speech recognition dataset. The task/ dataset does not require long-term memorization. It is nice to see that the initialization helps in this case. However, it is still a little how this experiment corresponds to the messsage that the authors are attempting to deliver at the end of the introduction.

4. In general, it seems that the experiments could be more carefully designed to reflect the contributions of the proposed method. Some suggestions for future edits are, more analysis on gradients, maybe more experiments on the stability of training such as gradients could help.

Minor:

1. There are some confusions, on P2 "we can construct a simple linear recurrent model which uses the autoencoder to encode the input sequences within a single vector", I think the authors meant encode the input sequences into a sequence of vectors? Equation 1 and 2 suggest that there is a vector m^t per timestep (as oppose to having 1 for the entire sequence).

2. Although the copy task was used in ((Arjovsky et al., 2015), I believe the original task was proposed in the following paper and hence this citation should properly be the correct one to cite here,

Hochreiter, Sepp and Schmidhuber, Jürgen. Long short-term memory. Neural computation, 9(8):
1735–1780, 1997.


review_title: Official Blind Review #1

Meta-review: The paper explores an initialization scheme for the recently introduced linear memory network (LMN) (Bacciu et al., 2019) that is better than random initialization and the approach is tested on various MNIST and TIMIT data sets with positive results.

Reviewer 3 raised concerns about the breadth of experiments and novelty. Reviewer 2 recognized that the model performs well on its MNIST baselines and had concerns about applicability to larger settings. Reviewer 1 acknowledges a very well written paper, but again raises concerns about the thoroughness of the experiments. The authors responded to all three reviewers, responding that the tasks were chosen to match existing work and that the approach is complementary to LSTMs to solve different tasks. Overall the reviewers did not re-adjust their ratings.

There remains questions on scalability and generality, which makes the paper not yet ready for acceptance. We hope that the reviews support the authors further research.
Decision: Reject