Venue: ICLR 2018 conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This paper proposed a new parametrization scheme for weight matrices in neural network based on the Householder  reflectors to solve the gradient vanishing and exploding problems in training. The proposed method improved two previous papers:
1) stronger expressive power than Mahammedi et al. (2017),
2) faster gradient update than Vorontsov et al. (2017).
The proposed parametrization scheme is natrual from numerical linear algebra point of view and authors did a good job in Section 3 in explaining the corresponding expressive power. The experimental results also look promising. 

It would be nice if the authors can analyze the spectral properties of the saddle points in linear RNN (nonlinear is better but it's too difficult I believe). If the authors can show the strict saddle properties then as a corollary, (stochastic) gradient descent finds a global minimum. 

Overall this is a strong paper and I recommend to accept.
review_title: Review

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper introduces SVD parameterization and uses it mostly for controlling the spectral norm of the RNN. 

My concerns with the paper include: 

a) the paper says that the same method works for convolutional neural networks but I couldn't find anything about convolution. 

b) the theoretical analysis might be misleading --- clearly section 6.2 shouldn't have title ALL CRITICAL POINTS ARE GLOBAL MINIMUM because 0 is a critical point but it's not a global minimum. Theorem 5 should be phrased as 

all critical points of the population risk that is non-singular are global minima.

c) the paper should run some experiments on language applications where RNN is widely used

d) I might be wrong on this point, but it seems that the GPU utilization of the method would be very poor so that it's kind of impossible to scale to large datasets? 

review_title: review

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper suggests a reparametrization of the transition matrix. The proposed reparametrization which is based on Singular Value Decomposition can be used for both recurrent and feedforward networks.

The paper is well-written and authors explain related work adequately. The paper is a follow up on Unitary RNNs which suggest a reparametrization that forces the transition matrix to be unitary. The problem of vanishing and exploding gradient in deep network is very challenging and any work that shed lights on this problem can have a significant impact. 

I have two comments on the experiment section:

- Choice of experiments. Authors have chosen UCR datasets and MNIST for the experiments while other experiments are more common. For example, the adding problem, the copying problem and the permuted MNIST problem and language modeling are the common experiments in the context of RNNs. For feedforward settings, classification on CIFAR10 and CIFAR100 is often reported.

- Stopping condition. The plots suggest that the optimization has stopped earlier for some models. Is this because of some stopping condition or because of gradient explosion? Is there a way to avoid this?

- Quality of figures. Figures are very hard to read because of small font. Also, the captions need to describe more details about the figures.
review_title: SVD reparametrization of the transition matrix

Meta-review: Pros:
+ Clearly written paper.
+ Good theoretical analysis of the expressivity of the proposed model.
+ Efficient model update is appealing.
+ Reviewers appreciated the addition of results on the copy and adding tasks in Appendix C.

Cons:
- Evaluation was on less-standard RNN tasks.  A language modeling task should have been included in the empirical evaluation because language modeling is such an important application of RNNs.

This paper is close to the decision boundary, but the reviewers strongly felt that demonstration of the method on a language modeling task was necessary for acceptance.

Decision: Reject