Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 1: Reject
review_content: This paper proposes a parameter space, called path space for RNNs with ReLU activation. For the construction of the path space, this paper utilises a reduction graph approach to minimise the difficulty brought by the parameter-sharing scheme in RNNs. Furthermore, the authors propose a Skeleton method for the efficient identification of basis path for RNNs in reduction graph.
The G-SGD approach used in this work is not new and has been initially applied by Meng et al., 2018.
Meng et al., 2018 have primarily introduced G-space for neural networks and designed SGD in G-space (G-SGD) to optimise the value vector of the basis paths of neural networks.  The main difference between this manuscript and previous works (Meng et al., 2018 and Neyshabur et al., 2016.) is that the authors apply G-SGD to RNNs instead of MLPs/CNNs.
The reviewer is not sure if this marginal difference is sufficient to get the paper accepted in ICLR. The structure, the methodology, and the content of this paper is highly similar to the work published by Meng et al., 2018.  
This paper claims to obtain significantly more effective RNN models than using optimization methods in the weight space without providing any statistically significant measures.
The authors could have added a visual performance comparison (e.g., training loss/epoch curve and test error/epoch curve) between the RNN G-SGD and other state-of-the-art approaches.
The theorems and their proofs are fine (similar to Meng et al. (2018)).
 
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: Motivation: The authors motivate their work with the observation that neural networks with ReLU activations are positively scale invariant. Recent works have proposed parameter space called path space to leverage this insight for feedforward as well as CNNs, but this is not really the case for Recurrent Neural Networks (due to the recurrent structure and the parameter sharing). 

Contribution:  The authors  construct path space for RNN to employ optimization algorithms in path space. The intuition is to leverage the reduction graph of RNN to removes the influence of time-steps. Reduction graph only contains information about the weight connectivity patterns and not about the time steps. Hence,  the number of paths in reduction graph is fixed (i.e independent of number of time steps.

Clarity: The clarity of the paper can be improved a bit. It might be useful to give a "top-down" introduction somewhat at the end of introduction. 

Experimental Results: In order to validate the proposed method, the authors conduct experiment on sequential MNIST tasks, as well as language modelling task. The results are a bit weak in general, and improvements are very minor over the vanilla RNN baseline. It might be interesting to compare to other baselines as in PathSGD paper.




review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: This paper aims to improve the training of recurrent neural networks (RNN) with ReLU activations by optimizing in the path space instead of the weight space. Studies on multi-layered perceptrons (MLP) and convolutional neural networks (CNN) have shown that these architectures are positively scale invariant (PSI), however traditional SGD optimizes in the weight space that does not have this property. It has been shown in prior work that this mismatch can slow down the optimization process for SGD and it has been demonstrated that optimizing in the so called path space, which has the PSI property, can be faster and more efficient.  The authors of this paper aim to extend an existing path-space framework (G-SGD) that is used for ReLU networks to facilitate RNNs.  To tackle the challenge posed by the time-dependency of RNNs, they use a static representation, called reduction graph, to define the path space. First, they prove that any path in the original RNN graph can be easily obtained from paths in its reduction graph by simple operations. Then, they define the basis for the path space in the reduction graph, and show that basis paths are sufficient to represent the output of the RNN. They propose a method (Skeleton method) to generate a basis path in the reduction graph and specify an equivalent of the G-SGD algorithm for RNNs.  Through numerical simulations they demonstrate that (1) G-SGD for RNNs has a fairly low additional computational cost compared to SGD and (2) G-SGD for RNNs achieves better test accuracy compared with SGD and an additional path space based approach.

Even though the paper has some original contribution, due to issues with the delivery, clarity and execution of the paper I would lean to reject it at this point. I would be willing to change my decision if the authors made significant changes to the paper in aspects detailed below. The paper also has many grammatical and typographical mistakes that hinders the reader in understanding key ideas.

First,  Section 2 could be much better explained. It seems like the notation introduced for a node and its value are used interchangeably, the edges are not clearly defined and Definition 1 is extremely confusing (what is s’ an arbitrary time step or all time steps between t and t+s, where is the list of weights mentioned in the first sentence and so on).  The definition for a key concept in the paper, the value of the path, is also not clear due to the lack of sufficient definition of the graph itself. 

Second, in Section 3 the definition of the reduction graph and how it is obtained from the directed graph is not clear, which is problematic since it is one of the key ideas in the paper. Clear explanation of a recurrent edge is also missing.

Third, in Section 4 terms such as ‘without recurrent edges parts’  that are used in defining the Skeleton method are very lax and needs to be defined more rigorously. It is hard to understand what the boxed steps are saying. Moreover,  in Algorithm 2 division by a matrix occurs at several points. Does this mean entry-wise division? What does the function BP(.) do? I couldn’t find it defined anywhere.

Lastly, in Section 5 it would be fair to make a comparison with adaptive optimization methods such as Adam or AdaGrad as it is done in [Neyshabur et al.: Path SGD]. 

I would like to provide some additional feedback that do not impact my decision, but could potentially improve the paper. In Section 2: I cannot find an explanation what an ‘unfold directed graph’ means. In eq. (3) x is used without any prior explanation/definition. In Section 3, the message around eq. (4) is not clear, how we pick p2…p5? Moreover, in the definition of basis paths P is used without explaining what it denotes.  Comments on Appendix: in Table 4 it is confusing to highlight the results corresponding to the papers method, because it suggests that these are the best results across the table, however there is a lower value in the second row. In B.1. taking derivative w.r.t p doesn’t seem to make sense, it should probably be v_p (this holds for the whole derivation). In Table 6, it would be a clearer to present relative (%) results than absolute time cost.  Comment on the proofs: they are extremely verbose and could potentially be explained with more math and less words. It is difficult to follow what is going on. There is also some notation not introduced before in eq. (28). In general, I would recommend only numbering equations that are referred to in the text. 
review_title: Official Blind Review #3

Meta-review: The scores of the reviewers are just far to low to warrant an acceptance recommendation from the AC.
Decision: Reject