Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In the paper, the authors study stable architectures for RNNs. On the theoretical side, the authors present a series of conditions such that a weight matrix of an RNN is contractive. On the modeling side, the authors propose RNN architectures that have contractive weight matrices. The proposed methods are evaluated on benchmark datasets including sequential MNIST, permuted MNIST, and sequential CIFAR-10.

main_review: The theoretical results seem interesting, although not very surprising. However, I think there is disconnection between the theoretical results and the proposed model: not all theorems are relevant to the proposed model.

For the proposed model (Section 3.1), I might have missed something, but I fail to fully understand the model; the presentation could be improved to help the readers. For example, the mentioning of "subnetworks" at the beginning of Section 3.1 is not defined/explained. It's unclear to me how the subnetworks are combined. I can only infer from the "recursive construction" in the title and Figure 2 that the resulting weight matrix is a block matrix. 

Furthermore, the idea of parametrizing orthogonal weight matrices by exponentiating skew-symmetric matrices is not novel and has been explored in expRNN [1].

The writing of the introduction section could also be improved. Instead of discussing AlphaGo and modules in evolution, the reader might benefit from a more thorough literature review of the RNN trainability and long-term dependence.

[1] Lezcano-Casado, Mario, and David Martınez-Rubio. "Cheap orthogonal constraints in neural networks: A simple parametrization of the orthogonal and unitary group." International Conference on Machine Learning. PMLR, 2019.
summary_of_the_review: The theoretical results in the paper seem interesting. However, the presentation of the proposed model is not clear, and the model itself does not seem novel. Overall, I think the paper needs improvement to meet the acceptance threshold.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors studied contraction properties of continuous-time recurrent neural networks. They further showed that a network of provably stable RNNs (net of nets) can be trained to reach competitive performance on several benchmarks, including sequential CIFAR10, even when only connections between modules are trained.
main_review: Strength
How to assemble a network of RNNs is an interesting problem. The theorems on contraction properties are helpful to people thinking about provably stable RNNs.


Weaknesses
(1)	Section 3.2.1 is pretty dry to read. Reporting results from many individual AxB networks seem unnecessarily.

(2)	The authors showed performance comparison with other types of networks in Table 1. I think it would be quite informative to show performance of networks where everything is kept the same, except that the RNNs are no longer provably stable. 

(3)	It would also be good to know what happens if all connection weights are trained, not just the connections between modules. Does the performance actually decrease despite having more parameters?

(4)	The provably stable part is kind of separated from the training modular network part. How closely are they related? Is having stable RNN modules particularly important for sparsely connected modular networks?


summary_of_the_review: Overall, this is an interesting paper that takes a less common approach to RNNs: provable stability and net-of-nets. The results are at places more difficult to read, but overall it is clear.

I want to add that I cannot evaluate whether the mathematical derivations are correct.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper is primarily a theoretical contribution to the construction of assemblies of recurrent neural networks. We know that combinations of learned modular components can be powerful and far more tractable than learning bespoke models from scratch, particularly in applied domains (e.g. AlphaGo). Yet so far, we have no theoretical guarantees that these combinations will actually remain stable. This paper develops the theory behind provably-stable combinations of RNNs using weight constraints and feedback mechanisms. Then, using fixed RNNs generated according to these constraints (leaving the connections between them as antisymmetric learnable parameters), the authors show that their sparse combination network is able to achieve SOTA performance on sequential image classification benchmarks with far fewer learned parameters and the previous stability guarantee.
main_review: Strengths:
- I thought that the empirical results were rather convincing for what is primarily a theoretical contribution. The authors first thoroughly investigate various permutations of their modular sparse combination network framework (# RNNs vs size of each using absolute value weight constraints) and do another investigation of their alternative SVD weight constraint network (which doesn’t perform as well or train as quickly). Most importantly, they then show that they can best SOTA algorithms on some of the common (albeit easier) benchmarks in the field, even under (and perhaps because of) these constraints. 
- The theoretical contribution is quite powerful. There has been a lot of recent work in networks with many individual recurrent components, such as the aforementioned AlphaGo or the more general recurrent independent mechanisms (RIMs) framework, but for the most part, they rely on intuitive explanations and empirical results over theoretical guarantees. Clearly specialized RNN modules can be quite powerful, but RNNs are notoriously unstable and difficult to learn, and learning such models end-to-end is tricky. If we can apply these constraint conditions and still achieve good performance (which seems like it could be realistic, particularly in the absolute value constraint case), then we can develop sets of useful modules and mix-and-match to the task in question. This paper doesn’t answer all of the intermediate questions, but the stability analysis is a key step.
- The proofs in the appendix are well-done and easy-to-follow, given a sufficient math background.

Weaknesses:
- This paper is very dense and difficult to follow. It took me a few reads to really understand the value of network stability and how it’s achieved in this case. The appendix is a mandatory read as are some of the references. None of the use cases are particularly intuitive. I think I would have liked to see a graphical representation of the sparse combo network (rather than the weight matrices in Figure 2), some pseudocode for the algorithms (tossed in the Appendix), and maybe an example case of an unstable network assembly diverging. I also feel like my familiarity with AlphaGo and other methods gave me more of an insight into how this would help in practice than the actual paper did.
- As much as I liked the empirical results that were provided, they’re all of a kind: sequential image prediction. I would have liked to see at least one application in a different domain (NLP, RL, continuous control, etc).

summary_of_the_review: Overall, I would accept this paper. Although it was difficult to follow and required a lot of consultation with the literature, I do ultimately think that this is a direction that DL algorithms are going in and that the theoretical and practical results from this work could be quite powerful. To make the paper better, I would like to see some results in a different domain and more effort towards improving the readability. Too often, valuable theoretical works go underutilized because they’re difficult to understand or don’t seem relevant to the empiricists and engineers who could build on them.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The submission proposes new theorems showing the stability of a class of RNNs. Further by combining these RNNs into hierarchical and feedback superstructures, the submission achieve SOTA performance on a number of tasks.
main_review: ## Theoretical results

The theorems 1-5 constitute an evolutionary step in the understanding the conditions of stability. The authors also show a counterexample for the common belief that linear contraction leads a sufficient condition for non-linear stability. These results are then (partly) used to construct provably stable RNN combinations.

## Experimental claims

The experimental section, while interesting, seems to lack a main takeaway. Also some very dubious choice of reporting in the table.

* The authors claim that for the result shown on the Tab. 1, they run the Perm-MNIST trial 4 times and the results fall between 96.65 and 96.94. I was frankly shocked to find that they choose to only report 96.94 on Tab.1! Running the same experiment multiple times and only reporting the best case scenario is not good practice and leads to misunderstanding at best. I would recommend that the authors report mean + variance. (If one were to only report the best case, one could get much better performance than is achievable on average by running the experiment many many times.)

* Some claims are backed up by only a single data point. For example, the claim that increased modularity benefits performance to some point is only backed up by the fact that 44x8 performs better than 22x16 in Sec. 3.2.1. To draw a significant conclusion and demonstrate a trend, the authors can perhaps look at 50x7 and 39x9. 

* In general, more data points and error bars would help convince the reader that the conclusions are real and not flukes. 

* The results of 'performance vs network size' and 'performance vs modularity' for Sec. 3.1 and 3.2 are opposite each other. In 3.1 increased size makes the network better monotonically but in 3.2 it is inverted U shape. Similarly in 3.2 modularity makes the performance better monotonically, but it is inverted U in 3.1. What is the conclusion to be drawn here? 

* In general, the experiment section is a little hard to read and can use a summary of the main conclusions and clearly demarcated paragraphs and sections of the experiment that demonstrate each point.



### Clarity and other minor points

* The paper is at times very clear and at times very confusing. For example, the discussion of stability and contraction are clear but then in the paragraph at the top of page 3, the authors use the symbol g for two different thing in the same paragraph. 
* Theorem 7 should be slightly reworded so that it is clear that the first inequality is a condition and not a statement (this is rather obvious in hindsight but for a new reader it is very confusing).
* For denoting multiplication, I would suggest using $\times$ instead of x (e.g. 22x16 etc.)
* Why is there a section 3.2.1 instead of just 3.2?
* I would suggest an overall re-read of the paper to maximize readability.



summary_of_the_review: The paper constitutes an evolutionary step in understanding and designing stable RNNs. The theoretical results are novel and noteworthy. Unfortunately the experimental results lack a clear conclusion and at times do not follow best practices (i.e. reporting only the best run out of many).
review_title: null

Meta-review: In the context of recurrent neural networks, the motivation of the paper is to explore the "space" between fully trained models and almost not trained models, e.g. echo state networks, using a formal approach. In fact, a modular approach has proven to be very successful in many practical applications, and in addition brain seems to adopt this strategy as well. The addressed theoretical issue is stability of the network (i.e., the network implements a contraction map.) Specifically, it is assumed that a network is composed of a set of subnetworks that meet by construction some stability condition, and the problem is to design a mixing weight matrix, interconnecting the latent spaces of the subnetworks, able to give stability guarantees during and after training. Some novel stability conditions are proposed as well as two different approaches to design a successful mixing weight matrix. The original submitted paper was not easy to read, and after revision major problems with presentation have been resolved, although the current version looks more like an ordered collection of results/statements than a smooth and integrated flow of discourse. The revision has also addressed some concerns by reviewers on the role of size and sparsity of the modules, as well as the sensitivity of the stabilization condition on the mixing weight matrix has been experimentally assessed, obtaining interesting results. Overall the paper reports interesting results, however the novelty of the contribution seems to be a bit weak, e.g. stability conditions on recurrent networks (although different from the reported ones) were already presented in literature. Also the idea of exploiting, in one of the proposed models,  the fact that the matrix exponential of a skew-symmetric matrix is orthogonal to maintain the convergence condition during training, is not novel. Moreover, the experimental assessment does not provide a direct comparison, under the same architectural/learning setting, of the novel stability results versus the ones already presented in literature. Empirical results are obtained on simple tasks (using datasets with sequences of identical length), and relatively small networks, which limits a bit the scope of the assessment, as well as it is not clear if the observed improvements (where obtained) are statistically significant (especially when compared with results obtained by networks with the same order of parameters.) The quality of the assessment would increase significantly by considering datasets with sequences of different lengths, and involving more challenging tasks that do require larger networks.
Decision: Reject