Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: To reduce tne number of connections in a neural network needed to compute a Boolean function, the authors use a recurrent neural network instead. Given input x, the RNN must settle to an equilibrium, and the first node of the network is transformed to give the output. The authors use the Cohen-Grossberg theorem to show stability if the connections are symmetric. They obtain requisite connections for the N-bit parity function by trial and error.

strengths_and_weaknesses: The authors motivate the work well suggesting that connectivity is the bottleneck for neuromorphic hardware rather than neuron numbers, and thus suggest recurrent neural networks that reach equilibrium as more efficient to compute static (binary) input-output mappings. They show some tantalizing connections to algebraic geometry, and try to do a piecewise linear analysis. To my inderstanding, these yield only necessary but not sufficient conditions. They instead show Lyapunov stability of the system assuming symmetric connections. The authors point out some very interesting connections that have the potential to lead to interesting results. However, the authors have been unable to develop these further theoretically. The N-bit parity function was found by trial and error and the authors do not provide a constructive algorithm to learn more general binary input-output transforms. Thus the experimental work is also rather limited and does not prescribe how these networks may be constructed or learned. 
questions: Would say using backprop through time to learn the symmetric connections be possible while maintaining the equilibrium guarantees (during learning and at the end)? Or could you provide further insight into learning or constructing such networks. If this or some other constructive algorithm was proposed showing the connections to your theoretical methods, I would be happy to improve my rating.
limitations: The work is still in a rather nascent stage and these issues do not arise yet.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper analyses the dynamics of recurrent neural networks and studies the equilibrium points to produce stable combinatorial output.
The authors propose a physical model amenable for analysis with algebraic geometry, solve for the equilibrium points under the assumption of piecewise-linear non-monotonic output, and perform Lyapunov stability analysis. The experimental phase validates the theoretical result by realizing the parity function.
strengths_and_weaknesses: - *Significance*: This work advances the theoretical analysis of recurrent neural network dynamics, which might significantly contribute to the community and help design more efficient architectures. However, the paper is very concise, and part of the details is difficult to grasp at first read. In the Introduction, the final aim is not sufficiently stressed, and the discussion about the need for more efficient architectures is limited to a paragraph. The experimental phase consists of a simple experiment on standard parity function with few input dimensions. Nevertheless, it is a nice proof of concept to validate the proposed theoretical result.
    
- *Originality*: The paper adequately mentions prior works along the text and explains how the contribution differs from them. However, a separate section on related work might be helpful to compare the approach with prior works easily. 
The analysis of recurrent neural networks is novel, and this work uses appropriate analysis techniques from algebraic geometry and Lyapunov stability.
    
- *Clarity*: The paper is well written but difficult to follow. It assumes a strong knowledge from the reader, presenting only the principal results and citing prior works from the literature. Considering it is a theoretical work, I would have appreciated a section to provide a minimal technical background for the readers. To this aim, the authors could have used all the allowed pages and provided additional content in the appendix.
    
- *Quality*: The submission looks technically solid, and the analysis techniques look appropriate. However, the soundness of the technical claims is beyond my knowledge.


### Minor remarks

- Figure 2 has no labels on the axis. Moreover, the vertical axis shows different lengths (see first and second plots) and scales (see last and second-last plots) for the various quantities visualized.
questions: - The current work assumes piecewise-linear output that fits well combinatorial boolean functions. However, a broader class of functions can be represented or approximated as piecewise-linear (e.g., absolute value function). Is the current work limited to boolean function in its assumptions?
- I would recommend the authors extend the manuscript with more motivations, background notions, running examples, and material to make the current work accessible to a broader audience. In particular, clearly communicating the goal and its applicability is essential. Otherwise, the submission of theoretical work, even if providing novel and strong results, could not be appreciated by the readers.
- Section 2.2 about algebraic geometry is partially unclear. For example, equations (3) and (5) define the properties and assumptions of the problem. It would be nice to extend them with some visualizations of a running example.
limitations: The main limitations of this work are the clarity in communicating the theoretical results and the limited experimental phase.
As an extension, it might be worth validating the architecture scalability results by conducting experiments for varying N.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The idea of this paper is to limit the number of connections in a physical neural network that are required to realize $N$-input Boolean functions, while focusing on the parity functions as a proof-of-concept. In feed-forward networks the Shannon bottleneck demands that there are $O(2^N)$ interconnections between neurons to realize the desired Boolean functions. The authors try to circumvent this bottleneck by using recurrent physical networks that have been shown to be able to realize certain functions with fewer units.

After defining the state-space model of the recurrent physical network, the authors are looking for sufficient conditions of the conductance function that allow for the existence of its equilibrium points in order to prove that a stable output activation can be reached. This is achieved by committing to analyzing a conductance function that is piecewise-linear algebraically. Further, due to the nature of the proposed state-space model, the system becomes a special case of the Cohen-Grossberg model which can be shown to be globally absolute stable using a Lyapunov stability analysis. 

The authors use the result of the stability analysis to simulate a shallow recurrent neural network that solves the parity problem.
strengths_and_weaknesses: The paper shows an interesting and scientifically important direction for future physical neural network designs, which potentially could allow for many more efficient physical neural network implementations in practice. The authors provide a sound theoretical analysis of their proposed system.

The writing is very compact but of high quality. Readability could be improved by providing more background information on physical neural networks, as these are not particular common in the general machine learning community.
In general I would appreciate a more detailed descriptions of their methods and results - the paper is very short in its current form and I see a lot of opportunity to improve clarity without getting close to the page limit.
I would also like to see the authors putting the paper better into context with other publications by citing related work in the field.
questions: Please provide more information on how you think the analysis can be scaled up to more realistic problems.
limitations: The authors consider the work as a proof-of-concept and describe several opportunities for future work. An analysis on potential negative societal impact has been omitted, which is reasonable given the theoretical nature of the paper.
review_title: null

final_score: rating: 2: Strong Reject: For instance, a paper with major technical flaws, and/or poor evaluation, limited impact, poor reproducibility and mostly unaddressed ethical considerations.
review_content: summary: A feedforward logic circuit follows certain scaling constraints (for example, the number of layers or logic switches required to compute some function). However, a cyclic (analogue) network can, in some cases, avoid these limits.

This work introduces an approach to using analogue, cyclic networks to compute the parity (highly non-linear) logic function. They analyze and simulate (to find the parameters) this for a N=3 problem.
strengths_and_weaknesses: Strengths:
- Using recurrent functionality to reduce circuit sizes in an interesting problem (although the practical realizations make be non-trivial at scale).

Weaknesses:
- Paper is very hard to follow and makes a lot of assumptions on prior knowledge (and there is plenty of remaining space to include more background, its only 5 pages right now).

- Its not clear what the main contributions of this work are.

- The scale of this work is trivial, and the solution was found numerically. This approach is much more interesting for much larger scale.
questions: - What are the main contributions of this work?

- Could you explain more clearly what you expect NeurIPS audiences to take away from this paper and which parts of the community are most likely to be interested.
limitations: na
review_title: null

Meta-review: metareview: We had quite a bit of discussion on this paper. I read the paper and agree with some of the discussions that the paper in its current form might not attract interest in the community due to the following reasons:

- While it is highly interdisciplinary and potentially of high impact, the authors did not manage to connect all topics to tell a concise story on the use of equilibrium-point control theory in making a new recurrent neural network model. Here are some final comments from our discussions with the reviewers:
    - While the content looks technically sound, we haven't seen from the authors the expected revision to improve the paper. We were expecting a better introduction to the problem and further discussion to convey the contribution. 
    - We think that it would be in the interest of the authors to extend the paper with clarifications and more background information such that a larger audience will be able to learn something from their work.
    - We believe that this issue is fixable as there is ample space to add additional background/introduction to explain the problem/relevance. However, the authors did not manage to convince the reviewers how they are going to address this during the discussion period.

- The authors could elaborate much more in detail on all topics involved to build a better understanding first and convey their contribution and impact better. 

- Also, there are recent advances in state-space models and their use as expressive representation learning algorithms within the community which are disregarded in the paper. In particular, the stability condition, memorization, the efficiency of computing transition matrices (which are very relevant to this paper), and more properties are discussed the last few years. Here is an example:

[1] Gu et al. Efficiently modeling long sequences with structured state spaces, ICLR 2022 https://arxiv.org/abs/2111.00396

Based on these points, I vote for the rejection of this paper. 
confidence: Certain
Decision: Reject