Deep feedforward functionality by equilibrium-point control in a shallow recurrent network.
Authors: Celestine Preetham Lawrence
Venue: NeurIPS 2022 Submitted (2022)
Keywords: recurrent physical network
combinational logic
equilibrium-point control
piecewise-linear
parity function

Abstract: Recurrent neural network based machine learning systems are typically employed for their sequential functionality in handling time-varying signals, such as for speech processing. However, neurobiologists find recurrent connections in the vision system and debate about equilibrium-point control in the motor system. Thus, we need a deeper understanding of how recurrent dynamics can be exploited to attain combinational stable-input stable-output functionality. Here, we study how a simplified Cohen-Grossberg neural network model can realize combinational multi-input Boolean functionality. We place our problem within the discipline of algebraic geometry, and solve a special case of it using piecewise-linear algebra. We demonstrate a connectance-efficient realization of the parity function as a proof-of-concept. Small-scale systems of this kind  can be easily built, say for hobby robotics, as a network of two-terminal devices of resistors and tunnel diodes. Large-scale systems may be energy-efficiently built as an interconnected network of multi-electrode nanoclusters with non-monotonic transport mechanisms. 

Deep feedforward functionality by equilirrium-point control in a shallow recurrent network
Anonymous Author(s)
Affiliation
Address
email
Abstract
Recurrent neural network based machine learning systems are typically employed for their sequential functionality in handling time-varying signals, such as for speech processing. However, neurobiologists find recurrent connections in the vision system and debate about equilibrium point control in the motor system. Thus, need deeper understanding of how recurrent dynamics can be exploited to attain combinational stable-input stable-output functionality. Here, we study how simplified Johen-Grossbers neural network model can realize combinational multi-input Boolean functionality. We place our problem within the discipline algebraic geometry, and solve special case of it using piecewise-linear algebra. We demonstrate a comnectance-efficien realization of the parity function as a proof-of-concept. Small-scale systems of this kind hobby robotics, as network of two-terminal devices of resistors and tunnel diodes. Large-scale systems may be energy-efficiently built an interconnected network of multi-electrode nanoclusters with non-monotonic transport mechanisms.
2
6
10
11
be easily built, say for
12
13
14
Introduction
18
Shallow recurrent neural networks are being investigated for more context-aware object recognition
17
(5 and brain-like behaviour (33 They can be more compact (by trading space for time) and are naturally robust alternative to deep neural networks (which are easily fooled by input perturbations or transformations [18 29 when the role of recurrent dynamies is not to produce time- varying output but instead to produce transient (hidden) state-dynamics that facilitate deep, robust and transformation-invariie fixed-input fixed-output functionality. To better engineer such dynamics, we shall study equilibrium-pon
19
21
control, which can be defined as the process of steering to target 23 in state-space space by fixing the input signal, instead of driving it by continuously varying input signal. Historically, equilibrium-por -pontt control [14] was first formulated to provide plausible solution to the degrees freedom problem in motor control 33, that is, we mentally represent intermediate destination points rather than continuum of velocity information required to execute movement.
25
27 Here, we shall focus on using equilibrium-poin control to realize multi-input Boolean functionality, in particular the parity function, which is canonical proxy for nonlinear classification. Theoretical results circuit complexity are known already for realizing Boolean functionality out of feedforward neural networks, with weighted-sum thresholded binary-output neurons (35). It has been shown that arbitrary -iinput Boolean functions can be realized depth-3 feedforward networks with fewer neurons (m N/2) instead of the (2 in total required for depth-2). However, with the advent of nanoelectronics, the size of artificial neuron has been downscaled to such an extent
28
29
31
32
34 that it is rather the interconnect wiring that now occupies greater area in chip design. Thus for
fully-connected deep network, the area scales as the number of interconnects m² O22 Such
35
Submitted to 36th Conference on Neural Information Processing Systems (NeurIPS 2022). Do not distribute.

---

()2") scaling law was earlier obtained by Shannon [34 for realizing arbitrary N N-input Boolean functions by an interconnection of input-controlled switches (or equivalently a feedforward network of 2-input Boolean gates). Thus, unless we employ higher-order neurons [16]. we can say that
36
37
38
39 a Shannon bottleneck limits the maximum N-input Boolean logic realizable (nanoscale) feedforward networks. We aim to circumvent this Shannon bottleneck by employing
given area by
40
41
recurrent physical networks. is known that certain combinational logic functions can be realized by fewer logic gates in cyclic network than in acyclic network [3]], and with analog signal processing the improvement factor could be even higher.
43
44 In the following section, we introduce a state-space model formalism to study equilibrium-pon point control, and commit to a physically realizable model, and discuss how general solution for its equilibrium points is a difficult problem in algebraic geometry. Thus, we proceed to idealize the non- monotonic output of the physical system as a piecewise- function and solve for the equilibrium points. Finally, piecewise-quadratic Lyapunov function is obtained for stability analysis and conditions for a unique equilibrium-poii point are provided.
45
48
47
48
49
50 Aftr the theory, in the results section, we provide comnectance-fficien realization of the parity function. The discussion section puts our results into a broader context and offers avenues for further
51
52 research. Our objective here is to work at the intersection of nonlinear dynamical systems, neural networks, unconventional neuromorphic hardware, cyclic Boolean eircuits. piecewise- control systems, and algebraic geometry.
53
54
Theory
55
2.1
State-space model
56
57
For cquilibrium-povt control, in general we have an input vector æ, a state st) for i == 1 N, and an output y obtained from system of equations
s (t) Fi(((t), y lim G(s(t)). to00
(1)
59 In this paper, we commit to physically realizable recurrent network with voltage nodes s; from 60 i 1 N, with a capacitive time-constam 7s, using resistors (of a constant conductance fiy) and
61 tunnel diodes (of voltage- conductance (s)):) as shown in Fig. yielding a state-space
62 model of the form
T = Ii
-ls(si - -.((s) = (i
(2)
jf1
where fúj 2 O. G. is a nonlinear passive function such that G)(s)s 2 0 and lim, 81 (t) is the stable gquilibrium-poing if one exists (note: y(x) can be multi-valued and depend on the basin of attraction that the initial state s()) lies in). Brain-scale systems of this kind may be realized by interconnected network of nanoclusters with non-monotonic transport mechanisms as proposed in 24 Chapter 5]. However, finding suitable network parameters that result in practical functionality 87
64
66
68
remains a challenge. Note that, although not the focus of this work, Eq. (2) can also represent
89
state-space models with noisy rectified-linear units, for which semi-analytical results are known from perspective.
70
a computational neuroscience [12 and a machine learning 33
2.2
Algebraic geometry of the equilibrium points
71
72 A study of the set of equilibrium points of a state sptee-ppce model, So(x) = {s (L, = 0}.
á E S. C So. but also provide 74 necessary (but not sufficient) conditions in the parameters defining the functions FLNN and G, to realize desired equilibrium-pon functionality y(x). For example, to realize Boolean function
73 can not only help in characterising the stable equilibrium-poinn
76 y: {0, 1 (0, 1}. the following property has to be satisfied:
min G(s) < V Sc))
max SE
G(s) 2 Va E N..
(3)

---

Figure 1: Recurrent physical network corresponding to the state-space model (2) where the inputs U1:N are currents, the states $1:N are voltages, the output is a measured current, the linear interac- tions are due to resistors with conductance fay between node i and j. and nonlinear interactions due to tunnel diodes from node i to GND with conductance G; (s:).
The set of equilibrium points of our recurrent physical network model (2 are the roots of the system of nonlinear equations
78
-f SI.N Gs(s;) Z;
(4)
where the linear-interaction matrix JNNN has terms fü = E
80 Solving the multivariate nonlinear equation (4) a difficult problem in algebraic geometry, a discipline of mathematics which classically grew around efforts to understand the roots of multivariate polynomials and later metamorphosed by the study of integer-coefficient piecewise-lined with an abstract language that has even recently been applied explain circuit complexity results of deep feedforward networks 335 30 through the lens of rational piecewise-linear functions [40].
82
functions,
83
84
Algebraic geometry originally dealt with qualitative approach by geometrical arguments [15 in contrast to a quantitative approach by numerical methods. An example of that kind is Harnack's curve theorem [19] which states that for 2-D polynomial curve of degree n, the maximum number of connected components is (n² 3n + 4)/2. Now, with the advent of computer algebra, the roots of multivariate nonlinear equations
86
86
studied by the elimination of variables, using techniques such as resultants 13 38 and Groebner bases 7 for polynomial systems, and as an instance of the cnneemccomplementarriy problem [1] or equivalently as absolute-value equations (26 for
90
piecewise-linear systems [37]. However, computer algebra is not scalable for higher dimensions. Thus there need convey the richness in algebraic geometry using analytical expressions. While it is unlikely that analytical expressions may be obtained for any general form of nonlinearity, we may hope that the set of exactly solvable models can be extended well beyond linear equations,
92
93
94
95
96
hope banking on our successful experience from other areas of mathematics such as integral calculus
97
9 section IX] and iterated mappings 339 page 1098].
98
2.2.1
Piecewise-linear algebra
99
In this paper, we commit to a piecewise-linear analysis by considering
{
9:18 (911 g:3)g.2 9338
0 0 3 s S 9:2
Gx(+)
(5)
9:2 S 8 < 9:2
+ 21
gk3
100 where 311.2.3 > so that G; is a triangular peak function in a limited range of s, thus defining an idealized -eeffive-differentiall behaviour. Shifting the state-space about its inflection points as 102 z 92 and then combining (5) with (4) yields
faNNNN z + JA, 92 9:192
9i S 2, S
I;
fuNN z 9.324 fINN 92 + 9-1922
0 < 2 < 9:2
(11) 913
(6)
which can be simplified to
21 fr.NNN 2 # 22elz1 fuNN 92 + 9:192
(7)
3

---

104 for 9:2 S S 9:2 (iil with Iie = (9:1 9.s) and 9:0 (91 + 9:s)/2.
105 The system in can be expressed the Absolute-valus equation normal form
A= - ||| |
(8)
108 with A.j = fig Lig 9.0. b, (z; + fi,I.N 92 + 911922) 9 and the bounds
92 9291 g3-
(9)
Similarly, can be expressed
T = 9 @ (b- Az + |z]).
(10)
Two sufficient conditions are known for the absolute value equation (8) have unique solution based on the largest singular value min
[26 and the spectral radius 32
miin () > 1, p( A
(11)
1D <.
(12)
108
However, those are not yet sufficient conditions for unique equilibrium- point solution for (2) and I0] because the bounds in (9) were not enforced. Thus, we shall proceed to obtain Lyapunov
109
110
function to guarantee that a stable equilibrium-poin reached.
2.3
Lyapunov stability analysis
111
Equilibriummppoi stability for large complex systems is not guaranteed in general [17 27], and the effective dimensionality of stable-input stable-output responses is richly dependent the parameter 114 space [2]. However, the interaction matrix for our physical system is symmetric, and hence the
112
113
115
system a special case of the Johen-Grossberg model [IO]
8 = (,(())(bb((s))
Σ (1dd(188)1, #=1
(13)
with (s(s)) 1/T,, (s;) = - (s)) Cjj -fg and dz(s;) = sj. Thus, it is known to be globally absolute stable, with a Lyapunov function
Σ l b.(w)d du i
Cij di (s)))d(((s)
V V=
(14)
Σ (Pi(ss) - -48; -
Σ figs183 fa
2 x
(15)
J31
1/ G.( du.
where output power P((s:)
(16)
116
Alternatively, since our system (5) is piecewise-lineat piecewise-quadrati Lyapunov function may be obtained by piecewise-affine system 222 analysis. While this approach is more powerful and
118 holds even for asymmetric interaction matrices, also seems to analytically complex. From another angle, global asymptotic stability [21] Theorem 3] is guaranteed if the Jacobian matrix J satisfies
120
Ja + 11/2 jz JJg
+ Jpil
+ (s,),) > because in our system Jμ Ejjt fiz (s,)))
121
and Jjj- = Jej- Since our network employs non-monotonic functionality, G,(ss) > cannot be
122
guaranteed for all reachable states 8;, and thus the above criteria is unfortunately inapplicable. Hence,
123 we shall proceed with the Cohen-Grosseee Soohen-Groosbergg approach.
The power function
21 simplifies to
Sot u du 91s² 9192 2 + f (g1 + ga)g2 gzu du 92 S & = = 1/2 g (s -
S $ S 92
P(s)
(17)
and using the rectifier function (a:) = max (r, 0) may be expressed conveniently 125
((s) s² 2 /2/222222 gals 92
(18)
when the system is within its operational bounds.

---

3
Results
127
128
Given the Lyapunov stability result of our system, it is computationally efficient to simulate our state-space model and probe for combinational functionality. Here, will simulate for the simplest proof-of-concept for deep functionality in a shallow recurrent - solving a parity problem.
130
Using cascade of 2-input XOR gates, the N-bit parity function can be realized with N/2 + N/4 + 1 N gates and 2N connections. Thus its total cost in area is at least 3N 2 units. A minimally-connede
132
133
network has N input wires, output wire, and N 11 interconnect wires with a total area cost of 2N units, assuming that the area occupied by the remaining components is negligible. Thus for N = 3, while a conventional digital circuit costs 7 units, our recurrent physical network takes just wiring units.
134
136
137
Our simple model has N 3, f12 = f13 f. f23 0. 911 91 913 = 93. 921 = 931 71
923 933 73. 912 = 92 and 722 Y32 = 72. We find from symbolic evaluation that m (A) * 1/pl p A I) in general, and conditions for unique stability were not obtainable (which is not surprising symmetry). Thus, parity functionality was found by trial-and
138
139
140
due to the $2
yielding the
141
parameters (ʃ = 1.751, 91 = 1.876, 922 = 93 = 0.126, 71 0.876, 722 1.6, 73 = 0.751} and
142
simulated using Wolfram Mathematica 13 (code in Appendix). When 21 = 222 Ir = 1, the states
143
were foreed to transition beyond the bounds 5 value. The results are plotted in Fig.
so its range was extended by taking an absolute
4 Discussion
146
Our result should be seen as a theoretical proof-of-concept and as a motivation for continued research in this area. Future work must extend our simulations much higher dimensions serve as a practical demonstration of deep functionality by shallow recurrent networks. Moreover, the theoretical formalism introduced here is not yet fully exploited. We hope find an analytical method to design functionality out of piecewise-linear Cohen-Grrossherg networks.
147
148
149
150
Our style of reasoning circumvent the Shannon bottleneck may also be applied other systems such as networks of coupled oscillators
152
(28). Our non-modular mode of signal processing, offers an alternative to not just circuit designers, but also systems biologists who typically understand chemical reaction networks [6] as composition of modules [20]. While, we have discussed equilibium-point functionality in state-space model driven by an additive input, it is also worth investigating autonomous systems where the input is set as an initial state. An example is realizing unboundedly-finit parity functions using just a radius-4 cellular automaton [4] Finally, we hope that this paper can serve as a call to action for neuromorphic engineers to look at physical reservoir computing 36) from another angle, besides temporal input-output functionality.
153
154
155
156
157
158
159
References
160
161
[1] M. A. Alcorn, Q. Li, Z. Gong, C. Wang. L. Mai, W.-S. Ku, and A. Nguyen. Strike (with) a pose: Neural networks are easily fooled by strange poses of familiar objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4845-4854, 2019.
162
163
164
[2] R. D. Beer. Parameter space structure of continuous time recurrent neural networks. Neural computation, 3802233009,,35,, 2006.
165
186
[3] N. Bernstein. The co-ordination and regulation of movements. The co-ordination and regulation of movements, 1966.
167
168
[4] H. Betel, P. P. de Oliveira, and P. Flocchini. Solving the parity problem in one-dimensional cellular automata. Natural Computing, 12(3):323-337, 2013.
189
[5] E. Bizzi, N. Hogan, F. A. Mussa-Ivaldi, and S. Giszter. Does the nervous system use equilibrium- point control guide single and multiple joint movements? Behavioral and brain sciences, 15(4):603-613 613, 1992.
171
5

---

X2
X3
XOR(xx,X2,x3)
$2
1
0.4
l(y>.15)
1
Figure 2: Numerical simulation of our 3-state network over 200 timesteps.

---

173
[6] D. Bray. Protein molecules computational elements in living cells. Nature, 376(6538)307-
174
312, 1995.
175
[7] B. Buchberger. Ein algorithmus zum auffinden der basiselemente des restklassenringes nach einem nulldimensionalen polynomideal. PhD thesis, Universitat Insbruck, 1965.
177
[8] B. Buchberger. Bruno buchberger's phd thesis 1965: An algorithm for finding the basis elements of the residue class ring of a zero dimensional polynomial ideal. Journal of symbolic computation, 41(3-4):45-511. 2006.
178
179
180
[9] G. S. Carr. Synopsis of elementary results in pure mathematics. 1886.
[10] M. A. Cohen and S. Grossberg. Absolute stability of global pattern formation and parallel memory storage by competitive neural networks. IEEE transactions on systems, man, and cybernetics, (5):815-226 1983.
182
183
[11] R. W. Cottle. Linear complementarity problem, pages 1873-1878. Springer US, Boston, MA, 2009.
185
186
[12] D. Durstewitz. A state space approach for piecewise-linear recurrent neural networks for identifying computational dynamics from neural measurements. PLoS computational biology. 1306):e100554,, 2017.
187
188
[13] I. Z. Emiris. On the complexity of sparse elimination. Journal of Complexity, 12(2):134-166. 1996.
190
181
[14] A. G. Feldman. Functional tuning of the nervous system with control of movement or main-
192
tenance of steady posture-ii. controllable parameters of the muscle. Biofizika, 11:565-578, 1966.
193
[15] W. Fulton. Intersection theory, volume 2. Springer Science & Business Media, 2013.
195
[16] S. Gao, M. Zhou, Y. Wang. Cheng, H. Yachi, and J. Wang. Dendritic neuron model with ef- fective learning algorithms for classification, approximation, and prediction. IEEE transactions on neural networks and learning systems, 30(2):601-614, 2019.
196
197
198
[17] M. R. Gardner and W. R. Ashby. Connectance of large dynamic (cybernetic) systems: critical values for stability. Nature, 228(5773)784-784, 1970.
200
[18] I. J. Goodfellow, J. Shlens, and C. Szegedy. Explaining and harnessing adversarial examples, 2015.
[19] A. Harnack. Ueber die vieltheiligkeit der ebenen algebraischen curven. Mathematische Annalen, 10(2):189-198, 1876.
203
204
[20] L. H. Hartwell, J. J. Hopfield, S. Leibler, and A. W. Murray. From molecular to modular cell biology. Nature, 402(6761) 1999.
205
206
[21] M. W. Hirsch. Convergent activation dynamics in continuous time networks. Neural networks, 2(3):33,,349, 1989.
[22] M. Johansson and A. Rantzer. Computation of piecewise quadratic Iyapunov functions for hybrid systems. In 1997 European Control Conference (ECC), pages 2005-2010. IEEE, 1997.
209
[23] J. Kubilius, M. Schrimpf, K. Kar, R. Rajalingham, H. Hong, N. Majaj, E. Issa, P. Bashivan, Prescott-Roy, K. Schmidt, et al. Brain-like object recognition with high-performing shallow recurrent anns. Advances in neural information processing systems, 32, 2019.
211
[24] C. P. Lawrence. Evolving Networks To Have Intelligence Realized At Nanoscale, PhD thesis, University of Twente, 2018
214
[25] M. Liang and X. Hu. Recurrent convolutional neural network for object recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3367- 3375, 2015.
216
7

---

218
[26] O. Mangasarian and R. Meyer. Absolute value equations. Linear Algebra and Its Applications, 419(2-3):359-377, 2006.
219
220
[27] R. M. May. Will large complex system be stable? Nature, 238(5364):413 1972.
221
[28] S. N. Menon and S. Sinha. "defective" logic: Using spatiotemporal patterns in coupled relaxation oscillator arrays for computation. In 2014 International Conference on Signal Processing and Communications (SPCOM), pages 1-6. IEEE, 2014.
223
[29] S.-M. Moosavi-Dezfooli, A. Fawzi, O. Fawzi, and P. Frossard. Universal adversarial pertur- bations. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1765-1773, 2017.
225
227
[30] M. Raghu, B. Poole, Kleinberg, S. Ganguli, and Sohl-Dickstein. On the expressive power of deep neural networks. In international conference machine learning, pages 2847-2854. PMLR, 2017.
229
[31] M. D. Riedel and J. Bruck. Cyclic boolean circuits. Discrete Applied Mathematics, 160(13- 14):1877-1900, 2012.
231
[32] J. Rohn, V. Hooshyarbakhsh, and R. Farhadsefat. An iterative method for solving absolute value equations and sufficient conditions for unique solvability. Optimization Letters, 8(1):35 2014.
233
235
[33] D. Schmidt, G. Koppe, Z. Monfared, M. Beutelspacher, and D. Durstewitz. Identifying nonlinear dynamical systems with multiple time scales and long-range dependencies. In International Conference on Learning Representations, 2021.
237
[34] C. E. Shannon. The synthesis of two-terminal switching circuits. The Bell System Technical Journal, 28(1):59-98, 1949.
240
[35] K.-Y. Siu, V. P. Roychowdhury, and Kailath. Depth-size tradeoff's for neural computation. 1991.
IEEE Transactions on Computers,
242
[36] G. Tanaka, T. Yamane, J. Héroux, Nakane, N. Kanazawa, S. Takeda, H. Numata, D. Nakano, and A. Hirose. Recent advances in physical reservoir computing: A review. Neural Networks, 115:100-122 2019.
243
244
245
[37] W. M. Van Bokhoven and D. M. Leenaerts, Explicit formulas for the solutions of piecewise linear networks. IEEE Transactions on Circuits and Systems I: Fundamental Theory and Applications, 1609):1110-1117, 1999.
246
247
248
[38] M. Williams. Solving polynomial equations using linear algebra. Johns Hopkins APL Technical Digest, 28(4):354-36) 363, 2010.
250
[39] Wolfram, A new kind of science, volume 5. Wolfram media Champaign, II. 2002.
251
[40] L. Zhang, G. Naitzat, and L.-H. Lim. Tropical geometry of deep neural networks. In Interna-
252
tional Conference on Machine Learning, pages 5824-5832. PMLR, 2018
Checklist
253
254
1. For all authors...
255
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] The concrete result is the realization of parity function by our recurrent physical network by using just 6 wiring units, while a conventional digital circuit costs 7 units. That being said, the paper is written to cover much broader scope this matter of taste (an earlier version of this manuscript recieved both positive and negative comments about the scope of this article).
256
257
259

---

261
(b) Did you describe the limitations of your work? [Yes] It is mentioned that future work must extend our simulations to much higher dimensions serve as practical demonstration of deep functionality by shallow recurrent networks. Also the simulation parameters were found by trial and error, instead of being derived analytically from the
262
264
theoretical formalism these limitations are mentioned in the discussion.
265
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
267
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
288
269
2. If you are including theoretical results...
270
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
271
(b) Did you include complete proofs of all theoretical results? [N/A]
272
3. If you ran experiments...
273
(a) Did you include the code, data, and instructions needed reproduce the main ex- perimental results (either in the supplemental material or as a URL)? [Yes] Check Appendix for the code to reproduce Figure 2
274
275
276
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [N/A]
277
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [N/A]
279
280
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [N/A] It is evident that Figure is not a large-scale deep learning experiment but small-scale conceptual simulation which takes less than seconds on a modern desktop CPU.
281
282
283
284
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [N/A]
286
(b) Did you mention the license of the assets? [N/A]
287
(c) Did you include any new assets either in the supplemental material or as URL? [No]
208
(d) Did you discuss whether and how consent was obtained from people whose data you'l 're using/curating? [N/A]
290
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
281
292
5. If you used crowdsourcing or conducted research with human subjects...
293
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
294
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
296
297
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
298
A
Appendix
299
Wolfram Mathematica code to reproduce Figure 2
Mi-e simulatelf_ 9_, x_] (sys NomninnersStateSpccchhelll
(x1- (2.f) s1+ (52 53) Abs s1 (9[A] 9I3I) Ramp[s1- 9[2011,
x2 (f) 52+ (51) [b][>[[]] 52 (>[II+ r13) Ramp >[2]],,
x3 (f) f (51) [b $3 [x[[I+ x(3) Ramp y[2]]]),
(x1, x2, x3, Xor[x1, x2, x3], s1, s2, $3, y Abs[g[i] s1- (9[3] 9[31) Ramp[s1- [mpp(1 HeavisideThetaly- .15))
}, (s1, s2, s3), (x1, x2, x3)];
inputs (.5 Sq t/50), , .5 SquareMave t/ 1/10]] , .5 Squarehaaee t/ 200)); - (1s (0, 0)), inputs, (t, 9, 200) GraphtscoouumeeaabbePpoooooooor (t, 9, 200), PlotRange - All, Ticks + (Autonatic, (e, 1/ 5, 1, 2))], (1, 9)1)
simulate(1.751, (1.876, .12,, .12)), (.876, 1.6, .751)]


Reviews:
[{"review_id": "eN3Rwl2Z820", "reviewer": "Reviewer_NkSi", "summary": "A feedforward logic circuit follows certain scaling constraints (for example, the number of layers or logic switches required to compute some function). However, a cyclic (analogue) network can, in some cases, avoid these limits.\n\nThis work introduces an approach to using analogue, cyclic networks to compute the parity (highly non-linear) logic function. They analyze and simulate (to find the parameters) this for a N=3 problem.", "questions": "- What are the main contributions of this work?\n\n- Could you explain more clearly what you expect NeurIPS audiences to take away from this paper and which parts of the community are most likely to be interested.", "limitations": "na", "rating": 2, "confidence": 1, "soundness": 2, "presentation": 1, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Using recurrent functionality to reduce circuit sizes in an interesting problem (although the practical realizations make be non-trivial at scale).\n\nWeaknesses:\n- Paper is very hard to follow and makes a lot of assumptions on prior knowledge (and there is plenty of remaining space to include more background, its only 5 pages right now).\n\n- Its not clear what the main contributions of this work are.\n\n- The scale of this work is trivial, and the solution was found numerically. This approach is much more interesting for much larger scale.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "a65WYRbclqf", "reviewer": "Reviewer_jxFt", "summary": "The idea of this paper is to limit the number of connections in a physical neural network that are required to realize $N$-input Boolean functions, while focusing on the parity functions as a proof-of-concept. In feed-forward networks the Shannon bottleneck demands that there are $O(2^N)$ interconnections between neurons to realize the desired Boolean functions. The authors try to circumvent this bottleneck by using recurrent physical networks that have been shown to be able to realize certain functions with fewer units.\n\nAfter defining the state-space model of the recurrent physical network, the authors are looking for sufficient conditions of the conductance function that allow for the existence of its equilibrium points in order to prove that a stable output activation can be reached. This is achieved by committing to analyzing a conductance function that is piecewise-linear algebraically. Further, due to the nature of the proposed state-space model, the system becomes a special case of the Cohen-Grossberg model which can be shown to be globally absolute stable using a Lyapunov stability analysis. \n\nThe authors use the result of the stability analysis to simulate a shallow recurrent neural network that solves the parity problem.", "questions": "Please provide more information on how you think the analysis can be scaled up to more realistic problems.", "limitations": "The authors consider the work as a proof-of-concept and describe several opportunities for future work. An analysis on potential negative societal impact has been omitted, which is reasonable given the theoretical nature of the paper.", "rating": 7, "confidence": 2, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper shows an interesting and scientifically important direction for future physical neural network designs, which potentially could allow for many more efficient physical neural network implementations in practice. The authors provide a sound theoretical analysis of their proposed system.\n\nThe writing is very compact but of high quality. Readability could be improved by providing more background information on physical neural networks, as these are not particular common in the general machine learning community.\nIn general I would appreciate a more detailed descriptions of their methods and results - the paper is very short in its current form and I see a lot of opportunity to improve clarity without getting close to the page limit.\nI would also like to see the authors putting the paper better into context with other publications by citing related work in the field.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "_jAUacdof3", "reviewer": "Reviewer_HA6S", "summary": "This paper analyses the dynamics of recurrent neural networks and studies the equilibrium points to produce stable combinatorial output.\nThe authors propose a physical model amenable for analysis with algebraic geometry, solve for the equilibrium points under the assumption of piecewise-linear non-monotonic output, and perform Lyapunov stability analysis. The experimental phase validates the theoretical result by realizing the parity function.", "questions": "- The current work assumes piecewise-linear output that fits well combinatorial boolean functions. However, a broader class of functions can be represented or approximated as piecewise-linear (e.g., absolute value function). Is the current work limited to boolean function in its assumptions?\n- I would recommend the authors extend the manuscript with more motivations, background notions, running examples, and material to make the current work accessible to a broader audience. In particular, clearly communicating the goal and its applicability is essential. Otherwise, the submission of theoretical work, even if providing novel and strong results, could not be appreciated by the readers.\n- Section 2.2 about algebraic geometry is partially unclear. For example, equations (3) and (5) define the properties and assumptions of the problem. It would be nice to extend them with some visualizations of a running example.", "limitations": "The main limitations of this work are the clarity in communicating the theoretical results and the limited experimental phase.\nAs an extension, it might be worth validating the architecture scalability results by conducting experiments for varying N.", "rating": 6, "confidence": 1, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- *Significance*: This work advances the theoretical analysis of recurrent neural network dynamics, which might significantly contribute to the community and help design more efficient architectures. However, the paper is very concise, and part of the details is difficult to grasp at first read. In the Introduction, the final aim is not sufficiently stressed, and the discussion about the need for more efficient architectures is limited to a paragraph. The experimental phase consists of a simple experiment on standard parity function with few input dimensions. Nevertheless, it is a nice proof of concept to validate the proposed theoretical result.\n    \n- *Originality*: The paper adequately mentions prior works along the text and explains how the contribution differs from them. However, a separate section on related work might be helpful to compare the approach with prior works easily. \nThe analysis of recurrent neural networks is novel, and this work uses appropriate analysis techniques from algebraic geometry and Lyapunov stability.\n    \n- *Clarity*: The paper is well written but difficult to follow. It assumes a strong knowledge from the reader, presenting only the principal results and citing prior works from the literature. Considering it is a theoretical work, I would have appreciated a section to provide a minimal technical background for the readers. To this aim, the authors could have used all the allowed pages and provided additional content in the appendix.\n    \n- *Quality*: The submission looks technically solid, and the analysis techniques look appropriate. However, the soundness of the technical claims is beyond my knowledge.\n\n\n### Minor remarks\n\n- Figure 2 has no labels on the axis. Moreover, the vertical axis shows different lengths (see first and second plots) and scales (see last and second-last plots) for the various quantities visualized.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "-DJ86b8lMsM", "reviewer": "Reviewer_ZLGu", "summary": "To reduce tne number of connections in a neural network needed to compute a Boolean function, the authors use a recurrent neural network instead. Given input x, the RNN must settle to an equilibrium, and the first node of the network is transformed to give the output. The authors use the Cohen-Grossberg theorem to show stability if the connections are symmetric. They obtain requisite connections for the N-bit parity function by trial and error.\n", "questions": "Would say using backprop through time to learn the symmetric connections be possible while maintaining the equilibrium guarantees (during learning and at the end)? Or could you provide further insight into learning or constructing such networks. If this or some other constructive algorithm was proposed showing the connections to your theoretical methods, I would be happy to improve my rating.", "limitations": "The work is still in a rather nascent stage and these issues do not arise yet.\n", "rating": 4, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The authors motivate the work well suggesting that connectivity is the bottleneck for neuromorphic hardware rather than neuron numbers, and thus suggest recurrent neural networks that reach equilibrium as more efficient to compute static (binary) input-output mappings. They show some tantalizing connections to algebraic geometry, and try to do a piecewise linear analysis. To my inderstanding, these yield only necessary but not sufficient conditions. They instead show Lyapunov stability of the system assuming symmetric connections. The authors point out some very interesting connections that have the potential to lead to interesting results. However, the authors have been unable to develop these further theoretically. The N-bit parity function was found by trial and error and the authors do not provide a constructive algorithm to learn more general binary input-output transforms. Thus the experimental work is also rather limited and does not prescribe how these networks may be constructed or learned. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Reject