A time-resolved theory of information encoding in recurrent neural networks
Authors: Rainer Engelken
Sven Goedeke
Venue: NeurIPS 2022 Accept (2022)
Keywords: network dynamics
encoding
mean-field theory
balanced state
information theory
recurrent neural networks
dynamical systems
chaos
noise

Abstract: Information encoding in neural circuits depends on how well time-varying stimuli are encoded by neural populations.
Slow neuronal timescales, noise and network chaos can compromise reliable and rapid population response to external stimuli.
A dynamic balance of externally incoming currents by strong recurrent inhibition was previously proposed as a mechanism to accurately and robustly encode a time-varying stimulus in balanced networks of binary neurons, but a theory for recurrent rate networks was missing. 
Here, we develop a non-stationary dynamic mean-field theory that transparently explains how a tight balance of excitatory currents by recurrent inhibition improves information encoding. We demonstrate that the mutual information rate of a time-varying input increases linearly with the tightness of balance, both in the presence of additive noise and with recurrently generated chaotic network fluctuations. We corroborated our findings in numerical experiments and demonstrated that recurrent networks with positive firing rates trained to transmit a time-varying stimulus generically use recurrent inhibition to increase the information rate. We also found that networks trained to transmit multiple independent time-varying signals spontaneously form multiple local inhibitory clusters, one for each input channel.
Our findings suggest that feedforward excitatory input and local recurrent inhibition - as observed in many biological circuits - is a generic circuit motif for encoding and transmitting time-varying information in recurrent neural circuits.

A time-resolved theory of information encoding in recurrent neural networks
Rainer Engelken" Zuckerman Mind. Brain, Behavior Institute Columbia University New York, USA
Sven Goedeke
Neural Network Dynamics and Computation Institute of Geneties
University of Bonn
Bonn, Germany
Abstract
Information encoding in neural circuits depends how well time-varying stimuli are encoded by neural populations. Slow neuronal timescales, noise and network chaos compromise reliable and rapid population response external stimuli. A dynamic balance of externally incoming currents by strong recurrent inhibition was previously proposed a mechanism to accurately and robustly encode a time-varying stimulus in balanced networks of binary neurons, but a theory for recurrent rate networks was missing, Here, we develop non-stationary dynamic mean-field theory that transparently explains how a tight balance of excitatory currents by recurrent inhibition improves information encoding. We demonstrate that the mutual information rate of time-vary varying input increases linearly with the tightness of balance, both in the presence of additive noise and with recurrently generated chaotic network fluctuations. We corroborated our findings in numerical experiments and demonstrated that recurrent networks with positive firing rates trained to transmit time-varying stimulus generically use recurrent inhibition to increase the information rate. We also found that networks trained to transmit mul- tiple independent time-varying signals spontancously form multiple local inhibitory clusters, one for each input channel. Our findings suggest that feedforward excita- tory input and local recurrent inhibition observed in many biological eircuits-Be is a generic circuit motif for encoding and transmitting time-varying information recurrent neural circuits.
Introduction
How fast and reliable time- varying incoming stimuli are encoded in the population activity of recurrent neural networks constrains information transmission between local circuits and inter-areal communication. While the brain has respond rapidly and reliably changes in the world, several factors can drastically hamper information encoding in recurrent network models: the timescales of synapses be slow, single neurons exhibit Poisson-like temporally irregular dynamics and dynamic instability of recurrent neural circuit dynamics can be a source of chaotic variability [ 2121 5].
Excitation and inhibition in biological neural networks is usually conveyed by different types of neurons with a predominance of recurrent inhibitory feedback, a property known as 'inhibition dominance' 6 [ g 99. In networks with very strong recurrent inhibition, where recurrent weights scaled as I/VN with network size N, dynamic balance arises between excitatory currents and recurrent inhibitory currents that was originally proposed as a robust mechanism to describe the emergence of asynchronous irregular activity [I]. Networks of binary neurons in such balanced state track time-varying signals 2 While subsequent experimental work found evidence for a dynamic
"Correspondence re2365 columbia.edu
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

balance of input currents, the 'tightness' or 'looseness' of such dynamic balance in cortical circuits, as well as the computational implications, are subject of active scientific debate [TO] 'Loosely' balanced networks, where excitatory and inhibitory currents are respectively O (1) compared to the distance from reset to threshold, can respond in their population firing rate nonlinearly to external inputs, which was argued to be favorable for sensory processing [7]. 'Tight balance' refers to more precise tracking of total excitatory and total inhibitory input currents in time originating from excitatory and inhibitory currents that are respectively O(vN) or larger compared to the distance from reset to threshold [T0 2 Such 'tight balance' was also studied in series of works that arrived at spiking balanced networks from a normative predictive coding ansatz [T] [2] 13 [4].
How the dynamic tracking of time-dependent inputs described in binary networks 2 extends to firing rate models and how 'tightness of balance' affects the information encoding of time-varying stimuli in the presence of chaos and noise has not yet been addressed. Previous dynamic mean-field theory (DMFT) approaches to input-driven rate networks assumed that the mean of the external inputs across neurons does not depend on time, which facilitates DMFT [15] 16 17 14, but prohibits to investigate encoding of dynamic stimuli in the population firing rate. To address this gap, we study how a time-varying stimulus is encoded in the population rate of inhibition-dominateed rate network under the influence of additive noise and chaos.
We show that the accuracy of neuronal population encoding improves in more tightly balanced recurrent networks because of a speedup of the effective timescale of the population mcan dynamics. Conventional methods of dynamic mean-field theory [18 19 20 17 are not adequate to capture the effects of a time-varying common input. Therefore, we use a dynamic mean-field theory that is non-stationary, meaning that the order parameters are time- time-depnddent. Beyond similar recent approach (21), our theory can treat both time-varying common and independent external input, which is crucial analytically treat signal-to-ng -ooise ratio that constrains population coding. This novel technique accurately captures the statistics of the input-driven networks. Specifically, we calculate the cross-spectral density between input and output and the power spectral density of the population firing rate. Together with the knowledge of the input statistics, this allows us to calculate the mutual information between stimulus and population rate in the Gaussian channel approximation based on the spectral coherence.
We examine how the frequency-respone and mutual information rate depends on the tightness of balance, added noise, network chaos, and statistics of the input stimulus, using both theory and simulation. All the analytic results match those from network simulations. We show that recurrent networks that are trained on tracking a dependenn stimulus develop strong recurrent inhibition and strong positive input weights, a fingerprint of the balanced state. Concomitantly, the mutual information rate between their network readout and the stimulus increases, as predicted by our theory. This indicates that more tightly balanced state is a generic solution to reliably transmitting a time-varying input in the presence of noise or chaos. Lastly, we find that networks trained on simultancously transmitting multiple independent time-varying stimuli spontancously break up into weakly connected subnetworks with strong local inhibition.
Our findings have important implications for information encoding in RNNs and for understanding how neural network architecture design shapes noise- robusteessssss and information encoding.
The main contributions are the following:
A novel non-stationary mean-field theory describing the dynamics of balanced RNNs driven by a time-varying input signal and noise (section 2 and appendix A
A transparent mathematical link between tightness of balance and the mutual information
rate between a scalar input signal and the population rate (section 4 and appendix D).
A frequency-resolved mutual information rate analysis showing the effects of noise and chaos (section 3 and appendix C).
Training results indicating that vanilla networks trained on transmitting high-frequency signals become more tightly balanced throughout training (section and appendix E
Additional training results showing that networks trained to transmit multiple independent signal exhibit a spontaneous symmetry breaking into weakly connected subnetworks with strong local inhibition (section and appendix [I.
Additional training results with a nonlinear task (section [). different activation function (appendix H and excitatory-inhibioo. architecture G demonstrate generality of our results.

---

Population coding in recurrent networks
We study how well varying scalar input [tt) is encoded in the mean population firing rate (tt) = (t) of a recurrent network of N nonlinear rate units ("neurons') that obey
dh T dt
hh, + Σ (h,) + 61 t((),
(1)
with the coupling weights Jij = -JoN + Jay. The Jıy are drawn independently from a Gaussian distribution with zero mean and variance g²/N. where g is a gain parameter that controls the heterogeneity of weights. We use threshold-linear transfer function Φ(x) max(æ; 0). The
A
D
E
Input ((t)
0.25
100
theory simulation
0.00
-0.25
Mutual information (bit/r)
10a
5
10
Population rate Population rate
B
0.6
1
Coherence
0.4
5
10
102
1.25
1.00
0.75
10-1 1
10
100
102
100
10a
102
Time (z)
Frequency (1/T)
balance b
Figure 1: Mutual information rate between input and population response grows with tightness of balance A) Each neuron in the recurrent network receives an identical input stimulus [(t) and additive Gaussian white noise (t) that is independent across neurons. B) For small values ("loosely' balanced networks), the population response only tracks slow fluctuations of the stimulus = 1). C) For large values of ("tightly" balanced networks), the population response also tracks fast fluctuations of the stimulus (b = 100). D) The spectral coherence for different values of balance direct numerical simulations (shaded line) and mean-fiell field theory (dashed line) superimposed. in Gaussian channel approximation based on spectral coherence, mean-field theory (dashed line) and direct numerical simulation (squares). Model parameters: N = 212 g 0, Al 1/210,, Laim 230,, To Jo Ts = 0.1, 11 1/23. nperseg-213.
(For color-code, points in figure 1E). E) The mutual information
external input contains signal component [()), which is identical across neurons, and noise (t),, which is independent across neurons. For concreteness, we choose & to be independent additive white Gaussian noise processes (AWGN) with autocorrelation function (L) + (')) 78()(())
and the signal [(t) to be an Ornstein- Fmsttin-Uhlenbeekk Uhlenbeck process given by
Ts dt
(t) = - + (t)
(2)
The amplitude of the AWGN (!) driving the time-varying part is denoted by 7i. The parameter b multiplies both externl input and recurrent coupling strength and thus regulates the tightness of balance [1 10 21. For firing-rate networks, the ability to track time-dependent input strongly

---

depends on the strength of balance b (F For loosely balanced network (b = 1) (Fig B3). the network population rate only tracks slow fluctuations of the input signal. contrast, for tightly balance networks (Fig [e), the population firing rate also tracks fast input fluctuations. To understand how this tracking arises in the model, it is useful to rewrite [ decomposing h; = m + h; into common component and residual fluctuations. With Jij -bJoNN + Jej. this results in
dm dt
= m --oo(()++b()))) +b((1)
(3a)
dh, -hi + (1)) &(()) dt
(3b)
Here, the signal [(t) enters the expression for m. because it is identical across all neurons. It thus directly impacts the mean population rate v(t) and recruits, through the negative mean coupling -JJo, strong recurrett feedback - (boo((t) that is anti-correlated with the input and cancels most of the common external input [tt. Rewriting Eq Eq[aalby by solving for v(t) gives
T(t) + bJo
dm
((t)
m
Jo
df
This equation is commonly referred to the 'balance equation' 12 19 20] when the input is not time-dependent. For large b, the population rate approaches v(t) = /1)/ which indicates a linear relationship between imput and population rate for tightly balanced networks pointed out before in binary networks 2 On the other hand, the noise & affects only the residual fluctuations hT.
As the balance parameter b is increased, the effective timescale of the dynamics of m becomes shorter by factor of b, as can be seen by dividing Eq 3a by b resulting in timescale of T/b. This leads to faster and more precise tracking. In contrast, the timescale of the equation for h, is unchanged by b. Assuming a separation of timescales, for increasing 6, the lincarized rate dynamics develops negative real outlier eigenvalue of size bb. Of course, the two equations for m and h; are interdependent via ((+)) = (mm + h2), so for a complete treatment, the joint dynamics of m and h, has to be solved. In appendix A we present non-stationary dynamic mean-field theory of temporal population coding.
3 Frequency-dependden signal encoding in the population response
We systematically analyze how different input frequencies are encoded in the population response and observe that the transmission of high frequency stimuli is improved for large b ('tightly balanced' regime) (Figure A and C). We caleulate the dynamic gain G(J) = |Stz //rrr of the recurrent network both in direct numerical simulations and with the non-stationary DMFT. St is the Fourier transformation of the input-output cross-correlation function between m(t)) and v(t). Si1 is the power spectral density of the stimulus T(t). The dynamic gain quantifies how variation in the input signal / (t) affects the population rate vtt) and is a standard analysis for characterizing the response properties of individual cells 222)
We calculate G(f)) using both direct numerical network simulations and dynamic mean- -ield theory. For details of the spectral analysis, see appendix C We observe that increasing b boosts the dynamic gain for high frequencies (Figure Z and C for g and g 2). This direct consequence of the speedup of the dynamics of the mean discussed above. Thus, as the network becomes more tightly balanced, high-frequene, signals can be encoded and transmitted more reliably. To quantify the improved information encoding we turn to an information-theoreee analysis next.
4 Mutual information rate between time-varying stimulus and population response
With information theory, we can treat the recurrent neural network as a noisy communication channel, transforming a signal embedded in noisy current into population rate that can be read out by another population. We use a Gaussian channel approximation to the mutual information rate between input current (()) and output mean population rate v()) based on the spectral coherence 23 24
Ras (I, = 10)= Sennot If (1 - Cl=(f))
(5)
4

---

19
17.5
25.0
Mac inforattnn dendi (N00
15.0
Oynamk gain (10)
12.5
12.5
10-)
10-
7.5
7.5
5.0
5.0-
10.2
100.0
1000.0
10
0.0
107
10
107
107
Frequency
Prequency 0ld
Frequetcy 13/
Frequency (3rl)
Figure 2: Brequency-resolve information rate reveals different effects of white noise and chaos A) The normalized dynamic gain for different values of balance b, direct numerical simulations (shaded line) and mean-field theory (dashed line) superimposed for 0. Note that for large values of b, the dynamic gain shows an improved encoding bandwidth (For color-code, see figure 2C). B) The mutual information rate density in the Gaussian channel approximation based on spectral coherence, mean-fichl field theory (dashed line) and direct numerical simulation (transparent full lines) for g 0. C) Same as A) but for g = 2. D) Same B) but for 2. The recurrent residual fluctuations in h; arising from
reduce the information rate for low-frequency information. For larger values of ("tight balance') recurrent fluctuations have weaker effect on low-frequency information rate, because of tracking of network fluctuations. Model parameters: N = 4096, f 0 for A) and B).
g = 2 for C) and D), At = 1/2 faim = 28T Io Jo = 1, Ts = T, 0 0.1, 11 = 1/8.
where (f) denotes the magnitude squared spectral coherence. The spectral coherence is the frequency-dome analog the correlation and measures the linear relationship between frequency components of input and output signal. Its magnitude square is
(fI
Civ(f)
(6)
(f)
It is normalized by the power spectrum S (f)) of the input signal (an OU process) and the power spectrum (f) of the population rate v(t).
We calculated the spectral coherence two independent ways: firstly based on the non-stationary DMFT and secondly based direct numerical simulations. The dependence of Crr the tightness of balance b is depicted in I. DMFT and direct numerical simulations with additive white noise {() are in excellent agreement.
We find that the mutual information rate grows approximately linear with for an OU-input signal and Gaussian white noise (see Figure IE and analytical approximation in appendix DD.. This is consistent with an analytical derivation of the mutual information rate in a lincar approximation, where we find the mutual information rate (in not/T) to be
bLVN
(I, v) 20 TT/S'
for bi TT/T > o/VN, i.e., for large networks and large signal-to-noise ratio. The mutual information rate saturates for large b; the saturation level is determined by the band limit of the incoming signal and noise (see figure appendix C). We conclude that the mutual information rate scales approximately linearly with the tightness of balance b for sufficiently high band limit. This conclusion not restricted to networks of rectified linear units, a similar reasoning holds also for the threshold-power-lag nonlinearities and other neuron models (See e.g. [19 20 for balanced rate networks with other input-output transfer functions). Finally, we note that for an input signal that is an OU-process, we would expect different scaling of the mutual information rate.
5 Frequency-resolved mutual information rate analysis
Inspired by recent work on the rrequency-resolvedd information encoding rate single cells 25), we analyze the population rate of RNNs and study how the encoding of different frequency components
5

---

of the input signal depends on the tightness of balance b, the strength of added noise and the heterogeneity of the recurrent weights g.
For that, we considered the mutual information rate density r(f), which gives a frequency-resolved
quantification of the information encoding rate (See for details of definition 25
(ff = - logg (1 - (f)(f))
(8)
Note that since the integral over frequencies of rf)(S) gives the Gaussian channel approximation of the mutual information rate, we thus call it mutual information rate density and remark that
has units of bit/r I/r We observe that tightly balanced networks can transmit more high-frequency O and y = 2). Overall, the information rate for more tightly balanced networks increased because of the boost of the signal-to-noise ratio coming from the fact that the signal (t) is scaled by b, while the noise is independent of b. The above-described frequency- dependent effect comes on top of that.
information (Figure 2 and D for both
For recurrent networks with weight heterogeneity (g > 0), there are additional slow residual fluctua- tions in h that act as an additional source of variability. Moreover, increasing leads to an increasing fraction of units in saturation, which reduces information transmission (Compare Figure 2 for g =
and D for = 2). These low-frequency fluctuations have a different effect than additive Gaussian white noise (see also 14)
For more tightly balanced networks (larger b), the residual fluctuations do not impair information encoding (see darker green lines in figure 23), because they canceled. A related phenomenon was observed in densely connected recurrent binary networks ((26) and spiking networks ((27)). That variance originating from weight heterogeneity (q > 0) is more effectively reduced by balanced networks as a function of b compared to variance originating from added Gaussian white noise was already observed in rate networks with static input [14]
6 Training networks on auto-encoding results in tight balance
We corroborate the theoretical finding that the tightness of balance has a crucial role in information encoding in numerical experiments trained recurrent networks, which demonstrates the generality of our findings. For that, we train recurrent networks on an auto-encoder task (Fig 3). We initialize small-sized recurrent networks (N = 100) that follow the dynamics of Eq. [ in a loosely balanced state, with recurrent weights drawn i...d. from Gaussian Jaj ~ (-11/NN 9 /N) and input weights whth and output weights wit are drawn from Gaussian distribution with a positive mean. This initialization would correspond to b = 1 in our theoretically studied model (Eq. D. The network receives the vime-narying input signal (t)
wif"((t), so in contrast to our theoretical setup, in principle each neuron can receive the input signal I(t) with a different scale w.". We then trained input weights wi output weights n and recurrent weights Jıs using backpropagation tirooonttthhghhhhh time with the ADAM optimizer with standard hyperparameters (see appendix [ for additional details on training setup and additional controls of training results). We minimize the mean squared loss 1 1= ((((t) I2 dt between a time- varying input signal [tt and a linear readout of the recurrent network activity ī(t) 1/N Ee "o ((h,(t)).
Initially, the network only poorly tracks the time-varying input signal, as theoretically expected for small 6. However, we find that the network learns to track the time-varying input signal (Figure 3A). as the test error falls throughout training epochs (Figure 33)). Analyzing the eigenvalues of the trained network, indicates an emerging strongly negative mode implementing the fast tracking (FigureBB) This is confirmed by measuring the empirical tightness of balance b in the connectivity, which we quantify simply by the absolute value of the mean of the sum of incoming recurrent weights into each neuron b == |1/N Σ, E, Jayl- This definition of b is consistent with the definition of in our theory above. We find that the trained networks that are initialized with 6 become progressively more tightly balanced (Figure 3C) over training epochs, as can be seen by a growing b and by an improving high-frequency encoding (Figure 3t. We prematurely stop after 100000 training epochs to avoid numerical instabilities for very tightly balanced networks. We note that the average of the external input weights & E; wit becomes more positive over training epochs (Figure 3C inset, green line)
observe an outlier eigenvalue with a very negative real part after training that

---

B
C
target
pre training
4
pre training
post training*
readout
1.5
post training
Imaginary(Ar
balance
40
0
-25
1.0
-2
20
1
0.5
-4
0
1e5
2.5
5
-50
-25
0.0
0.5
1.0
Time (T)
Real(A))
Epoch
le5
D
E
Mutual information (bit/r)
F
10
pre training
0.100
Dynamic gain (1/r)
100
post training
test error
0.075
0.050
50
0.025
0.000
10-1
0.0
0.5
1.0
10a Frequency (1/r)
10-1
0.0
0.5
1.0
Epoch
1e5
Epoch
1e5
Figure RNNs trained on tracking time-varying input become more tightly balanced through- out training. A) Threshold-linear RNNs are trained to approximate time-varying external input T(t) by linear readout f(t) 1/N Σ. w (hh4), by minimizing the mean squared loss = f - (t)) dt. B) The eigenvalue spectrum of the dynamics linearized the fixed point without lime-dependent input indicates that during training, an eigenmode with strongly negative real part emerges. C) The networks become more balanced throughout training. The tightness of balanced is quantified here by the magnitude the mean recurrent coupling |1/N Σ, Σ; Jiyl-
Inset shows that average of the external input weights N Σ, w becomes more positive over epochs (green line) while the average row sum of the recurrent weights (black line) become more negative. D) The loss across learning epochs. E) The normalized dynamic gain G(f) before and after training indicates improved high-frequency encoding as the network becomes more tightly balanced. The mutual information between the time-varying external input and the linear readout function of training epochs. Information rate increases throughout training. Model parameters initialization:
N 100, 9 V2, At 0.0lT, Łim 10r, b 1, To Jo = 1,7 0.1, TS 0.1, a 1, epochs= 10
while the average sum of the recurrent weights (Figure 3C inset, black line) become more negative.
Consistent with our theory, we find that the mutual information rate (Gaussian channel approxima- tions) grows over training epochs as the network becomes more tightly balanced (Fig 3F.. Note that we did not train the network to maximize the mutual information rate, but to minimize the mean-squared error between target output and actual output, but the two are closely related (28) note that this tracking does not rely the threshold-linear input-output function, in fact, we also successfully trained sigmoid (Fig. 99, linear (Fig. TO) and threshold-quadratic input-output function
(Fig. [[] (see appendix H for additional results).

---

7 Training on multiple inputs results in tightly balanced subnetworks
We note that networks trained to simultancously track multiple input Ornstein-Uhlenbet signals by a linear readout exhibit spontaneous symmetry breaking into weakly connected subnetworks with strong local inhibition (Fig. and see appendix [ results on training on multiple signals). We note that this finding is not limited to training networks two independent OU-processes; strongly inhibitory coupled subnetworks also emerge when training on more OU proceeess (See Figure 4F for the resulting connectivity of network trained OU-processss). For networks trained multiple time-varying stimuli, we observed an interesting training dynamics. First, they develop a single global inhibitory mode (figure 7 similar networks trained track single OU-signal. After some time, we observe the emergence of a second outlier eigenvalue. Concomitantly, the network breaks into inhibitory subnetworks and the readouts start track both OU-input signals independently.
targets
initial trained
outputs
c
pre training
2 stimuli
1.5
0.5
2
imag(A)
M
50
0.0
0
-2
1
-0.5
-2
0.5
-4
100
E
-25
1.5
G
H
continous XOR
3 stimuli
readouts/ inputs
1.0
W
imag(A,)
0.25
0.1
0
50
0.00
0.0
0.5
0.25
0.1
-1
0
50
100
0.0
2
55
0
Time (T)
rea((A))
Figure 4: Training on multiple signals and nonlinear task yields balanced subnetworks. A) RNNS trained to approximate two time-varying external inputs [(t) by two linear read- outs ((t)= = 1/N E, wiy o(₂)), by minimizing the mean squared loss = E. f [(((t) - (t)))² dt. B) Two eigenmodes with strongly negative real part emerge. C) An example random recurrent network Jıj depicted at initialization. Note that input and output weights are also random. D) Same example network after trained on two independent OU-processes. The network spontaneously breaks into two balanced subnetworks that have strong local inhibition but weak connectivity in between throughout training. Clustered by k-means clustering on the input weights. E) Same RNNs trained to calculate on the two time-varying external inputs (t()) the continuous-time XOR (the function f(I (t), T2)))) (t) [2( (Green dashed line is input, black dotted line is target output (t), T2())), red line is actual output). F) For XOR one eigenmode with a strongly negative real part emerges. G) Recurrent weight matrix after training on XOR shows balanced subnetworks that have strong local inhibition. Clustering by k-means clustering of input weights. F) An example recurrent network Jıj that is trained to track 3 OU-processes develops three balanced subnetworks.
Model parameters at initialization: = 100, 22, At 0.27, faim = 100r, = 1, To = Jo = Ts = 0.1, 1, epochs= 10a.
Training on nontrivial computation results in tightly balanced network
Finally, we demonstrate the generality of our findings on two nontrivial tasks where the RNN not only copies time-vare sure-narying input but performs computation on the input streams.

---

Firstly, we trained recurrent networks that receive two time- varying input signals to output a continu- ous extension of the XOR function (t), [2)))) [t) (t)). which satisfies for 11, 12 E 0, 1 the binary XOR function (F. A similar, but not time-dependent task was used earlier to demon- strate that balanced networks can do nonlinear computations [29]. Consistent with our theory, we observe that the networks developed an outlier eigenvalue with strongly negative real part (Fig (A). K-means clustering of the recurrent weight matrix revealed that the RNNs developed tightly balanced subnetworks throughout training (Fig 4).. See appendix [ for further details. Note that a substantial fraction of neurons in the RNN are silent after training, we leave a further analysis these silent fraction for future work.
Secondly, we trained RNNs to compute a linear transformation of time-varying vector of stochastic processes (t( = Σ; (t) at every moment in time. We observed that the RNNs develop number of negative outlier eigenvalues and concomitantly the same number of tightly balanced inhibitory subnetworks that corresponds to the rank of the linear input-output transformation A (See Fig 12 and appendix F for further details).
9
Limitations
Here we considered the encoding of multivariate scalar input signals only numerically, it remains an important challenge to extend our theory to multidimensional input signals, including the training dynamics. For mathematical tractability, we considered firing rate networks here, but for more detailed neuron models, other biophysical features shape the information encoding rate. For example, in spiking neuron models, fluctuating background input can enhance the high-frequency encoding
[30] Moreover, the spike generation mechanism [31 [3] 33, 34 [35 (36,, the synaptic dynamics (137 (8) and other latent variables (c.g. adaptation [39]] shape the
It remains an important future work to extend the work here by such contributions. Moreover, moving beyond point-neurons, the shape of the dendritic tree can also affect the frequency response [36..
While we show that the dynamics of RNNs trained on several linear and nonlinear tasks is consistent with our DMFT, it remains an important challenge to consider tasks with more complex temporal structure, e.g. involving memory, where the task dynamics might interfere with the tightness of balance.
Finally, our theory becomes exact only in the limit of large network size, and it is important to also consider finite-size effects. For finite network size, both noise and chaotic fluctuations also contribute fluctuations of the mean m. which are also recurrently canceled. This was described previously in recurrent networks in a linear regime (27). we merely provide a heuristic extension of the DMFT for finite-size fluctuations (Appendix but full theory would be desirable.
10
Discussion
We show how tight balance of excitatory currents by recurrent inhibition improves information encoding of a time-varg varying signal. We demonstrate that the mutual information rate of a lime-varying signal increases linearly with the tightness of balance, both in the presence of additive noise and with chaotic fluctuations of the recurrent network activity. A non-stationary dynamic mean-field theory reveals a separation of timescale between the mean currents and the timescale of residual fluctuations. The mean dynamics become linearly faster with the tightness of balanced and enable reliable encoding of Lime-varying signals. In contrast, the timescale of the chaotic dynamics in the residual fluctuations are largely unaffected by the tightness of balance. We find that networks become more robust to deteriorating effects of fluctuations from noise and chaos as the network becomes more tightly balanced.
Our study relevant in the recent debate on the functional implications of how tightly excitatory [4]. We address this question by building measures of information encoding that were previously used
currents are tracked by recurrent inhibition [26 [3]
a bridge from ieformation-thhoreee in neuroscience mostly in sensory systems 40 42
24 to dynamic mean-field descriptions of recurrent network dynamics that were previously used to describe the often chaotic dynamics of
23
recurrent rate networks [18 19
20
16 43
14

---

Besides the implications on information encoding of more loosely or more tightly balanced networks, of course also biophysical, energetic, and evolutionary constraints should be considered. Biological networks can naturally not be arbitrarily tightly balanced, which would require arbitrary large synaptic currents. Very tight balance might also questionable for energetic reasons, as was asked previously, "Why should the cortex simultaneously push on the accelerator and on the brake?" 45]. However, such biophysical and energetic constraints may be better addressed in biophysically more detailed models.
We found that training networks on tracking time time-dependent signal by a linear readout by minimizing the squared error make them more tightly balanced, as reflected in more negative mean recurrent weights and more positive feedforward input weights. Moreover, the training also increased the mutual information rate between the input signal and linear readout. This is consistent with our theoretical result. Furthermore, the fact that training arrives at tightly balanced solution may suggest that this is a typical solution for a network. It is an interesting question when such balanced solutions will emerge with biologically plausible learning rules [46
Previous studies on the effect of time-varying input in chaotic rate networks were limited to in- dependent inputs across neurons in the form of stochastic [T3] I7] and sinusoidal [6] drive, but the networks were not balanced, and their connectivity had zero mean coupling. In these previous studies, the distribution of imputs across the population is indepeenennd [TS] 16 17] and stationary
dynamic mean-fiele theory was sufficient to describe the results. However, the treatment of common input is only possible by the non non-stationary dynamic mean-field approach introduced here.
The dynamic cancellation of time-varying input through recurrent inhibitory feedhack has been previously studied in balanced networks with binary 2 26 and spiking neurons 27 50 51
20 52. [7]. but the dynamic cancellation of correlated input and its implications on information encoding in rate networks were not investigated, nor were the implications for training such networks in machine learning setup with backpropagation through time. It would be interesting to extend our mean-field analysis rate
Chaos in balanced firing-rate networks was studied previously [9
networks with pre-exine existing low-rank structures on top of the random structure [53 54 55 (56].
The underlying mechanisms of tracking of time- varying input analyze here are not specific to fully- connected threshold-lines RNNs driven by Ornstein-Uhhe Uhlenbeck signals with additive Gaussian white noise, which we merely chose for the sake of simplicity and analytical tractability. Rate networks with other non-negative input-output transfer functions exhibit a qualitatively similar dynamic tracking in the tightly balanced regime. Moreover, the mechanism of tracking described here is closely related to balanced predictive coding networks 13 14].
A testable hypothesis based on our findings is that changing the tightness of balance, e.g. by pharmacological manipulation or by genetic knockout that affects net recurrent inhibitory strength (without generating runaway activity pathological network states), would also affect the high- frequency stimulus encoding and information transmission. Specifically, we predict that stronger effective recurrent weights in conjunction with stronger excitatory inputs would improve both the high-frequent encoding and the mutual information rate between a stimulus and the population response. Conversely, when weakening recurrent inhibition and external input strength, we predict an impaired high-frequency encoding and a lower mutual information rate. Such predictions could not only be tested in vivo, but also through in vitro experiments, where the number of recurrent synapses can be manipulated (57).
Acknowledgments and Disclosure of Funding
Research supported by NSF NeuroNex Award DBI-1707398 (RE), the Gatsby Charitable Foundation (RE), the Bernstein Award 2014, 01GQ171 (SG), and the Swartz Foundation Award (2021-6) (RE). We thank L.F. Abbott, A. Ingrosso, C. Cueva, J. Kadmon, Keijser, R. Khajeh, L. Logiaco, A. Neef, K. Rajan, C. Machens, K. Miller, T. Nguyen, S. Solla, C. v. Vreeswijk (f) and F. Wolf for fruitful discussions. The funders had no role study design, data collection and analysis, decision to publish, or preparation of the manuscript.
10

---

References
[T] C. van Vreeswijk and H. Sompolinsky. Chaos in Neuronal Networks with Balanced Excitatory and Inhibitory Activity. Science, 274(5293) -1726, December 1996.
[2] C. van Vreeswijk and H. Sompolinsky. Chaotic Balanced State in a Model of Cortical Circuits. Neural Computation, 10(6):1321-13711 1998.
[3] Michael Monteforte and Fred Wolf. Dynamical Entropy Production in Spiking Neuron Networks in the Balanced State. Physical Review Letters, 105(26):268104 December 2010.
[4] Guillaume Lajoie, Kevin K. Lin, and Eric Shea-Brown. Chaos and reliability in balanced spiking networks with temporal drive. Physical Review E, 87(5):052901, May 2013.
[5] Rainer Engelken, Fred Wolf, and L. F. Abbott. Lyapunov spectra of chaotic recurrent neural networks. S.KKv::2006.22222 [nlin, q-bio), June 2020. arXiv: 2006.02427.
[6] Hirofumi Ozeki, Ian M. Finn, Evan S. Schaffer, Kenneth D. Miller, and David Ferster. In- hibitory Stabilization the Cortical Network Underlies Visual Surround Suppression. Neuron, 62(4))578-592 May 2009.
[7] Yashar Ahmadian, Daniel B. Rubin, and Kenneth D. Miller. Analysis of the stabilized supralinear network. Neural computation, 25(8):1994 2037, August 2013.
[8] Fred Wolf, Rainer Engelken, Maximilian Puelma- Puelma-Touzel, Juan Daniel Flórez Weidinger, and Andreas Neef. Dynamical models of cortical circuits. Current Opinion in Neurobiology,
25:228- April 2014.
[9] Alessandro Sanzeni, Bradley Akitake, Hannah C Goldbach, Caitlin E Leedy, Nicolas Brunel, and Mark Histed. Inhibition stabilization is widespread property of cortical networks. eLife, 9ce54875, June 2020.
[10] Yashar Ahmadian and Kenneth D. Miller. What is the dynamical regime of cerebral cortex? Neuron, 109(21):3373-333 13991 November 2021.
[11] David GT Barrett, Sophie Denève, and Christian K Machens. Optimal compensation for neuron loss. eLife, 5.
[12] Martin Boerlin, Christian K. Machens, and Sophie Denève. Predictive Coding of Dynamical Variables in Balanced Spiking Networks. PLOS Comput Biol, 9(1 (1)011003325 November 2013.
[13] Sophie Denève and Christian K. Machens. Efficient codes and balanced networks. Nature Neuroscience, 19(3):375-388 March 2016.
[14] Kadmon, J. Timcheck, and S. Ganguli. Predictive coding in balanced neural networks with noise, chaos and delays. Advances in neural information processing systems, 33, December 2020.
[15] L. Molgedey, J. Schuchhardt, and H. G. Schuster. Suppressing chaos in neural networks by noise. Physical Review Letters, (9(2)) December 1992.
[16] Kanaka Rajan, L. Abbott, and Haim Sompolinsky. Stimulus-dependent suppression of chaos in recurrent neural networks. Physical Review E, 82(1):01 1901 July 2010.
[17] Jannis Schuecker, Sven Goedeke, and Moritz Helias. Optimal Sequence Memory in Driven Random Networks. Physical Review X, 8(4):041029, November 2018.
[18] H. Sompolinsky, A. Crisanti, and H. J. Sommers. Chaos in Random Neural Networks. Physical Review Letters, 61(3):259-263 July 1988.
[19] Jonathan Kadmon and Haim Sompolinsky. Transition to Chaos in Random Neuronal Networks. Physical Review X, 5(4):04 November 2015.
[20] Omri Harish and David Hansel. Asynchronous Rate Chaos in Spiking Neuronal Circuits. PLoS Comput Biol, 11(7) 1002666. July 2015.
[21] Rainer Engelken, Alessandro Ingrosso, Ramin Khajeh, Sven Goedeke, and L. F. Abbott. Input correlations impede suppression of chaos and learning in balanced firing-rate networks. PLOS Computational Biology, 18110)e December 2022.
[22] Mathew H. Higgs and William J. Spain. Conditional Bursting Enhances Resonant Firing in Neocortical Layer 2-3 Pyramidal Neurons. The Journal of Neuroscience, 2985):1285-1199 February 2009.
11

---

[23] Fred Rieke. David Warland, Rob de Ruyter van Steveninck, and William Bialek. Spikes: Exploring the Neural Code. Bradford Book, reprint edition edition, June 1999.
[24] W. Bialek, F. Ricke, RR de Ruyter van Steveninck, and D. Warland. Reading neural code. Science, 252(5014):1854-185533 June 1991.
[25] Davide Bernardi and Benjamin Lindner. A frequeeccy-reoolved mutual information rate and its application to neural systems. Journal of Neurophysiology, 1333):1342-1337, March 2015.
[26] Alfonso Renart, Jaime de la Rocha, Peter Bartho, Liad Hollender, Néstor Parga, Alex Reyes, and Kenneth D. Harris. The Asynchronous State in Cortical Circuits. Science, 327(5965):5875590 January 2010.
[27] Tom Tetzlaff, Moritz Helias, Gaute T. Einevoll, and Markus Diesmann. Decorrelation of Neural-Network Activity by Inhibitory Feedback. PLOS Comput Biol, 108)ee000256, August 2012.
[28] Dongning Guo, S. Shamai, and S. Verdu. Mutual information and minimum mean-square error in Gaussian channels. IEEE Transactions on Information Theory, 51(4): April 2005.
[29] Cody Baker, Vicky Zhu, and Robert Rosenbaum. Nonlinear stimulus representations in neu- ral circuits with approximate 16(9))e1008192, September 2020.
-iniiiitryyiiiiitorryy balance. PLOS Computational Biology,
[30] Bruce W. Knight. Dynamics of Encoding in Population of Neurons. The Journal of General Physiology, 596)7773-7 June 1972.
[31] Nicolas Fourcaud- David Hansel, Carl van Vreeswijk, and Nicolas Brunel. How Spike Generation Mechanisms Determine the Neuronal Response to Fluctuating Inputs. The Journal of Neuroscience, 23(37): 1328)): (etp::111281166444 1640, December 2003.
[32] B. Naundorf, T. Geisel, and F. Wolf. Action Potential Onset Dynamics and the Response Speed of Neuronal Populations. Journal of Computational Neuroscience, 18(3):997-309 June 2005.
[33] Wei Wei and Fred Wolf. Spike Onset Dynamics and Response Speed in Neuronal Populations. Physical Review Letters, 106(8):088 122, February 2011.
[34] Tatjana Tchumatchenko and Fred Wolf. Representation of Dynamical Stimuli in Populations of Threshold Neurons. PLoS Comput Biol, 7(10)x:1002239 October 2011.
[35] Elinor Lazarov, Melanie Dannemeyer. Barbara Feulner, Jörg Enderlein. Michael Gutnick, Fred Wolf, and Andreas Neef. An axon initial segment is required for temporal precision in action potential encoding by neuronal populations. Science Advances, :ceaauc388621, November 2018.
[36] Chenfei Zhang, David Hofmann, Andreas Neef, and Fred Wolf. Ultrafast population coding and axo-somatic compartmentalizztionn PLOS Computational Biology, 18(1): January 2022.
[37] Nicolas Brunel, Frances S. Chance, Nicolas Fourcaud, and L. F. Abbott. Effects of Synaptic Noise and Filtering on the Frequency Response of Spiking Neurons. Physical Review Letters, 86(10)2116-2188, March 2001.
[38] Tatjana Tchumatchenko, Aleksey Malyshev, Fred Wolf, and Maxim Volgushev. Ultrafast Population Encoding by Cortical Neurons. The Journal of Neuroscience, 31(34):12171 -12179, 2011.
[39] Maximilian Puelma Touzel and Fred Wolf. Complete Rate Response of Neurons with
Complex Intrinsic Dynamics. PLOS Computational Biology, 11(12):e1006636, December 2015.
[40] van Hateren and H. P Snippe. Information theoretical evaluation of parametric models of gain control in blowfly photoreceptor cells. Vision Research, 41(14))1851-188 June 2001.
[41] J. H. van Hateren. Processing of natural time series of intensities by the visual system of the blowfly. Vision Research, 37(23)33477-44
December 1997.
[42] J. H. van Hateren, L. Rüttiger, H. Sun, and B. B. Lee. Processing of Natural Temporal Stimuli by Macaque Retinal Ganglion Cells. The Journal of Neuroscience, 22(22):9945-9966 November 2002.
[43] M. Stern, Sompolinsky, and L.. F. Abbott. Dynamics of random neural networks with bistable units. Physical Review E, 90(6):062700, December 2014.
12

---

[44] Samuel P. Muscinelli, Wulfram Gerstner, and Tilo Schwalger. How single neuron properties shape chaotic dynamics and signal transmission in random neural networks. PLOS Computa- lional Biology, 15(6)=e100712) 1007122, June 2019.
[45] Jeffry S. Isaacson and Massimo Scanziani. How Inhibition Shapes Cortical Activity, Neuron, 72(2):231-243, October 2011.
[46] Wieland Brendel, Ralph Bourdoukan, Pietro Vertechi, Christian K. Machens, and Sophie Denève. Learning to represent signals spike by spike. PLOS Computational Biology, 1600)x10 March 2020.
[47] I.. Bernáez. Timón, P. Ekelmans, S. Konrad, A. Nold, and T. Tchumatchenko. Synaptic plasticity controls the emergence population- wide invariant representations in balanced network models. Physical Review Research, 4(1 (1):013362., February 2022.
[48] T. P. Vogels. H. Sprekeler. F. Zenke, C. Clopath, and W. Gerstner. Inhibitory Plasticity Balances Excitation and Inhibition in Sensory Pathways and Memory Networks. Science, 334(6062)
1573, December 2011.
[49] Pietro Vertechi, Wieland Brendel, and Christian K Machens. Unsupervised learning of an efficient short-term memory network. In Advances in Neural Information Processing Systems, volume 27. Curran Associates, Inc., 2014.
[50] Ran Darshan, William E. Wood, Susan Peters, Arthur Leblois, and David Hansel. A canonical neural mechanism for behavioral variability. Nature Communications, 8:15415, May 2017.
[51]
Robert Rosenbaum, Matthew A. Smith, Adam Kohn, Jonathan E. Rubin, and Brent Doiron. The spatial structure of correlated neuronal variability. Nature neuroscience, 20(1):107-114 January 2017.
[52] Francesca Mastrogiuseppe and Srdjan Ostojic. Intrinsically-generaeed fluctuating activity in excitaoory-inhibboory networks. PLOS Computational Biology, 13044)4 April 2017.
[53] Johnatan Aljadeff, Merav Stern, and Tatyana Sharpee. Transition to Chaos in Random Networks with Sell-Type-Specific Connectivity. Physical Review Letters, 1 11408)008101, February 2015.
[54] Johnatan Aljadeff, David Renfrew, Marina Vegué, and Tatyana O. Sharpee. Low-dimensional dynamics of structured random networks. Physical Review E, 93(2):222302, February 2016.
[55] Francesca Mastrogiuseppe and Srdjan Ostojic. Linking Connectivity, Dynamics, and Com- putations in Law-Rank Recurrent Neural Networks. Neuron, 99(3):609-633 0e0-6000-663229999 623.e29, August 2018.
[56] Itamar D. Landau and Haim Sompolinsky. Macroscopic fluctuations emerge in balanced networks with incomplete recurrent alignment. Physical Review Research, 3(2) June 2021.
[57] Jérémie Barral and Alex D. Reyes. Synaptic scaling rule preserves excitatory-inhibitory balance and salient neuronal network dynamics. Nature Neuroscience, 19(12): 1696. December 2016.
[58] Andrew M. Saxe, James L. McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. arXiv: 1312.6100 [cond-mat, q-bio, stat], December 2013. arXiv: 1312.6120.
Checklist
1. For all authors...
(a) Do the main claims made the abstract and introduction accurately reflect the paper's contribu- tions and scope? [Yes]
(b) Did you describe the limitations your work? [Yes] See Section 9
(c) Did you discuss any potential negative socictal impacts of your work? [Yes]] applicahle
(d) Have you read the ethics review guidelines and ensured that your paper conforms them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes], see result section and appendix.
13

---

3. If you experiments...
(a) Did you include code, data, and instructions needed to reproduce main experimental results (either in the supplemental material or as URL)? [Yes] See supplement and here training- balanced-
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See training section and supplement
(c) Did you report error bars (c.g., with respect to the random seed after running experiments multiple times)? [Yes] Error bars for figure IE are smaller than symbol size, figures on mutual information rate density and gain included a comparison theory as check of error
(d) Did you include the total amount of compute and the type of resources used (e.g.. type of GPUs, internal cluster, or cloud provider)? [Yes] See appendix
4. If you are using existing assets (c.g., code, data, models) or curating/relcasing new assets...
(a) your work uses existing assets, did you cite the creators? [N/A] Does not apply
(b) Did you mention the license of the assets? [N/A] Does apply
(c) Did you include any assets either in the supplemental material or URL? [N/A] Does not
apply
(d) Did you discuss whether and how consent obtained from people whose data you're ing/curating? [N/A] Does not apply
(e) Did you discuss whether the data you using/curating contains personally identifiable informa- tion or offensive content? [N/A] Does not apply
5. If you used crowdsourcing or conducted research with human subjects...
(a) you include full text of instructions given to participants and screenshots, applicablle? [N/A] Does not apply
(b) Did you descrihe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A] Does not apply
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A] Does not apply
14


Reviews:
[{"review_id": "whrXBiCIrZ2", "reviewer": "Reviewer_5Aa7", "summary": "The authors develop an extension of the Dynamic Mean Field Theory to non-stationary inputs. They then use this toolkit to study how the tight balance of excitation and inhibition in recurrently connected networks supports information encoding of time-varying inputs to the network. Specifically, they find that the more tightly balanced networks are, the more they can \"track\" high-frequency fluctuations in the input through an effective reduction in the time constant of integration. They also train (with BPTT) a vanilla RNN to \"copy\" the input signal and validate their theoretical results.\n", "questions": "Please refer to the critiques listed above!", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The authors report a technical extension of the Dynamical Mean Field Theory to non-stationary inputs and apply that to a \"copy\" task. The notion of preserving/maintaining external inputs to the system (amidst noise) is certainly of interest to the neuroscience community. More relevant questions perhaps include the ability of RNNs to \"keep track of time,\" which the authors can potentially tackle with their method. It also seems like certain metrics like the approximate mutual information are easier to calculate in their framework.\n\nHowever, my main critique of this paper is its inadequacy of empirical evaluation and scope. The \"copy\" task is quite basic, and it seems overkill to develop and test theoretical scaffolding. This is, in fact, confirmed by the ability of out-of-the-box RNNs to learn this without sophisticated training techniques/architectural choices. Also, since the authors care about neuroscientific relevance, it is worth pointing out that in most cases neural circuits carry out computations of interest beyond simple transmission. If the authors believe that the \"copy\" task is indeed deceptively hard, can they please offer an explanation? Recommending this paper for publication requires at least one (or two) non-trivial demonstrations.\n\nThere are some other questions remaining about the manuscript which I have tried to detail below.\n\na. It is mildly confusing that the time-varying input signal x(t) (L77) never features in the definition of I(t). Can the authors clarify?\n\nb. L88 \"$b$ is a parameter that multiplies both external input and recurrent coupling strength and thus regulates the tightness of balance\". As per Eq. 1 and L79, this isn't the case. Though the authors present a modified formulation for $J_{ij}$ in L92. Can the authors clarify?\n\nc. L29 \"where synaptic interactions scale O(sqrt(N)) with network size N\". Can the authors elucidate more?\n\nd. L73 \"Our findings have important implications for ...\". It would be quite helpful to have a contributions section and lay this and other contributions of this paper in the Introduction. Determining this submission's uniqueness/technical novelty is certainly challenging and can be partially alleviated through better writing principles.\n\ne. L96 \"by rewriting Eq 5a as\". This is referenced before it is defined. Breaks the reader's flow. Kindly reformulate.\n\nf. In Fig. 3, it would also be good to show the eigenvalue spectrum as a function of training epochs. Additionally, the authors state (L268) that their theory only describes the behavior of very large networks, yet the RNN they train is small (N = 100). \n\ng. Generally, the paper can also do with a thorough proofread and grammar check. I've only highlighted a few of these below:\n\nL69 \"to reliable transmitting a\"\n\nL259 \"other biophysically features\"\n\nL294 \"biophysical more detailed\"\n\nL132 \"how in linear response a\"\n\nL160 \"alphabet [26]. .\"\n\nL196 \"neural networks how how the\"\n\nL237 \"(Figure 3A.\"\n\nL226 \"g = sqrt2\"\n\nL238 \"(Figure 3B.\"\n\nL241 \"(Figure 3C\"\n\nL255 \", e.i.,\"", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "kTN3d55WFX", "reviewer": "Reviewer_bJkp", "summary": "The authors propose a non-stationary theory for recurrent neural network equations. In their framework, strong recurrent inhibition balances the inputs which in turn improves the mutual information rate between the inputs and ouputs of the network.\n\nThe contributions are the following : \n- a theory that combine recurrent neural network equation, mean-field approximation and information theory,\n- the relation between network balance and mutual information,\n- numerical simulation accompanying the theory,\n- additional simulation showing that training such a network on simulated date leads to more and more tightly balanced networks.\n\n    ", "questions": "- The theoretical results are hard to relate to the numerical results. The theoretical functions displayed are not explicit in the text eg what is the formula relating the mutual information I and the balance b.\n\n- Equation (10) has been proved recently in \nKomaee, A. (2020). Mutual information rate between stationary Gaussian processes. Results in Applied Mathematics, 7, 100107. \nIt holds for stationary Gaussian process. Any references supporting your claim that it holds for non-stationary process ?\n\n- l 246 : Even if you work in somehow more general context, there is not really anything surprising about the fact that minimizing the MSE increases the mutual information, see\nGuo, D., Shamai, S., & Verdú, S. (2005). Mutual information and minimum mean-square error in Gaussian channels. IEEE transactions on information theory, 51(4), 1261-1282.\n\n- l 315 : As such, the manuscript is not convincing (I am not saying that it is wrong and I encourage the authors to continue their effort), why do you think your result holds under more general settings ?\n\nFigure 1 A,B,C and Figure 3 D are not referred to in the main text.\nFigure texts are too small.\n\nNotations :\n- coherence until section 3 then x and y are introduced as some mute variables introducing extra burden. I would advise to use x and y along the manuscript or I and \\nu ...\n\nTypos :\n- l 196 => remove \"how\"\n- l 207 => \"I(t) is grows by b\" ?\n- l 225 => \"e.i.\"\n- l 226 => \"sqrt2\"\n- l 232 => remove \"mean squared loss\"\n- l 233 => y = \\hat x ??\n- l 252 => \"e.i.\"\n\n", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths : \n- the combination of multiple theories is appealing,\n- the paper combine theory and simulation and also the relation with classical NN training,\n\nWeaknesses :\n- The mathematical notation are not coherent along the manuscript, impairing the clearest understanding,\n- The structure of the manuscript conflates theory and numerical results,\n- Many coarse typos (sometimes impairing understanding),\n- Unsupported mathematical claims.\n ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Ztw0KUy53x1", "reviewer": "Reviewer_HmY8", "summary": "This paper investigates the encoding of temporally-varying signals by neuronal populations using theory, simulation, and neural network approaches.   The main analytical findings are: (1)  non-stationary dynamic mean-field theory shows that a tighter balance between excitation and inhibition in the recurrent network allows better encoding of higher-frequency signals; (2) information-theoretic analysis shows that tight E-I balance maximizes the mutual information between the input and output. These analytical results are confirmed by simulations. Furthermore, the authors show that an autoencoder network trained to encode temporarily-varying signals exhibit strong local inhibition, indicating tight E-I balance. Thus, this work illuminates the potential functional advantages of local recurrent inhibition and tight E-I balance in the encoding of temporally-varying signals.", "questions": "Issues discussed in the weaknesses should be addressed. ", "limitations": "The authors addressed the limitations of the approach but it would be helpful if they also address the concern about whether the implications of this work can be generalized to networks that perform a meaningful computation. There is no negative societal impact.  ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nThe work is substantial and addresses the stated problem with multiple approaches from multiple perspectives. The analytical contribution and the autoencoder are novel.  The paper also provides intuitive insights as to why and how tight E-I balance in such a network might enhance information encoding and transmission.\n\nWeaknesses:\nThe studied network primarily uses a neuronal population to encode the input time-varying signals rather than performing any meaningful information processing. It is not clear what representations those neurons assume. That is,  all those neurons appear to serve the simple function of a simple electrical wire, connecting the input and the output!   The only obvious function that the network performs seems to remove the noise and guard against the chaos caused by the imperfection of the neurons. Obviously, this cannot be the case.   Would the principle discussed in this paper generalize to  a network that actually performs some meaningful computation? ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "N6o6fqqGmml", "reviewer": "Reviewer_H6Fq", "summary": "This paper proposes a non-stationary dynamic mean-field theory to relate the tightness of the balance of a network to the quality of information encoding. More particularly, it shows that a tighter balance improves the mutual information rate. The theoretical analysis is backed up with numerical experiments. Finally, the paper shows that learning the input weights, output weights and recurrent weights of a network whose goal is to auto-encode its one-dimensional input increases the mean recurrent coupling of the network. A generalization to multi-dimensional inputs shows that the network breaks into subnetworks with intense local inhibitory connections and weak between-clusters connections.", "questions": "1. Could the context of \"depth of balance\" / \"empirical balance\" be used in other studies?\n2. What is the difference between a binary network and a spiking network?\n3. I suggest moving Fig 6 to the main text: it is a shame that you talk about it only in the abstract and in the supplementary material...\n4. It would be interesting to consider unsupervised learning rules to learn the input/output/recurrent weights, instead of supervised learning\n5. Section 8: Couldn't you have checked the finite-size effects through simulations? Or did you instead mean that the finite-size effects should be inspected analytically?\n6. Provide a supplementary figure for the claims in the last paragraph of the discussion\n\n----------------------------------\nMinor issues (e.g. typos):\n1. Eq 5: \"m\" should write \"m/b\" on the right-hand side\n2. End of section 3: \"in contrast to previous work [21]\": precise that [21] has no noise\n3. Problem in the definition of the dynamic gain G(f)\n4. End of section 5: \"transmitted more reliable\" --> \"reliably\"\n5. First sentence of section 6: formulation is wrong after \"we analyse\"\n6. Beginning of section 7: rather use present tense (\"corroborate\")\n7. Use \"i.e.\" instead of \"e.i.\" (appearing twice in the text)\n8. Missing parentheses in the paragraph referring to Fig 3a/b/c\n9. First paragraph of section 8: missing space before \"the synaptic dynamics\"\n10. Appendix G: \"we trained RNN are trained\" (mistake)\n11. Appendix G: definition of the mean-squared loss: missing sum on index i? Shouldn't it be \\hat{x}_i rather than \\hat{x}?\n12. Appendix G: \"bith\" = mistake\n13. Fig 6D/E: what is the color code orange/blue: before versus after training, or cluster 1 versus 2?\n14. Fix typos in the legend of Fig 8 (\"bottom right:)\" / missing dot before \"Model\" / etc.)\n15. Checklist: answer N/A would be more appropriate than YES for questions that don't apply", "limitations": "Limitations: The authors properly provide the limitations of their work\n\nSocietal impact: The authors don't discuss the potential negative societal impact of the work, which is fine given the nature of the topic", "rating": 7, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths (from most important to least important):\n1. The paper provides a significant theoretical advancement to the literature on balanced networks, and suits the NeurIPS conference well\n2. The work is very well related to the existing literature\n3. The setup of the paper generalizes well to other transfer functions\n4. Extremely pedagogical paper, especially when it comes to explaining each mathematical variable and the mathematical derivations (even too much pedagogical in section 3? You could probably move some steps to the supplementary material if you want to \"gain\" some space)\n5. Numerical simulations are detailed and parameters are provided, which favors reproducibility\n6. Interesting ideas are provided for follow-up work (low-rank networks / taking into account energetic and biophysical constraints)\n\nWeaknesses (from most important to least important):\n1. The \"depth of balance\" / \"empirical balance\" is an interesting notion (which seems new), but I would suggest using another notation than b. It is not clear what motivates the definition of empirical balance as mean recurrent coupling, and its link with the balance parameter b. Without a strong connection between the two notions, it is wrong to say that the higher mutual information after training was predicted by the theory, or that the network achieves tight balance after training. In the simulation, the parameter b is fixed at value 1...\n2. The final part of the abstract about the network architecture is deceptive. It is expected that a balanced network (therefore dominated by inhibition) solving independent tasks will naturally form clusters, with mostly inhibitory intra-cluster connections and mixed weak inter-cluster connections (ideally, the two populations should probably not communicate here). What is observed in this simple context of independent tasks does not imply that \"feedforward excitatory projections and local recurrent inhibition is a generic circuit motif for encoding and transmitting information\". Additionally, there doesn't seem to be a preference for excitatory connections between clusters in Fig 6C and 6F\n3. The theory lacks experimental predictions (one that comes to mind is the auditory cortex which encodes for different frequencies) and relation to existing experimental work\n4. No code provided. Providing the code would help for even easier reproducibility. Note that anonymized links can be provided at the time of submission, by using an anonymized GitHub account for instance; see ICML guidelines: https://icml.cc/FAQ/submitted-code-anonymized / https://icml.cc/FAQ/github-links / https://icml.cc/Conferences/2022/StyleAuthorInstructions", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept