Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The authors develop an extension of the Dynamic Mean Field Theory to non-stationary inputs. They then use this toolkit to study how the tight balance of excitation and inhibition in recurrently connected networks supports information encoding of time-varying inputs to the network. Specifically, they find that the more tightly balanced networks are, the more they can "track" high-frequency fluctuations in the input through an effective reduction in the time constant of integration. They also train (with BPTT) a vanilla RNN to "copy" the input signal and validate their theoretical results.

strengths_and_weaknesses: The authors report a technical extension of the Dynamical Mean Field Theory to non-stationary inputs and apply that to a "copy" task. The notion of preserving/maintaining external inputs to the system (amidst noise) is certainly of interest to the neuroscience community. More relevant questions perhaps include the ability of RNNs to "keep track of time," which the authors can potentially tackle with their method. It also seems like certain metrics like the approximate mutual information are easier to calculate in their framework.

However, my main critique of this paper is its inadequacy of empirical evaluation and scope. The "copy" task is quite basic, and it seems overkill to develop and test theoretical scaffolding. This is, in fact, confirmed by the ability of out-of-the-box RNNs to learn this without sophisticated training techniques/architectural choices. Also, since the authors care about neuroscientific relevance, it is worth pointing out that in most cases neural circuits carry out computations of interest beyond simple transmission. If the authors believe that the "copy" task is indeed deceptively hard, can they please offer an explanation? Recommending this paper for publication requires at least one (or two) non-trivial demonstrations.

There are some other questions remaining about the manuscript which I have tried to detail below.

a. It is mildly confusing that the time-varying input signal x(t) (L77) never features in the definition of I(t). Can the authors clarify?

b. L88 "$b$ is a parameter that multiplies both external input and recurrent coupling strength and thus regulates the tightness of balance". As per Eq. 1 and L79, this isn't the case. Though the authors present a modified formulation for $J_{ij}$ in L92. Can the authors clarify?

c. L29 "where synaptic interactions scale O(sqrt(N)) with network size N". Can the authors elucidate more?

d. L73 "Our findings have important implications for ...". It would be quite helpful to have a contributions section and lay this and other contributions of this paper in the Introduction. Determining this submission's uniqueness/technical novelty is certainly challenging and can be partially alleviated through better writing principles.

e. L96 "by rewriting Eq 5a as". This is referenced before it is defined. Breaks the reader's flow. Kindly reformulate.

f. In Fig. 3, it would also be good to show the eigenvalue spectrum as a function of training epochs. Additionally, the authors state (L268) that their theory only describes the behavior of very large networks, yet the RNN they train is small (N = 100). 

g. Generally, the paper can also do with a thorough proofread and grammar check. I've only highlighted a few of these below:

L69 "to reliable transmitting a"

L259 "other biophysically features"

L294 "biophysical more detailed"

L132 "how in linear response a"

L160 "alphabet [26]. ."

L196 "neural networks how how the"

L237 "(Figure 3A."

L226 "g = sqrt2"

L238 "(Figure 3B."

L241 "(Figure 3C"

L255 ", e.i.,"
questions: Please refer to the critiques listed above!
limitations: N/A
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The authors propose a non-stationary theory for recurrent neural network equations. In their framework, strong recurrent inhibition balances the inputs which in turn improves the mutual information rate between the inputs and ouputs of the network.

The contributions are the following : 
- a theory that combine recurrent neural network equation, mean-field approximation and information theory,
- the relation between network balance and mutual information,
- numerical simulation accompanying the theory,
- additional simulation showing that training such a network on simulated date leads to more and more tightly balanced networks.

    
strengths_and_weaknesses: Strengths : 
- the combination of multiple theories is appealing,
- the paper combine theory and simulation and also the relation with classical NN training,

Weaknesses :
- The mathematical notation are not coherent along the manuscript, impairing the clearest understanding,
- The structure of the manuscript conflates theory and numerical results,
- Many coarse typos (sometimes impairing understanding),
- Unsupported mathematical claims.
 
questions: - The theoretical results are hard to relate to the numerical results. The theoretical functions displayed are not explicit in the text eg what is the formula relating the mutual information I and the balance b.

- Equation (10) has been proved recently in 
Komaee, A. (2020). Mutual information rate between stationary Gaussian processes. Results in Applied Mathematics, 7, 100107. 
It holds for stationary Gaussian process. Any references supporting your claim that it holds for non-stationary process ?

- l 246 : Even if you work in somehow more general context, there is not really anything surprising about the fact that minimizing the MSE increases the mutual information, see
Guo, D., Shamai, S., & Verdú, S. (2005). Mutual information and minimum mean-square error in Gaussian channels. IEEE transactions on information theory, 51(4), 1261-1282.

- l 315 : As such, the manuscript is not convincing (I am not saying that it is wrong and I encourage the authors to continue their effort), why do you think your result holds under more general settings ?

Figure 1 A,B,C and Figure 3 D are not referred to in the main text.
Figure texts are too small.

Notations :
- coherence until section 3 then x and y are introduced as some mute variables introducing extra burden. I would advise to use x and y along the manuscript or I and \nu ...

Typos :
- l 196 => remove "how"
- l 207 => "I(t) is grows by b" ?
- l 225 => "e.i."
- l 226 => "sqrt2"
- l 232 => remove "mean squared loss"
- l 233 => y = \hat x ??
- l 252 => "e.i."


limitations: yes
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper investigates the encoding of temporally-varying signals by neuronal populations using theory, simulation, and neural network approaches.   The main analytical findings are: (1)  non-stationary dynamic mean-field theory shows that a tighter balance between excitation and inhibition in the recurrent network allows better encoding of higher-frequency signals; (2) information-theoretic analysis shows that tight E-I balance maximizes the mutual information between the input and output. These analytical results are confirmed by simulations. Furthermore, the authors show that an autoencoder network trained to encode temporarily-varying signals exhibit strong local inhibition, indicating tight E-I balance. Thus, this work illuminates the potential functional advantages of local recurrent inhibition and tight E-I balance in the encoding of temporally-varying signals.
strengths_and_weaknesses: Strengths:
The work is substantial and addresses the stated problem with multiple approaches from multiple perspectives. The analytical contribution and the autoencoder are novel.  The paper also provides intuitive insights as to why and how tight E-I balance in such a network might enhance information encoding and transmission.

Weaknesses:
The studied network primarily uses a neuronal population to encode the input time-varying signals rather than performing any meaningful information processing. It is not clear what representations those neurons assume. That is,  all those neurons appear to serve the simple function of a simple electrical wire, connecting the input and the output!   The only obvious function that the network performs seems to remove the noise and guard against the chaos caused by the imperfection of the neurons. Obviously, this cannot be the case.   Would the principle discussed in this paper generalize to  a network that actually performs some meaningful computation? 
questions: Issues discussed in the weaknesses should be addressed. 
limitations: The authors addressed the limitations of the approach but it would be helpful if they also address the concern about whether the implications of this work can be generalized to networks that perform a meaningful computation. There is no negative societal impact.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a non-stationary dynamic mean-field theory to relate the tightness of the balance of a network to the quality of information encoding. More particularly, it shows that a tighter balance improves the mutual information rate. The theoretical analysis is backed up with numerical experiments. Finally, the paper shows that learning the input weights, output weights and recurrent weights of a network whose goal is to auto-encode its one-dimensional input increases the mean recurrent coupling of the network. A generalization to multi-dimensional inputs shows that the network breaks into subnetworks with intense local inhibitory connections and weak between-clusters connections.
strengths_and_weaknesses: Strengths (from most important to least important):
1. The paper provides a significant theoretical advancement to the literature on balanced networks, and suits the NeurIPS conference well
2. The work is very well related to the existing literature
3. The setup of the paper generalizes well to other transfer functions
4. Extremely pedagogical paper, especially when it comes to explaining each mathematical variable and the mathematical derivations (even too much pedagogical in section 3? You could probably move some steps to the supplementary material if you want to "gain" some space)
5. Numerical simulations are detailed and parameters are provided, which favors reproducibility
6. Interesting ideas are provided for follow-up work (low-rank networks / taking into account energetic and biophysical constraints)

Weaknesses (from most important to least important):
1. The "depth of balance" / "empirical balance" is an interesting notion (which seems new), but I would suggest using another notation than b. It is not clear what motivates the definition of empirical balance as mean recurrent coupling, and its link with the balance parameter b. Without a strong connection between the two notions, it is wrong to say that the higher mutual information after training was predicted by the theory, or that the network achieves tight balance after training. In the simulation, the parameter b is fixed at value 1...
2. The final part of the abstract about the network architecture is deceptive. It is expected that a balanced network (therefore dominated by inhibition) solving independent tasks will naturally form clusters, with mostly inhibitory intra-cluster connections and mixed weak inter-cluster connections (ideally, the two populations should probably not communicate here). What is observed in this simple context of independent tasks does not imply that "feedforward excitatory projections and local recurrent inhibition is a generic circuit motif for encoding and transmitting information". Additionally, there doesn't seem to be a preference for excitatory connections between clusters in Fig 6C and 6F
3. The theory lacks experimental predictions (one that comes to mind is the auditory cortex which encodes for different frequencies) and relation to existing experimental work
4. No code provided. Providing the code would help for even easier reproducibility. Note that anonymized links can be provided at the time of submission, by using an anonymized GitHub account for instance; see ICML guidelines: https://icml.cc/FAQ/submitted-code-anonymized / https://icml.cc/FAQ/github-links / https://icml.cc/Conferences/2022/StyleAuthorInstructions
questions: 1. Could the context of "depth of balance" / "empirical balance" be used in other studies?
2. What is the difference between a binary network and a spiking network?
3. I suggest moving Fig 6 to the main text: it is a shame that you talk about it only in the abstract and in the supplementary material...
4. It would be interesting to consider unsupervised learning rules to learn the input/output/recurrent weights, instead of supervised learning
5. Section 8: Couldn't you have checked the finite-size effects through simulations? Or did you instead mean that the finite-size effects should be inspected analytically?
6. Provide a supplementary figure for the claims in the last paragraph of the discussion

----------------------------------
Minor issues (e.g. typos):
1. Eq 5: "m" should write "m/b" on the right-hand side
2. End of section 3: "in contrast to previous work [21]": precise that [21] has no noise
3. Problem in the definition of the dynamic gain G(f)
4. End of section 5: "transmitted more reliable" --> "reliably"
5. First sentence of section 6: formulation is wrong after "we analyse"
6. Beginning of section 7: rather use present tense ("corroborate")
7. Use "i.e." instead of "e.i." (appearing twice in the text)
8. Missing parentheses in the paragraph referring to Fig 3a/b/c
9. First paragraph of section 8: missing space before "the synaptic dynamics"
10. Appendix G: "we trained RNN are trained" (mistake)
11. Appendix G: definition of the mean-squared loss: missing sum on index i? Shouldn't it be \hat{x}_i rather than \hat{x}?
12. Appendix G: "bith" = mistake
13. Fig 6D/E: what is the color code orange/blue: before versus after training, or cluster 1 versus 2?
14. Fix typos in the legend of Fig 8 ("bottom right:)" / missing dot before "Model" / etc.)
15. Checklist: answer N/A would be more appropriate than YES for questions that don't apply
limitations: Limitations: The authors properly provide the limitations of their work

Societal impact: The authors don't discuss the potential negative societal impact of the work, which is fine given the nature of the topic
review_title: null

Meta-review: metareview: In spite of a somewhat weak empirical evaluation, reviewers appreciate the novel combination of multiple theories on recurrent networks.
confidence: Less certain
Decision: Accept