Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 9: Top 15% of accepted NeurIPS papers, strong accept
review_content: summary: The authors study generalized, nonlinear Hebbian learning rules, and find that neurons with such plasticity and feedforward inputs can learn tensor decompositions of higher-order correlations.  They further demonstrate that for simple, biologically plausible scenarios, the tensor eigenvectors are attractors. After determining the basins of attraction, the authors show that the basin of the dominant eigenvector has the largest volume.  Finally, they show that any plasticity rule with a finite Taylor expansion in synaptic input, neural output, and weight variables has equilibrium states  that are similar to those already found and describe higher-order input correlations.
main_review: The authors have extended a large body of prior work on local linear learning rules. These Hebbian (and generalized Hebbian) rules have been shown to allow neurons to model input statistics; for example, Oja's rule has been shown to be stable and instantiate an online algorithm for PCA.  The thorough analysis encapsulated in this manuscript represents a welcome advance in this classical unsupervised learning framework.
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors analyze a broad class of nonlinear Hebbian learning rules. These rules generalize the well-studied Oja's learning rule to higher-order input correlations, which can be represented in symmetric higher-order tensors of synaptic input statistics. There is a broad modeling literature on synaptic plasticity, and it is not my primary area of research. Nonetheless, this generalization of Oja's rule appears novel to me, and the extensions outlined in this manuscript feel very natural and interesting.
main_review: The authors analyze a broad class of nonlinear Hebbian learning rules. These rules generalize the well-studied Oja's learning rule to higher-order input correlations, which can be represented in symmetric higher-order tensors of synaptic input statistics. There is a broad modeling literature on synaptic plasticity, and it is not my primary area of research. Nonetheless, this generalization of Oja's rule appears novel to me, and the extensions outlined in this manuscript feel very natural and interesting.

The authors point out that "any plasticity model with a finite Taylor polynomial" can be seen as a special case of their framework. However, there are few insights into this general case, and most of their analysis is confined to simpler cases that are analytically tractable. These special cases are still interesting.

In the classic case of Oja's rule, the largest eigenvector is the unique/global attractor of the system. However, this is not the case for the generalized model considered here. Thus, the authors' results are somewhat pessimistic because the learning dynamics will reach different steady-states depending on the initialization. This makes it more difficult to interpret the functional purpose or outcome of the learning rule, unless one is able to make additional assumptions about the initial state of the system.

On the other hand, it appears that the dynamics do converge, in practice, to the top few eigenvectors when initialized randomly (see fig. 1h). This at least holds in the special cases the authors examined empirically. One of the more useful results in the paper is a proof on the relative sizes of the basins of attraction, which provides good intuition.

One unanswered question I have is whether there are reasons to think the top eigenvectors of higher-order input statistics would be useful features to learn? In Oja's rule, the synaptic weights converge to the top principal component, and thus the neuron becomes tuned to detect salient (i.e. high-variance) features. The interpretation of feature-detectors for higher-order statistics is less intuitive to me, though I know there has been some work done on this -- e.g., https://doi.org/10.1167/13.9.974

Overall, I think these results are very thorough and interesting, if still a bit abstract and general. If the authors could provide readers with more intuition about how higher-order features may be useful features for downstream neural computation, I think it would broaden the appeal and impact of the paper. Either way, I hope to see this paper accepted.

Other comments:

- I attempted to verify all derivations and statements in the supplemental material and I could not find any errors.

- Do the dynamics in fig. 1e actually converge? The trajectories look very volatile.

- The fact that the learning rule does not always converge to the largest eigenvector is not suprising, since recovering the top eigenvector is NP-hard even for symmetric tensors (Hillar & Lim, 2013).

- Notation is somewhat confusing and inconsistent in places. There are three different typefaces for the variable "x" in equations 1 and 2. The variable \mu_{i, \alpha} is boldfaced in equation 3, but is not boldfaced in equation 2. It would be helpful if the explanation that \alpha is a multi-index could be moved a few sentences earlier, right after equation 2.

- On lines 90-91, it is claimed that the first k E-eigenvectors define the best rank-k approximation approximation of \mu in terms of a least-squares criterion. Typically a rank-k approximation is understood to mean a CP-decomposition of the tensor (see Kolda & Bader, 2009), but it seems like here it is meant to be the best rank-k Tucker decomposition to approximate the tensor? This is the sense I got from reading the citation to De Lathauwer (2000), which distriguishes between a rank-R and a rank-(R_1, R_2, ..., R_n) approximation to a tensor. Can the authors clarify these points more explicitly?

- In the caption of Figure 2, a small typo appears: a "v2" should be "v_2"

- On line 553 of the supplement, a reference / citation is broken.
limitations_and_societal_impact: Limitations are adequately discussed and I foresee no negative societal impacts.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper shows that nonlinear Hebbian rules allow a neuron to learn tensor decompositions of the higher-order input correlation. There have been theoretical works on nonlinear Hebbian learning rules as well as triplet STDP BCM learning rules, but I believe this might be the first that showed systematically and mathematically it enables the learning of tensor eigenvectors and characterize their basins of attractions.  
main_review: While I understand that higher-order bit,  I found the paper too dense and hard to follow.  I found the math notations in the main text complex, perhaps unnecessarily complex,  and hard to follow.  I looked into the Appendix for clarification but found Eq. 1 in the main text and Eqn. 2 in the Appendix to be inconsistent -- was one of them wrong?  I also could not derive Eq. 3 from Eq. 2 in the Appendix. My puzzle is that even if expanding the norm of J of dt, the 1st order expansion still depends on the norm of vector J.  In contrast,  Eq. 3 in Appendix doesn't depend on the norm of  vector J, but only depend on each element J_j.  The definition of  \dot{J}_j(t) right after Eq. 4 in Appendix is also unclear -- what is the difference between J_i and the bold J_i on the right-hand side of the definition?   While they might be minor issues,  these issues in the first few steps made it sufficiently frustrating for me to press on to evaluate the details. In any case, it is unreasonable for the reviewers to plow through the 20 pages of math in the Appendix.

While I believe the work is probably solid and the contribution fundamental, I wonder if NeurIPS is the appropriate forum. Maybe Neural Computation or Plos Computational Neuroscience would be a better forum. It would be helpful to provide more explanations (e.g. explaining  \mu in Eq. 3 is the input correlation tensor ) and perhaps focus on one or two simple concrete examples to illustrate the results and show what higher-order correlations the neurons actually learn in those examples.  Perhaps something like the long contours in Lawlor and Zucker's mixture model, or perhaps corners or curvatures as in Olshausen's overcomplete sparse coding?    Perhaps they are trying to do this with Figure 1c? But more explanations are necessary.  I don't enjoy the presentation of the paper,  but I am willing to give it a high score because I do think the contribution is potentially important and am willing to give it the benefit of the doubt for now. 
limitations_and_societal_impact: Broader impact section missing.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This work proposes to determine the solutions of "generalized" Hebbian rules, i.e., applied to tensor matrices, and analyze the stability of these solutions. The model starts by postulating a biologically sensible form of nonlinear Hebbian rule as polynomials of local input activity, postsynaptic activity, and current synaptic weight. From this generalized form of Hebbian rule, the authors can determine the nature of the steady-states of the dynamic and relate it to the spectrum and eigenspace of the higher-order covariance matrix. Furthermore, the size of the basins of attraction is determined. The model is then used on toy experiments proving the convergence of the learning rules for specifically chosen parameters of the learning rule. 
main_review: First is addressed the organization and clarity of the paper. 

The paper is well-written, and results and theorems are nicely explained using informal explanations and leaving the full proof to the appendix. Unfortunately, the paper starts with a "Result" section without introducing the necessary background and existing rules and contextualizing the problem. Related work can certainly appear at the end, but I believe various things should have been introduced early on. 

For example:
1) What are the limitations of existing Hebbian rules, and why do we need higher-order power? And what is the difference with nonlinear inputs/outputs?

2) The point above would have naturally lead to the introduction of higher-order covariance matrices, which are naturally formulated as tensors. As a result, one can regret the mention of prevalent tasks that is tensor PCA or independent component analysis (very briefly mentioned by the authors). For example, the work of Cichocki, most recently in [i], presents examples where tensor decomposition is essential. The author of [i] is one of many interested in biological implementation for third and fourth-order tensors and has been omitted to be cited. For example, the work of [ii] proposes similar learning rules in the case of ICA and discusses the possible biological implementation using higher-order neurons. 

3) Tensor decomposition is also a very active field of research in the machine learning community, and one can regret that "gradient-based" methods (which Oja is one of) have not been discussed. 

The experiments are following each other as they happened chronologically, following the authors' discoveries as they happened rather than proposing an overarching methodology. It can be seen when using "We next investigated" "We next asked" followed by another "We next investigated" in section 2.3. Having a separate methodology section could help the reader understand what the paper aims to achieve. But it still important to point out that paragraphs taken separately are nicely written and clear. 

Now is addressed the originality of the paper. 

The main originality of the paper is the analysis of the solution of the ad-hoc learning rule. It cannot be said that such learning has never been proposed nor analyzed in other contexts cf reference given above. Hebbian/Oja's rules have long been known to be gradient-based methods of "Extended covariance" matrices, so this result is not very surprising, although important to be theoretically proved. Unfortunately, it is also true that how these rules can be biologically implemented is heavily discussed and shown to require recurrent connections or to operate on various timescales when considering more than one output neuron. These facts are largely overlooked, which greatly weakens the paper as its relevance can be questioned from the start if the main assumptions of the paper are not properly cleared out. 
The authors claim to address phenomenological rules, but no results seem to validate or invalidate known plasticity rules. One could have preferred fewer but clearer examples that would relate this work to neuroscientific facts, rather than speculations of what is and what is not biologically plausible. In the case of the spiking neuron (Sec 3.1), this work should have related to the paper from [iv] which would have strengthened the paper as it offers a nice theoretical framework for it.  

Now is addressed the significance of the paper. 

The paper appears to propose an interesting analysis of the solution of an ad-hoc learning rule. Unfortunately, this work does not make predictions on how (implementation-wise) and where (structurally) they could occur in the brain. There have also been various higher-order neuron models that have been used c.f. [ii] in concrete tasks, when the experiments are not really relatable and do not add much to the paper. One could have expected examples where tensor decomposition has been used either in neuroscience or machine learning. 



[i] Cichocki, Andrzej. "Tensor decompositions: a new concept in brain data analysis?." arXiv preprint arXiv:1305.0395 (2013).

[ii] Ziegaus, Ch, and Elmar Wolfgang Lang. "A neural implementation of the JADE algorithm (nJADE) using higher-order neurons." Neurocomputing 56 (2004): 79-100.

[iii] Ge, Rong, et al. "Escaping from saddle points—online stochastic gradient for tensor decomposition." Conference on learning theory. PMLR, 2015.

[iv] Gjorgjieva, Julijana, et al. "A triplet spike-timing–dependent plasticity model generalizes the Bienenstock–Cooper–Munro rule to higher-order spatiotemporal correlations." Proceedings of the National Academy of Sciences 108.48 (2011): 19383-19388.

limitations_and_societal_impact: The limitations of the paper are hard to find as the paper does not clearly contextualize in which fields it operates, as mentioned in the main review.
review_title: null

Meta-review: This paper shows how nonlinear Hebbian rules learn tensor decompositions of higher-order input correlations. This is an important contribution to the literature on Hebbian synaptic plasticity. Original submission missed important citations and related work, however the revised introduction proposed by the authors better frame the work and its contributions.
Decision: Accept (Poster)