SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The Gini index ranges from 0 to 1. When only a few neurons have very high activation values and others are small, the Gini index is close to 1; on the other hand, when most neurons have homogeneous activation values, this value is close to 0 (1 = absolute sparsity, 0 = all activations equal).
According to Fig. S13a, when the value of  $\alpha$  in  $\mathrm{NMDA}_{\alpha}(x)$  is increased, the Gini index also increases. This increase signifies that the population activities become more heterogeneous, leading to a heavy-tailed distribution. Furthermore, the Gini index of NMDA with an  $\alpha$  value of 10 is greater than that of the GELU activation function (indicated by the dashed line). These findings imply that  $\mathrm{NMDA}_{\alpha}(x)$  could potentially improve long-term memory formation by promoting sparsity in the activations of the feed-forward layer.
(b)
Figure S13: (a) Comparison of Gini index for different values of  $\alpha$  in  $\mathrm{NMDA}_{\alpha}$ . (b) Comparison of Gini index between different nonlinear activation functions.
Previous studies have explored the generalization performance and sparsity in overparameterized models [S40]. Although the mechanism behind the emergence of sparse representation in large models is not fully understood, it is worth noting that overparameterized models converge to simpler models than those with dense representations.
The high sparsity properties of the NMDA function may contribute to the increased score of place cells. In the case of Transformers, enforcing neuron activation sparsity in MLPs has been found to improve the interpretability or selectivity of a higher percentage of neurons [S41]. This evidence could potentially explain the higher place cell score observed when the alpha value is increased. We observed that NMDA with  $\alpha = 10$  exhibits the highest Gini index among different activation functions. See Fig. S13a for a comparison of Gini index among activation functions.
[S1] John O'Keefe. Place units in the hippocampus of the freely moving rat. Experimental neurology, 51(1):78-109, 1976.
[S2] Matthew A Wilson and Bruce L McNaughton. Dynamics of the hippocampal ensemble code for space. Science, 261(5124):1055-1058, 1993.
[S3] John O'keefe and Lynn Nadel. Precis of o'keefe & nadel's the hippocampus as a cognitive map. Behavioral and Brain Sciences, 2(4):487-494, 1979.
[S4] Howard Eichenbaum, Paul Dudchenko, Emma Wood, Matthew Shapiro, and Heikki Tanila. The hippocampus, memory, and place cells: is it spatial memory or a memory space? Neuron, 23(2):209-226, 1999.
[S5] Christopher J. Cueva and Xue-Xin Wei. Emergence of grid-like representations by training recurrent neural networks to perform spatial localization. In International Conference on Learning Representations, 2018. URL https://openreview.net/forum?id=B17JT0e0-.
[S6] Andrea Banino, Caswell Barry, Benigno Uria, Charles Blundell, Timothy Lillicrap, Piotr Mirowski, Alexander Pritzel, Martin J Chadwick, Thomas Degris, Joseph Modayil, et al. Vector-based navigation using grid-like representations in artificial agents. Nature, 557(7705): 429-433, 2018.
[S7] James CR Whittington, Timothy H Muller, Shirley Mark, Guifen Chen, Caswell Barry, Neil Burgess, and Timothy EJ Behrens. The tolman-eichenbaum machine: unifying space and relational memory through generalization in the hippocampal formation. Cell, 183(5):1249-1263, 2020.
[S8] James C. R. Whittington, Joseph Warren, and Tim E.J. Behrens. Relating transformers to models and neural representations of the hippocampal formation. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=B8DVo9B1YE0.