SlowGuess/NeurIPS_2023_OCR huggingface dataset:
A salient feature of our circuit model is that the inhibition from the granule cells onto a mitral cell is gated by the activity of the mitral cell itself (3). This state-dependent inhibition is reminiscent of in vitro experiments showing that granule cell mediated inhibition is activity dependent [95]. In these experiments, Arevian et al. measured the activity of a primary mitral cell  $(MC_{A})$  while increasing its level of stimulation under two conditions. In the first condition, no other cells in the circuit are being stimulated. In the second condition, they also activate another mitral cell  $(MC_{B})$ . The activation of the second mitral cell leads to the activation of granule cells connected to both mitral cells and a reduction in the firing evoked by stimulation of the primary cell alone. Strikingly, these authors showed that this inhibition is dependent on the activity of the primary cell [95].
Here, we show that our proposed circuit reproduces these observations of state-dependent inhibition, and that they do not arise in a similarly-constructed circuit for a Gaussian noise model. To model Arevian et al. [95]'s in vitro experiments, we simulated a reduced circuit with 2 mitral cells and 10 granule cells. We stimulated  $MC_{A}$  with an input  $s_A \in [1:400]$  while toggling on or off the stimulation  $s_B = 80$  of the second mitral cell  $MC_{B}$ . As observed experimentally, when  $MC_{B}$  is activated, the inhibition of the primary mitral cell  $MC_{A}$  is state dependent (Fig. 2B). To show that this effect arises from our Poisson circuit, we build a similar inference circuit with a Gaussian noise model (Appendix D). Under similar conditions, the inhibition in the Gaussian circuit is independent of the activity of the primary mitral cell  $MC_{A}$  (Fig. 2B). Furthermore, the dynamics of the relative inhibition qualitatively recapitulate those observed experimentally, with sustained inhibition throughout the stimulation period at the stimulation level of  $MC_{A}$  leading to maximal inhibition and inhibition followed by relaxation for stronger stimulation levels of  $MC_{A}$  (Fig. 2C and see [95] Fig. 3b).
5 Geometry, speed, and capacity
We have argued that the model introduced in §3 could be implemented biologically, but can it perform at scale? Concretely, can a circuit of this architecture with neuron counts comparable to the human OB correctly identify which among a large set of odorants are present in a given scene? This is precisely the question of the capacity of the CS algorithm [98, 99]. In Fig. 3A, we present our algorithm with scenes composed of varying numbers of randomly-selected odorants out of a panel of 1000, which for simplicity we take to be at the same concentration. We first ask how many odorants can be reliably detected within a single sniff, i.e.,  $200\mathrm{ms}$ . To convert MAP concentration estimates into presence estimates, we simply binarize the estimated concentrations based on whether they are larger than half of the true odorant concentration. The ability of the one-to-one code to successfully detect odorants falls off rapidly, with the detection fraction falling below one-half even if only a handful of odorants are present (Fig. 3A).
The limited capacity of the one-to-one code can be overcome by distributing the code in a way that takes into account the geometry of the sensing problem. Here, the information geometry of the
<sup>4</sup>We remark that, with a one-to-one code, our model is identical to that proposed by Grabska-Barwińska et al. [21] except for the introduction of the granule cells.
C
Figure 3: Fast detection of many odorants. A. Fraction of odorants correctly detected within  $100\mathrm{ms}$  (left),  $200\mathrm{ms}$  (center), and  $1\mathrm{s}$  (right) after odorant onset as a function of the number of odorants present, for models with one-to-one, naively distributed, and geometry-aware codes. Here, we consider a repertoire of 1000 possible odorants. B. Heatmap with overlaid smoothed contours of correct detection fraction as a function of number of present odorants and time window for models with one-to-one (left), naively distributed (center), and geometry-aware (right) codes. As in A, a repertoire of 1000 possible odorants is used. C. Threshold number of odorants for which half can be correctly detected as a function of total repertoire size within  $100\mathrm{ms}$  (left),  $200\mathrm{ms}$  (center), and  $1\mathrm{s}$  (right) after odorant onset. See also Supp. Figs. G.1 and G.2 for versions of panels A and B with varying repertoire sizes, from which the capacities shown here are derived. See Appendix G for details of our numerical methods. Shaded patches show  $\pm 1.96$  SEM over realizations throughout.
problem is governed by the sensing matrix  $\mathbf{A}$ , which introduces correlations in the input signals to mitral cells because the off-diagonal components of  $\mathbf{A}^{\top}\mathbf{A}$  are non-negligible. To counteract these detrimental correlations, we can choose a readout matrix  $\Gamma$  such that  $\Gamma\Gamma^{\top} \approx (\mathbf{A}^{\top}\mathbf{A})^{+}$  up to constants of proportionality, where we must take a pseudoinverse because  $\mathbf{A}^{\top}\mathbf{A}$  is highly rank-deficient (see Appendix G for details). Geometrically, this corresponds an approximation of natural gradient descent, in which we use the Gauss-Newton matrix instead of the Fisher information matrix because the latter is state-dependent for Poisson likelihoods [37-40]. As a control, we also consider a naively distributed code with  $\Gamma\Gamma^{\top} \approx \mathbf{I}_{n_{\mathrm{odor}}}$ . Distributing the code, even without accounting for the geometry of inference, markedly improves the single-sniff capacity to around 10-20, and taking into account the geometry produces a further improvement (Fig. 3A).
To gain a more granular view of how tuned geometry enables faster inference, in Fig. 3B we test the three models' detection capabilities at sub-sniff resolution. We can see that at long times—around a second after odorant onset—the naively distributed and geometry-aware codes achieve similar capacities of around 50-60 odorants. However, the geometry-aware code reaches this detection capacity within a single sniff, whereas the naively distributed code requires the full second of
processing time. The detection capacity of the one-to-one code reaches only around 20 odorants after 1 second, and even then does not appear to have reached its asymptote. These results illustrate two important conceptual points: First, when the strength of individual synapses is bounded, distributed coding can speed up dynamics by increasing the effective total input to a given neuron. Second, given a sensing matrix that induces strong correlations, geometry-aware distributed coding can accelerate inference by counteracting that detrimental coupling.
These tests show that our model can reliably detect tens of odorants from a panel of thousands of possible odorants, but do not probe how the model's capacity scales to larger odor spaces. While the true dimensionality of odor space remains unknown [100, 101], there may be orders of magnitude more than thousands of possible odorants. As an upper bound, there are on the order of millions of known volatile compounds that are plausibly odorous [102]. Thus, it is important to determine how our model scales to more realistically-sized odor spaces. From the literature on compressed sensing performance bounds, we expect the threshold sparsity to decay slowly—roughly logarithmically—with increasing  $n_{\mathrm{odor}}$  [28-33]. As a first step, in Supp. Figs. G.1 and G.2, we reproduce Fig. 3A-B for between 500 and 8000 possible odorants, showing that performance does indeed drop off slowly with increasing odor space dimension. To get a more precise estimate of how performance scales with repertoire size, in Fig. 3C we plot the threshold number of odorants for which half can be reliably detected as a function of the repertoire size, showing that for the geometry-aware code it decays only a bit faster than logarithmically. Our ability to simulate larger systems was limited by computational resources. This limitation is present in previous works, and the repertoires tested here are comparable to—or substantially larger than—those used in past studies [21, 22, 24, 26, 72].
6 Fast sampling for uncertainty estimation
Thus far, we have focused on a circuit that performs MAP estimation of odorant concentrations. However, to successfully navigate a dynamic, noisy world, animals must estimate sensory uncertainty at the timescale of perception [38, 103-107]. Fortunately, our circuit model can be easily extended to perform Langevin sampling of the full posterior distribution, allowing for uncertainty estimation while maintaining its attractive structural features. In Appendix B, we provide a detailed derivation of a model that implements Langevin sampling through the granule cells for a Poisson likelihood and Gamma prior. This yields a circuit that is identical to the MAP estimation circuit introduced in §3 up to the addition of Gaussian noise to the granule cell dynamics:
$$
\mathbf {c} (t) = \boldsymbol {\Gamma} \mathbf {g} (t) \quad \tau_ {\mathrm {g}} \dot {\mathbf {g}} (t) = (\mathbf {A} \boldsymbol {\Gamma}) ^ {\top} (\mathbf {p} - \mathbf {1}) + \boldsymbol {\Gamma} ^ {\top} (\mathbf {z} - \boldsymbol {\lambda}) + \boldsymbol {\xi} (t)
$$
$$
\tau_ {\mathrm {p}} \dot {\mathbf {p}} (t) = \mathbf {s} - \mathbf {p} \odot (\mathbf {r} _ {0} + \mathbf {A} \boldsymbol {\Gamma} \mathbf {g}) \quad \tau_ {\mathrm {z}} \dot {\boldsymbol {z}} (t) = \boldsymbol {\alpha} - \mathbf {1} - \mathbf {z} \odot \mathbf {c}. \tag {4}
$$
Here,  $\xi(t)$  is a vector of  $n_{\mathrm{g}}$  independent zero-mean Gaussian noise processes with covariance  $\mathbb{E}[\xi_j(t)\xi_{j'}(t')] = 2\tau_{\mathrm{g}}\delta_{jj'}\delta(t - t')$ , and once again the rates  $\mathbf{g}$  and  $\mathbf{p}$  should in principle be constrained to be non-negative. In this case, the readout matrix  $\Gamma$  both preconditions the effective gradient force and shapes the structure of the effective noise  $\Gamma\xi$ , allowing us to mold the geometry of the sampling manifold [37, 40]. By using a projected readout, we maintain the independence of the noise processes for different neurons. This both allows us to have many independent samplers—if  $n_{\mathrm{g}} > n_{\mathrm{odor}}$  and is important for biological realism if we interpret the sampling noise as resulting from fluctuations in membrane potential due to synaptic noise [108].
As a test of how this sampling circuit performs, we consider a simple setup in which one set of odorants appears at a low concentration, and then a second set of odorants appear at a higher concentration while the low odorants are still present. This setup tests both the circuit's ability to converge rapidly enough to give accurate posterior samples within a single sniff, and its ability to correctly infer odorant concentrations even in the presence of distractors [63, 66]. In Fig. 4, we show that circuits with one-to-one or naively distributed codes do not give accurate estimates of the concentration mean within  $200\mathrm{ms}$ , while tuning the geometry to match the receptor affinities enables fast convergence. All three circuits overestimate the posterior variance at short times, consistent with what one would expect for unadjusted Langevin samplers [109], but the geometry-aware model's estimate decays most rapidly towards the target. Therefore, when the synaptic weights are tuned, our circuit model can enable fast, robust estimation of concentration statistics, even in the presence of distractors.
Figure 4: Fast uncertainty estimation using Langevin sampling of the posterior. Here, we use a simple concentration estimation task in which 5 randomly-selected 'low' odorants out of a panel of 1000 appear at concentration 10 at time 0 s, and then a further 5 randomly-selected 'high' odorants appear at concentration 40 at time 1 s. A. Smoothed timeseries of instantaneous concentration estimates for low, high, and background odorants, for models with one-to-one (top), naively distributed (middle), and geometry-aware (bottom) codes. Background odorant estimates are shown as mean  $\pm$  standard deviation over odorants. Dashed lines show true concentrations over time. B. Cumulative estimates of concentration mean for low (top) and high (bottom) odorants after the onset of the low odorants for one-to-one, naively distributed, and geometry-aware codes. Black lines indicate baseline estimates of the posterior mean. Thick colored lines indicate means over odorants. C. As in B, but for the estimated variance. D. As in B, but after the onset of high odorants. E. As in C, but after the onset of high odorants. See Appendix G for details of our numerical methods, and for individual-odor traces.
In this paper, we have derived a novel, minimal rate model for fast Bayesian inference in early olfaction. Unlike previously-proposed algorithms for CS in olfaction, this model has a clear mapping onto the circuits of the mammalian OB. We showed that this model successfully performs odorant identification across a biologically-relevant range of scene sparsities and circuit sizes. This model therefore exemplifies how normative approaches can blur the lines between algorithmic and mechanistic understanding of neural circuits [82, 83]. We now conclude by discussing possible avenues for future inquiry, as well as some of the limitations of our work.
One limitation of our simulations is that we have chosen to distribute the neural code randomly, and have allowed for negative entries in the mitral-granule synaptic weight matrix  $\mathbf{A}\Gamma$  (see methods in Appendix G). These features are not entirely biologically satisfactory. However, our model only captures the mitral cells and granule cells, overlooking a number of other inhibitory cell types that could contribute to solving this problem, e.g., through feedforward inhibition onto mitral cells or lateral inhibition across granule cells [48, 49]. Fundamentally, negative values in the geometry-aware decoding matrix  $\Gamma$  arise as although the affinity matrix is positive, its inverse will contain negative elements. A biologically-plausible realization of the geometry-aware code through the introduction of additional cell types could be achieved by decomposing the inverse into several components, yielding sparse, consistently-signed connectivity [110]. As a first step, in Supp. Fig. G.4 we show that similar performance to Fig. 4 can be achieved using a sparse non-negative randomly distributed code. As a result, one objective for future work will be to develop better models for the decoding matrix  $\Gamma$  that result in more realistic connectivity.
Though our model captures two of the interesting features of the anatomy and physiology of the OB—symmetric dendrodendritic coupling and state-dependent inhibition of mitral cells—there are
many biological details which we have not addressed. First, our linear model for OSN mean firing neglects receptor antagonism, gain control, and other nonlinear effects [94, 111, 112], which are known to affect the performance of Gaussian CS models for olfaction [26, 113]. Second, our models are rate-based, while neurons in the OB spike. In spiking implementations of sampling networks, the noise is not uncorrelated across neurons, complicating their biological interpretation [38, 114]. Constructing models that capture these richly nonlinear effects will be an important objective for future work. A first step towards such a nonlinear model would be to build a spiking network that approximates the rate-based models considered here, which could be accomplished using the efficient balanced network formalism for distributed spiking networks [38]. Another step would be to add a Hill function nonlinearity to the OSN model to approximate competitive binding, as studied for Gaussian compressed sensing by Qin et al. [26]. One challenge in constructing models that incorporate additional nonlinearity is that the simple linear strategy for distributing the code used here may no longer be directly applicable. In Appendix E, we illustrate this obstacle for the relatively simple case of a model with the same linear OSNs and Poisson likelihood but an  $L_{0}$  instead of Gamma prior, building on recent work on circuits for Gaussian CS with  $L_{0}$  priors [115].
A closely related point is that we model the weights of the synapses between mitral and granule cells as fixed, and do not consider synaptic plasticity. In particular, we assume that they are tuned to the statistics of the receptor affinities without specifying a mechanism by which this tuning could take place. In biology, receptor abundances and other OSN properties display activity-dependent adaptation over long timescales, meaning that the optimal tuning is unlikely to be static [56, 116]. Some past works have sought to incorporate plasticity of the mitral-granule cell synapses into decoding models [24, 117], tying into a larger body of research on how plasticity can enable flexible feature extraction in olfaction [118, 119]. This learning should lead to measurable changes in the population response to odorant panels, which our model predicts should be linked in a precise way to account for receptor-induced correlations in the responses to the most frequently present odorants. Experimentally characterizing and carefully modeling these changes in responses across timescales will be an interesting avenue for future work [120]. Experimental techniques to probe these ideas at the neural and behavioral level have recently been proposed [121, 122] which allow more precise control of stimulus and subjective geometry.
Though the circuit model derived in §3 incorporates a general Gamma prior represented by cortical feedback, our simulations focus on the special case in which the prior reduces to an exponential, in which the feedback neurons are not needed. Future work will therefore be required to carefully probe the effect of incorporating a Gamma prior with non-unit shape and to dissect the structure of the resulting modeled cortical feedback. More generally, it will be interesting to extend our framework to incorporate data-adaptive priors. Importantly, the stimuli used in this paper constitute an extremely impoverished model for the richness of the true odor world; we do not account for its rich dynamical structure and co-occurrence statistics. Adaptive priors as encoded by cortical feedback would allow circuits to leverage this structure, enabling faster and more accurate inference [88-90, 123, 124].
We conclude by noting that our work provides an example of how distributed coding can lead to faster inference than axis-aligned disentangled coding. In recent years, the question of when axis-aligned coding is optimal has attracted significant attention in machine learning and neuroscience [125-135]. Much of this work focuses on the question of when axis-aligned codes are optimal for energy efficiency or for generalization, whereas here we focus on the question of which code yields the fastest inference dynamics. These ideas are one example of the broader question of how agents and algorithms should leverage the rich geometry of the natural world to enable fast, robust learning and inference [136]. We believe that investigating how task demands and biological constraints affect the optimal representational geometry for that task is a promising avenue for illuminating neural information processing in brains and machines [41, 136, 137].
We thank Naoki Hiratani, Shanshan Qin, and Vikrant Kapoor for useful discussions. This work was supported by NSF grants DMS-2134157 and CAREER IIS-2239780 to CP, NIH grants R01DC017311 and R01DC016289 to VNM, and NTT Research award A47994 to VNM. CP received additional support from a Sloan Research Fellowship. PM was partially supported by a grant from the Harvard Mind Brain Behavior Interfaculty Initiative. JDZ was partially supported by NIH grant K99DC017754. This work has been made possible in part by a gift from the Chan Zuckerberg Initiative Foundation to establish the Kempner Institute for the Study of Natural and Artificial Intelligence. A subset of the computations in this paper were run on the FASRC cluster supported by the FAS Division of Science Research Computing Group at Harvard University.
[1] Jakob Johann von Uexkull. Umwelt und Innewelt der Tiere. Springer, 1909. doi:10.1007/978-3-662-24819-5.
[2] Ed Yong. An Immense World: How Animal Senses Reveal the Hidden Realms Around Us. Random House, New York, first edition, 2022. ISBN 9780593133231.
[3] D. H. Hubel and T. N. Wiesel. Receptive fields of single neurones in the cat's striate cortex. The Journal of Physiology, 148(3):574-591, 1959. doi:https://doi.org/10.1113/jphysiol.1959.sp006308. URL https://physoc.onlinelibrary.wiley.com/doi/abs/10.1113/jphysiol.1959.sp006308.
[4] D. H. Hubel and T. N. Wiesel. Receptive fields, binocular interaction and functional architecture in the cat's visual cortex. The Journal of Physiology, 160(1):106-154, 1962. doi:https://doi.org/10.1113/jphysiol.1962.sp006837. URL https://physoc.onlinelibrary.wiley.com/doi/abs/10.1113/jphysiol.1962.sp006837.
[5] Karl Kandler, Amanda Clause, and Jihyun Noh. Tonotopic reorganization of developing auditory brainstem circuits. Nature neuroscience, 12(6):711-717, 2009. doi:https://doi.org/10.1038/nn.2332.
[6] Fred Attneave. Some informational aspects of visual perception. Psychological Review, 61(3): 183, 1954. doi:https://doi.org/10.1037/h0054663.
[7] Horace B. Barlow. Possible Principles Underlying the Transformations of Sensory Messages. In *Sensory Communication*. The MIT Press, 09 2012. ISBN 9780262518420. doi:10.7551/mitpress/9780262518420.003.0013. URL https://doi.org/10.7551/mitpress/9780262518420.003.0013.
[8] Mandyam Veerambudi Srinivasan, Simon Barry Laughlin, A. Dubs, and George Adrian Horridge. Predictive coding: a fresh view of inhibition in the retina. Proceedings of the Royal Society of London. Series B. Biological Sciences, 216(1205):427-459, 1982. doi:10.1098/rspb.1982.0085.
[9] R. Linsker. Self-organization in a perceptual network. Computer, 21(3):105–117, 1988. doi:10.1109/2.36.
[10] Johannes H. van Hateren. A theory of maximizing sensory information. Biological Cybernetics, 68(1):23-29, Nov 1992. ISSN 1432-0770. doi:10.1007/BF00203134. URL https://doi.org/10.1007/BF00203134.
[11] Joseph J. Atick and A. Norman Redlich. What Does the Retina Know about Natural Scenes? Neural Computation, 4(2):196-210, 03 1992. ISSN 0899-7667. doi:10.1162/neco.1992.4.2.196. URL https://doi.org/10.1162/neco.1992.4.2.196.
[12] Bruno A. Olshausen and David J. Field. Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature, 381(6583):607-609, Jun 1996. ISSN 1476-4687. doi:10.1038/381607a0. URL https://doi.org/10.1038/381607a0.