Title: Shaping the distribution of neural responses with interneurons in a recurrent circuit model
Venue: NeurIPS 2024
Accepted: yes
Keywords: Efficient coding

optimal transport

Gaussianization

Hebbian plasticity

gain modulation

neural adaptation
Abstract: Efficient coding theory posits that sensory circuits transform natural signals into neural representations that maximize information transmission subject to resource constraints. Local interneurons are thought to play an important role in these transformations, shaping patterns of circuit activity to facilitate and direct information flow. However, the relationship between these coordinated, nonlinear, circuit-level transformations and the properties of interneurons (e.g., connectivity, activation functions) remains unknown. Here, we propose a normative computational model that establishes such a relationship. Our model is derived from an optimal transport objective that conceptualizes the circuit's input-response function as transforming the inputs to achieve a target response distribution. The circuit, which is comprised of primary neurons that are recurrently connected to a set of local interneurons, continuously optimizes this objective by dynamically adjusting both the synaptic connections between neurons as well as the interneuron activation functions. In an application motivated by redundancy reduction theory, we demonstrate that when the inputs are natural image statistics and the target distribution is a spherical Gaussian, the circuit learns a nonlinear transformation that significantly reduces statistical dependencies in neural responses. Overall, our results provide a framework in which the distribution of circuit responses is systematically and nonlinearly controlled by adjustment of interneuron connectivity and activation functions.

Shaping the distribution of neural responses with interneurons in a recurrent circuit model

Abstract

Efficient coding theory posits that sensory circuits transform natural signals into neural representations that maximize information transmission subject to resource constraints. Local interneurons are thought to play an important role in these transformations, dynamically shaping patterns of local circuit activity to facilitate and direct information flow. However, the relationship between these coordinated, nonlinear, circuit-level transformations and the properties of interneurons (e.g., connectivity, activation functions, response dynamics) remains unknown. Here, we propose a normative computational model that establishes such a relationship. Our model is derived from an optimal transport objective that conceptualizes the circuit's input-response function as transforming the inputs to achieve an efficient target response distribution. The circuit, which is comprised of primary neurons that are recurrently connected to a set of local interneurons, continuously optimizes this objective by dynamically adjusting both the synaptic connections between neurons as well as the interneuron activation functions. In an example application motivated by redundancy reduction, we construct a circuit that learns a dynamical nonlinear transformation that maps natural image data to a spherical Gaussian, significantly reducing statistical dependencies in neural responses. Overall, our results provide a framework in which the distribution of circuit responses is systematically and nonlinearly controlled by adjustment of interneuron connectivity and activation functions.

1 Introduction

The problem of transforming a signal into a representation with a given target distribution (or within a target set of distributions) is a classical problem whose origins can be traced back more than two centuries [1]. Many methods in statistics, signal processing and machine learning can be interpreted within this context. For example, data whitening is a common preprocessing step that linearly transforms a signal to have identity covariance [2]. Independent component analysis [ICA; 3] is a signal processing method that linearly transforms a signal so as to minimize higher-order statistical dependencies in addition to removing second-order dependencies. Nonlinear transformations of skewed or heavy-tailed data to approximately Gaussianize their distribution can facilitate statistical analyses [4, 5]. Machine learning methods for density estimation such as Gaussianization [6-10] and normalizing flows [11-13] nonlinearly transform high-dimensional signals with complex densities into more tractable representations with approximately Gaussian densities.

Close-up

of adaptive interneuron

Signal:

Response:

Network

W1

W3

W1

W3

This problem may also provide a framework for constructing and evaluating normative theories of sensory processing. Efficient coding theory posits that sensory systems maximize the information they transmit about sensory signals to downstream areas subject to resource constraints [14-17]. In one instantiation of this theory, the redundancy reduction hypothesis posits that sensory circuits transform natural signals into representations to minimize or eliminate statistical dependencies between coordinates, essentially producing factorized response distributions [14, 15, 18]. In a separate, but related, instantiation, sparse coding theory posits that population responses are optimized for sparsity [19-21], which is naturally interpreted as a constraint on the shape of the distribution of responses. In another line of theoretical work, sensory representations are posited to maximize the Fisher information of the inputs [22-24], which can be interpreted as a statement about the joint distribution of the inputs and responses. Importantly, each of these theories can be formulated as a transformation of the signal into a representation with target distribution that is optimal under information theoretic and metabolic constraints. However, it is not clear how neural circuits learn or implement these potentially nonlinear transformations.

Neural circuits are typically comprised of populations of primary (excitatory) neurons and local (inhibitory) interneurons. Extensive experimental measurements have led to the idea that local interneuron populations allow neural circuits to flexibly shape patterns of primary neuron responses so as to coordinate or regulate information flow [25-31]. Consequently, local interneurons are natural candidate mechanisms responsible for shaping circuit responses into efficient representations. However, the precise relationship between the physiological properties and connectivity of local interneurons and the coordinated response properties of populations of primary neurons remains unclear.

Several normative mechanistic models have been proposed to explain how neural circuits can linearly transform their inputs into a representation whose distribution lies within a target set (e.g., the set of distributions with identity covariance) [32-36]. These models are derived from optimization objectives for linear redundancy reduction, including (adaptive) decorrelation and ICA, and the circuit parameters (e.g., synaptic weights, response gains) are optimized to match the data distribution. In these cases, the optimization steps correspond to processes such as gain modulation and synaptic plasticity, thus demonstrating how adjustments of circuit parameters according to local signals can optimize a global, circuit-level objective for redundancy reduction of the neural responses. While linear transformations can remove second-order statistical dependencies, they cannot remove higher-order statistical dependencies that are prominent in sensory signals [7]. Furthermore, early sensory systems exhibit a host of prominent nonlinear response properties [37-39] that are not well-approximated by linear models.

Here, we seek a normative circuit model that nonlinearly transforms inputs to produce responses with a (spherical) target distribution. We develop an optimal transport objective for transforming the input signal into a neural representation with a given target distribution, and derive an algorithm (Alg. 1) that can be mapped onto a dynamical model of a neural circuit, Fig. 1. The circuit model is comprised of primary neurons that are recurrently connected to local interneurons and the circuit adapts its responses using a combination of Hebbian synaptic plasticity and interneuron adaptation. Complementary computational roles for Hebbian plasticity and interneuron adaptation emerge from this analysis: (i) synapses are updated according to a Hebbian learning rule to identify projections of the signal that are least aligned with the target distribution (essentially, a form of projection pursuit [40]); (ii) gains and nonlinear activation functions of interneurons are adjusted to transform the marginal circuit responses along directions defined by the synaptic weights. Together these operations transform the distribution of responses to approximate the target distribution.

As a primary test case motivated by redundancy reduction theory, we apply our algorithm to inputs derived from natural images (responses of local oriented filters, qualitatively similar to those found in the primary visual cortex) and the target distribution is the spherical Gaussian. 1 We find that the algorithm learns a nonlinear transformation that approximately Gaussianizes the responses and significantly reduces statistical dependencies between coordinates. Overall, our results demonstrate how local interneurons may adjust their connectivity and response properties to nonlinearly reshape the distribution of circuit responses, thus facilitating efficient transmission of information.

2 Circuit objective

Consider a neural circuit with N ≥ 2 primary neurons that transforms input signals s ∈ R N , which are distributed according to a density p s , into circuit responses r ∈ R N (Fig. 1). The inputs may represent a direct sensory input (e.g., the rate at which photons are absorbed by a cone) or the weighted sum of multiple inputs (e.g., the postsynaptic current). The responses r represent the firing rate (or the logarithm of the firing rate) of the neuron. For simplicity, we assume that the circuit responses are a deterministic function of the input signals; that is, r = T ( s ) for a function T : R N ↦→ R N .

2.1 Optimal transport objective

We assume that the objective of the circuit is to transform its inputs s so that the circuit responses r = T ( s ) follow a (spherical) target distribution p target while minimizing the L 2 -distance between responses and input signals. Mathematically, this corresponds to an optimal transport problem [42]

where the minimization is over a suitable class of functions T , λ ∈ R is a regularizing term and, unless otherwise noted, expectations are over the input distribution p s . Note that the choice of λ does not affect the optimal solution as E [ ∥ T ( s ) ∥ 2 ] is fixed provided T ( s ) ∼ p target; however, it will affect the optimization algorithm. Assuming p s is sufficiently regular, the minimum in eq. (1) is the squared Wasserstein-2 distance between p s and p target and the input-response transformation T is the so-called optimal transport plan that maps the input distribution p s to the target distribution p target .

Our goal is to derive an online algorithm for optimizing the objective in eq. (1) that can be implemented in a neural circuit model. We accomplish this by defining a distance between the response distribution p r and the target distribution p target that can be estimated in a neural circuit, and then solving the optimization problem by incorporating this measure as a constraint, using the method of Lagrange multipliers.

2.2 Measuring the discrepancy between the response distribution and the target distribution

Common candidates for measuring the discrepancy between two distributions include KullbackLeibler (KL) divergence or integral probability metrics [43]. The former typically require numerous

samples to estimate the density p r , whereas neural circuits must operate in the online setting without access to the full history of their responses. Therefore, we use an integral probability metric that quantifies the difference between the random variables when evaluated using constraint functions that can be rapidly estimated online.

We restrict our solution to use constraint functions that compare the marginal response distributions to the marginals of p target, denoted p marginal, which are all equal under our assumption that p target is spherical. Such constraint functions are well matched to signals that are linear mixtures of independent sources, as in generative models for ICA. Even when the signal statistics are not generated according to a linear mixture model, the Cramér and Wold theorem [44] suggests that transforming sufficiently many marginals of the response distribution may effectively transform the multivariate response distribution (though the number of marginals required may be quite large). Our motivation for measuring marginal response distributions is due, in part, by our goal of modeling local interneurons, whose inputs are naturally modeled as weighted sums of primary neuron responses (i.e., their input distributions are marginals of the primary responses).

To compare the marginal response distributions, we first select a finite set of directions defined by K ≥ 1 unit vectors w 1 , . . . , w K ∈ R N , which can be either randomly sampled, chosen based on prior knowledge of the signal statistics, or learned from data using projection pursuit [40]. We then choose a class of scalar functions { h ( θ, · ) } parameterized by θ , which defines a semi-metric between the marginal of r in the direction w and p marginal to be max θ | E [ ϕ ( θ, r · w )] | , where

For example, when { h ( θ, · ) } parameterizes all Lipschitz-1 (indicator) functions, this induces the Wasserstein-1 (total variation) distance between the marginal response distribution and p marginal . Given directions w 1 , . . . , w K and constraint functions { h ( θ, · ) } , we express the distance between the response density p r and the standard Gaussian distribution as the sum of the marginal distances:

where W := [ w 1 , . . . , w K ] is the N × K matrix of concatenated unit vectors. 2

How do we choose the constraint functions { h ( θ, · ) } ? In general, the choice should be well-suited to the input distribution p s and the target distribution p r . For example, consider the simple case when the inputs follow a centered Gaussian distribution with unknown covariance structure, and the target distribution is the spherical Gaussian distribution N ( 0 , I ) . The Wasserstein-2 distance between a marginal distribution of the inputs and the standard normal distribution N (0 , 1) can be expressed in terms of the difference between the second moments, so a quadratic function h ( z ) = 1 2 z 2 suffices. In section 4.2, we consider a parametric class motivated by natural signal statistics.

2.3 Optimization using Lagrange multipliers

We replace the condition r ∼ p target in eq. (1) with the condition max W d W ,ϕ ( p r ) = 0 , which we enforce using Lagrange multipliers. This results in the minimax optimization problem

where L is defined by

and θ := ( θ 1 , . . . , θ K ) is the set of concatenated parameters, g := ( g 1 , . . . , g K ) is a K -dimensional vector of Lagrange multipliers, and the circuit transform is defined by T ( s ) = arg min r L ( W , θ , g , s , r ) . The maximization over g and θ effectively minimizes the distances between the marginal distributions of the responses r along the directions w 1 , . . . , w K and p marginal , whereas the maximization over the matrix W learns directions along which the marginals of s are least aligned with p marginal, essentially performing projection pursuit [40].

3 Algorithm and circuit implementation

We now derive an online gradient-based algorithm for optimizing the objective in eq. (2), then map the algorithm onto a recurrent neural circuit. Spiking activity operates on a much faster timescale than neural or synaptic adaptation mechanisms, so we assume that the neural activities equilibrate before the neural activations and synapses are updated.

3.1 Fast recurrent neural dynamics

At each iteration, the circuit receives a stimulus s . The (discretized) recurrent neural response dynamics (Fig. 1) correspond to gradient-descent minimization of L with respect to r :

where η r > 0 is a small constant, µ := 1 + λ represents a leak term, z i := r · w i is the weighted input to the i th interneuron, f ( θ i , · ) := ∂ϕ ( θ i , · ) /∂z is the activation function, g i is a multiplicative gain that scales the output, and n i denotes the output. Notably, the interneuron activation response function g i f ( θ i , · ) is parameterized by ( g i , θ i ) , which can vary across interneurons, so the interneuron responses are heterogeneous. For each i , synaptic weights w i connect the primary neurons to the i th interneuron and symmetric weights -w i connect the i th interneuron to the primary neurons. From eq. (4), we see that the neural responses are driven by the signal s , a leak term -r , and recurrent weighted feedback from the interneurons -Wn , where n := ( n 1 , . . . , n K ) .

Since the neural activities equilibrate before other updates are performed, the responses r are a fixed point of L ( W , θ , g , s , · ) . In general, we do not have a closed-form expression for r ; however, if g i ≥ 0 and ϕ ( θ i , · ) are convex, then L ( W , θ , g , s , · ) is convex and we can express the transform as

In Appx. A we show that the transform T ( · ) is invertible whenever g i ≥ 0 and ϕ ( θ i , · ) are convex and thus it defines a precise relationship between the input distribution p s and response distribution p r .

3.2 Gain modulation, activation function adaptation and Hebbian plasticity

After the neural activities reach equilibrium, we maximize L by taking concurrent gradient-ascent steps with respect to g i , θ i and w i :

where η g , η θ , η w ≥ 0 are the respective learning rates, which control the relative speeds of gain modulation, neural adaptation and synaptic plasticity, respectively. For example, at the extremes, we can fix the gains, activation functions or synaptic weights by setting η g = 0 , η θ = 0 or η w = 0 , respectively. Notably, while synaptic plasticity [47] and gain modulation [48] are well-studied circuit mechanisms that support learning and adaptation, adjustments of nonlinear neural activation functions are not as well established. Nevertheless, there is some emerging evidence that neurons also adapt their activation functions in response to changes in their input statistics [49, 50].

The circuit operates online and the updates are local in the sense that the updates to the gain and activation function of the i th interneuron depend only on variables θ i and z i . The updates to the synapses w i (and -w i ) are proportional (inversely proportional) to the product of the preand postsynaptic activities, so they are both local and Hebbian (anti-Hebbian) [51]. Finally, to ensure that the vectors w 1 , . . . , w K have unit norm, we normalize the weights after each update: w i ← w i / ∥ w i ∥ . This can be viewed as form of homeostatic plasticity such as synaptic scaling [52]. While the feedforward weights w i and feedback weights -w i are constrained to be symmetric, these can be decoupled due to the symmetry of the Hebbian learning rule. Both theoretical and empirical evidence of this is shown for a related adaptive whitening circuit in [36, appendix E.2].

3.3 Online algorithm

Combining the neural dynamics, the interneuron adaptation and synaptic plasticity steps yields our online algorithm (Alg. 1), which we write in vector-matrix notation by defining the normalization function P ( W ) := [ w 1 / ∥ w 1 ∥ , . . . , w K / ∥ w K ∥ ] .

3.4 Relation to existing algorithms

Algorithm 1 is naturally viewed as a nonlinear extension of a number of existing algorithms for linear data whitening that have neural circuit implementations [34, 36, 53, 54]. In particular, when the constraint function is quadratic, h ( z ) = 1 2 z 2 , and the target distribution is the spherical Gaussian, N ( 0 , I ) , then the activation function is the identity, f ( z ) = z , and the optimization in eq. (2) enforces that the second moments of the responses match the second moments of N ( 0 , I ) , which corresponds to data whitening. If the gains are fixed ( η g = 0 ) and K ≥ N (i.e., synaptic adaptation only), then Alg. 1 corresponds to the adaptive whitening algorithm presented in [34, 53]. Alternatively, if the synaptic weights are fixed ( η w = 0 ) and K ≥ N ( N +1) / 2 (i.e., interneuron gain adaptation only), then Alg. 1 corresponds to the adaptive whitening algorithm presented in [54]. Finally, if the gains adapt on a fast timescale and the synapses update on a slow timescale (i.e., η g ≫ η w > 0 ), Alg. 1 corresponds to the multi-timescale adaptive whitening algorithm presented in [36].

When the target distribution is the spherical Gaussian and W is constrained to be an orthogonal matrix, Alg. 1 is related to existing iterative algorithms for Gaussianization that alternate between (a) orthogonal transformations and (b) marginal Gaussianization of the coordinates [6, 8]. In the case that the column vectors of W are orthogonal, Gaussianization along one marginal does not affect the responses along other marginals, allowing these operations to be performed independently of one another. In general, we allow the column vectors of W to be non-orthogonal and potentially overcomplete so that marginal Gaussianization along one basis vector affects the other marginal distributions. Consequently, the algorithm is more complicated to analyze mathematically (e.g., obtaining convergence guarantees), but it is more biologically realistic since neural systems are unlikely to have orthonormal synaptic weight vectors.

4 Gaussianization of natural image statistics

We apply our algorithm to the problem of efficient nonlinear encoding of natural signals, specifically oriented filter responses to visual images. 3 Redundancy reduction theories posit that early sensory systems transform natural signal into neural representations with reduced statistical redundancies [14, 15, 17]. In support of this hypothesis, early sensory representations exhibit far less spatial and temporal correlation than natural signals [55, 56] and methods such as linear ICA have been used to derive optimal representations of natural signals that are approximately matched to early sensory neural properties [18, 19, 57].

Linear whitening and ICA transforms can eliminate simple forms of statistical dependency, but their responses exhibit higher-order statistical dependencies when applied to natural signals [58], suggesting that sensory systems can more efficiently represent natural signals by implementing nonlinear transforms. Consistent with this, nonlinear phenomenological models of neural responses (e.g., divisive normalization [37, 39]) effectively reduce these higher-order statistical dependencies [38, 59]. However, the circuit mechanisms that support these nonlinear transformations are unknown.

A)

natural image

6

4

2

0

2

4

6

10

3

10

2

10

1

10 0

B)

signal density

ps

= 0.45,

= 0.8

6

4

2

0

2

4

6

6

4

2

0

2

4

6

C)

interneuron act.

gf

(

,

z

)

g

= 0.41,

= 2.04

6

4

2

0

2

4

6

6

4

2

0

2

4

6

D)

circuit transform

r

=

T

(

s

)

6

4

2

0

2

4

6

10

3

10

2

10

1

10 0

E)

response density

pr

6

4

2

0

2

4

6

10

3

10

2

10

1

10 0

= 0.25,

= 0.65

6

4

2

0

2

4

6

6

4

2

0

2

4

6

g

= 0.24,

= 2.22

6

4

2

0

2

4

6

6

4

2

0

2

4

6

6

4

2

0

2

4

6

10

3

10

2

10

1

10 0

6

4

2

0

2

4

6

10

3

10

2

10

1

10 0

= 0.03,

= 0.41

6

4

2

0

2

4

6

6

4

2

0

2

4

6

g

= 0.01,

= 2.6

6

4

2

0

2

4

6

6

4

2

0

2

4

6

6

4

2

0

2

4

6

10

3

10

2

10

1

10 0

We consider the particular target of Gaussian responses, which may be interpreted as firing rates, or as their logs (i.e.., membrane potentials are Gaussian, and firing rates are exponentiated). From the perspective of coding efficiency and computation, Gaussian representations are appealing for a variety of reasons. First, among distributions with given covariance structure, Gaussian distributions have maximum entropy. Therefore, if metabolic demands are a function of the (co)variance of the response distribution, then the Gaussian distribution maximizes information transmission under metabolic constraints. Second, compression based on information theoretic objectives often reduces to linear projection when the data distribution is Gaussian [60-62], so Gaussianization facilitates downstream computation. In addition, the efficiency of the representation is preserved under orthogonal transformation [7]. Finally, experiments have shown that single neurons in the fly early visual system adaptively Gaussianize their univariate responses [63], and neural populations in early sensory systems decorrelate their multivariate responses [56, 64-67]. Therefore, neural circuit models that nonlinearly transform signals to jointly Gaussianize their responses may offer normative, parsimonious explanations of nonlinear transformations in early sensory systems.

4.1 Description of the input signal

We computed the responses of a local oriented filter [68], roughly matched to typical receptive field selectivity of neurons in primate visual cortex [69], applied to natural images from the Kodak dataset [70]. These local filter responses are notorious for their sparse heavy-tailed statistical properties that can be well approximated by generalized Gaussian distributions of the form p s ( s ) ∝ exp( -| s/α | β ) , where α is referred to as the scale parameter and β is referred to as the shape parameter [71]. Fig. 2AB shows example images and histograms of the local filter responses along with fitted generalized Gaussian distributions whose scale and shape parameters ( α, β ) vary across images. While linear methods such as variance normalization are sufficient for rescaling the distribution, adaptive nonlinear transformations are required to reshape these heavy-tailed distributions.

Next, we generated 2-dimensional signals from pairs of the local filter responses for images at fixed horizontal spatial offsets ranging between d = 2 and d = 64 . Contour plots (using kernel density

A) signal

B)ZCA

C)responses (K=2)

E)responses (K=4)

MI=1.33e-1

MI=1.08e-1

MI=2.91e-2

MI=5.02e-3

MI=4.61e-3

d=2

MI=3.93e-2

MI=4.02e-2

MI=3.5e-2

MI=5.6e-3

MI=4.41e-3

8

d

MI=1.02e-2

MI=1.04e-2

MI=4.94e-3

MI=4.64e-3

MI=5.89e-3

d=32

estimation) of the local filter response pairs and symmetric (or ZCA) whitened local filter response pairs for a natural image (specifically, the top-left image from Fig. 2) are shown for select d in Fig. 3AB. To quantify the statistical dependencies between coefficients, we estimated the mutual information between the pairs of coefficients after discretizing them into bins of width 0.5. In Fig. 4, we plot the estimated mutual information (using bin size 0.5) between the local filter response pairs (blue line) and ZCA whitened local filter response pairs (orange line) for spatial offsets between d = 2 and d = 64 . Note that aside from d = 2 , the linear ZCA whitening transform does not significantly reduce the mutual information between coordinates. (Similar results have been found when applying linear ICA transforms; see, e.g., [72, Figure 6].) Fig. 4 also suggests that ZCA whitening can even slightly increase the mutual information between coordinates-see also, rows d = 8 and d = 32 of Fig. 3AB-though these effects are quite small and may be a consequence of the discretization step when estimating mutual information.

4.2 Choice of activation functions

How do we choose the family of activation functions { f ( θ, · ) } ? One approach is to choose a kernel that can approximate a general class of functions. An alternative approach, adopted here, which is motivated by the efficient coding hypothesis [14, 15, 17], is to choose a family of activation functions that is well matched to the marginal statistics of natural signals. In Fig. 2C, we plot examples of optimal activations for transforming local filter responses from different images (thick gray curves).

Since the marginals of the local filter responses are well-approximated by generalized Gaussian distributions [71], a sensible approach is to identify a family of interneuron activation functions that are optimal for transforming generalized Gaussian distributions with varying ( α, β ) into the standard Gaussian distribution. When µ = 0 , this implies (see Appx. B) that for each choice of scale α and shape β , there is a gain g and parameter θ such that

where Φ( · ) is the cdf of N (0 , 1) . However, if we define f ( θ, z ) in terms of the above display, then we do not have a closed-form solution for ϕ ( θ, z ) or ∇ θ ϕ ( θ, z ) , which are both required to implement

2

4

8

16

32

64

spatial offset (pixels)

10

2

10

1

mutual information (bits)

signal

whitened signal

responses (

K

=2)

responses (

K

=3)

responses (

K

=4)

Alg. 1. Instead, we found that F -1 α,β ◦ Φ( · ) can be well approximated by the simple algebraic form

where a ( θ ) and b ( θ ) are specified nonnegative functions of θ > 1 . Intuitively, the linear component shapes the marginal density locally around zero, while the higher-order monomial shapes the tails of the marginal distribution. In Appx. B, we show that the monomial activation f ( θ, z ) = sign ( z ) | z | θ is optimal for Gaussianizing scalar signals whose marginal tail densities satisfy p s ( s ) ∝ | s | q -1 exp( -| s | 2 q ) , where q = 1 /θ . This closely resembles the tail densities of generalized Gaussian densities (when α = 1 ), suggesting monomial activations are effective for shaping the tails.

4.3 Marginal density of local filter responses

We first apply Alg. 1 in the scalar setting N = K = 1 to demonstrate that our choice of activation function in eq. (6) is indeed well matched to the shape of heavy-tailed marginals of local filter responses. For each image, we ran Alg. 1 on the local filter responses with µ = 0 , learning rates ( η g , η θ ) = (10 -5 , 10 -5 ) and batch size 10 for 10 5 iterations. Fig. 2CD shows the learned interneuron activation functions gf ( θ, · ) and the learned transforms T ( · ) . Fig. 2E shows histograms of the circuit responses. Compared to the local filter responses, the circuit responses are visually much closer to Gaussian. We found that the circuit performs worse when the distribution p s is more 'peaked' around zero (i.e., when s is sparser and the fitted shape parameter β is smaller), as evidenced by the mismatch between the response distribution p r and the Gaussian distribution N (0 , 1) near zero in the bottom row of Fig. 2E (see Appx. C for more examples). However, even in this case, the interneuron activation and circuit transform are close to optimal (Fig. 2CD, bottom row).

4.4 Joint density of pairs of local filter responses

Next, we apply Alg. 1 in the multivariate setting N = 2 , to pairs of spatially offset filter responses. For each image and offset, we ran Alg. 1 with K = 2 , 3 , 4 interneurons, µ = 0 , learning rates ( η g , η θ , η w ) = (10 -4 , 10 -6 , 10 -4 ) , and batch size 10 for 10 6 iterations. Contour plots of the learned circuit responses for one image are shown in Fig. 3CDE; see Appx. C for more examples. The mutual information between circuit responses is shown in Fig. 4. We see that K = 3 interneurons significantly reduces the mutual information between circuit responses for spatial offsets less than d = 32 . The reduction is much greater than obtained using K = 2 interneurons and about the same as obtained using K = 4 interneurons. For spatial offsets greater than d = 32 , the raw filter responses already have low mutual information and the circuit does not not offer any significant reduction.

5 Discussion

We derived a novel online algorithm for transforming a signal to approximate a target distribution, using a recurrent neural circuit with Hebbian synaptic plasticity, gain modulation, and adaptation of interneuron activation functions. The model draws inspiration from the extensive neuroscience literature on efficient coding [15], Hebbian synaptic plasticity [47], interneuron function [29] and gain modulation [48], proposing complementary roles for different physiological processes: Hebbian synaptic plasticity learns stimulus axes that are least matched to the target distribution, and interneurons adapt their gains and activation functions to transform the marginal responses along these axes. The form of the input-output function for the local interneurons-linear-nonlinear with gain modulation-closely resembles phenomenological models of neurons [73], and the parameters W , θ , g can potentially be fit to neural recordings and compared with the optimal parameters that can be derived from the signal statistics p s . Furthermore, our model predicts a relationship between the interneuron activation function and the circuit transform; see Fig. 2CD for an example of an expansive interneuron activation function that corresponds to a compressive circuit transformation.

There are, however, some aspects of our circuit that are not biologically realistic. For example, our model focuses on the role of local interneurons in reshaping the response distribution and, for simplicity, assumes that the primary neurons have linear activation functions. A more realistic model would also include nonlinearity and adaptation in the primary neurons. Moreover, our model only includes synaptic connections between primary neurons and interneurons, which may be consistent with some sensory circuits (e.g., olfactory bulb), but cannot account for excitatoryexcitatory connections or inhibitory-inhibitory connections in cortical circuits. Finally, the synaptic weights are not sign-constrained, and violate Dale's law. This can be addressed by modifying the objective in eq. 2 so that the optimization is over non-negative weight matrices W ≥ 0 , which will result in a projected gradient step in Alg. 1; however, the circuit responses will generally be less aligned with the target distribution.

A limitation of our simulations is that we only test our method on two-dimensional inputs, demonstrating that three interneurons are sufficient to dramatically reduce the redundancy in the circuit responses. However, natural signals are generally high-dimensional and it is not clear how the number of interneurons required to effectively reduce redundancy will scale with the dimension of the signal. There is some basis for optimism. For example, visual inputs are highly structured-e.g., statistical dependencies between inputs rapidly decay with the distance between the inputs-so local interneurons only need to connect to neurons with overlapping or adjacent receptive fields, limiting the number of interneurons that are required as the dimensionality of the input signal grows [54].

There are a number of existing computational models that also explain how neural circuits can implement nonlinear transformations to efficiently encode their inputs. For example, several neural circuit models implement forms of divisive normalization [74-76], a transformation that is optimal for efficient encoding of natural signals [38, 59]. In addition, there is a body of work on normative spiking models derived from objectives that maximize the information encoded per spike [77-79], which can account for neural adaptation mechanisms such as gain control. Our work differs from these, by proposing a novel framing of sensory circuit computation in terms of transformations of probability distributions, which can be viewed as a population level version of the seminal work by Laughlin [16]. We then demonstrate in a normative circuit model how interneurons can play a critical role in optimizing this objective by measuring the marginal distribution of circuit responses and adjusting their feedback accordingly.

Finally, our results may also be relevant beyond the biological setting. Gaussianization and normalizing flows are active areas of research [10, 13, 80]. We offer a novel continuous-learning solution inspired by neuroscience that learns using a combination of weight updates and activation function updates (related to trainable activation functions [81]). In low-dimensional settings, when the constraint functions are matched to the signal statistics, we show that Gaussianization can be achieved using relatively few parameters. It is of primary interest to understand how the methods introduced here scale to high-dimensional signals, where the curse of dimensionality presents significant challenges.

Acknowledgments

We thank Colin Bredenberg and members of the Center for Computational Neuroscience at the Flatiron Institute for helpful feedback on an earlier draft of this work.

References

Gaspard Monge. Mémoire sur la théorie des déblais et des remblais. Mem. Math. Phys. Acad. Royale Sci. , pages 666-704, 1781.

Agnan Kessy, Alex Lewin, and Korbinian Strimmer. Optimal whitening and decorrelation. The American Statistician , 72(4):309-314, 2018.

Aapo Hyvärinen and Erkki Oja. Independent component analysis: algorithms and applications. Neural Networks , 13(4-5):411-430, 2000.

Mike H Hoyle. Transformations: An introduction and a bibliography. International Statistical Review/Revue Internationale de Statistique , pages 203-223, 1973.

Remi M Sakia. The Box-Cox transformation technique: a review. Journal of the Royal Statistical Society Series D: The Statistician , 41(2):169-178, 1992.

Scott Chen and Ramesh Gopinath. Gaussianization. Advances in Neural Information Processing systems , 13, 2000.

Siwei Lyu and Eero P Simoncelli. Reducing statistical dependencies in natural signals using radial gaussianization. Advances in Neural Information Processing Systems , 2008:1009-1016, 2008.

Valero Laparra, Gustavo Camps-Valls, and Jesús Malo. Iterative Gaussianization: from ICA to random rotations. IEEE Transactions on Neural Networks , 22(4):537-549, 2011.

Johannes Ballé, Valero Laparra, and Eero P Simoncelli. Density modeling of images using a generalized normalization transformation. International Conference on Learning Representations , 2016.

Chenlin Meng, Yang Song, Jiaming Song, and Stefano Ermon. Gaussianization flows. In International Conference on Artificial Intelligence and Statistics , pages 4336-4345. PMLR, 2020.

Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In International conference on machine learning , pages 1530-1538. PMLR, 2015.

Laurent Dinh, David Krueger, and Yoshua Bengio. NICE: Non-linear independent components estimation. International Conference on Learning Representations , 2015.

George Papamakarios, Eric Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lakshminarayanan. Normalizing flows for probabilistic modeling and inference. The Journal of Machine Learning Research , 22(1):2617-2680, 2021.

Fred Attneave. Some informational aspects of visual perception. Psychological Review , 61(3): 183-193, 1954.

H B Barlow. Possible Principles Underlying the Transformations of Sensory Messages. In Sensory Communication , pages 216-234. The MIT Press, 1961.

Simon Laughlin. A simple coding procedure enhances a neuron's information capacity. Zeitschrift fur Naturforschung C, Journal of Biosciences , pages 910-912, 1981.

Eero P Simoncelli and Bruno A Olshausen. Natural image statistics and neural representation. Annual Review of Neuroscience , 24(1):1193-1216, 2001.

Anthony J Bell and Terrence J Sejnowski. The 'independent components' of natural scenes are edge filters. Vision research , 37(23):3327-3338, 1997.

Bruno A Olshausen and David J Field. Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature , 381(6583):607-609, 1996.

Peter Foldiak. Sparse coding in the primate cortex. The Handbook of Brain Theory and Neural Networks , 2003.

Bruno A Olshausen and David J Field. Sparse coding of sensory inputs. Current Opinion in Neurobiology , 14(4):481-487, 2004.

Alexandre Pouget, Sophie Deneve, Jean-Christophe Ducom, and Peter E Latham. Narrow versus wide tuning curves: What's best for a population code? Neural Computation , 11(1): 85-90, 1999.

Kechen Zhang and Terrence J Sejnowski. Neuronal tuning: To sharpen or broaden? Neural Computation , 11(1):75-84, 1999.

Deep Ganguli and Eero P Simoncelli. Efficient sensory encoding and Bayesian inference with heterogeneous neural populations. Neural computation , 26(10):2103-2134, 2014.

TF Freund and G Buzsáki. Interneurons of the hippocampus. Hippocampus , 6(4):347-470, 1996.

Chris J McBain and André Fisahn. Interneurons unbound. Nature Reviews Neuroscience , 2(1): 11-23, 2001.

Henry Markram, Maria Toledo-Rodriguez, Yun Wang, Anirudh Gupta, Gilad Silberberg, and Caizhi Wu. Interneurons of the neocortical inhibitory system. Nature Reviews Neuroscience , 5 (10):793-807, 2004.

Carl P Wonders and Stewart A Anderson. The origin and specification of cortical interneurons. Nature Reviews Neuroscience , 7(9):687-696, 2006.

Adam Kepecs and Gordon Fishell. Interneuron cell types are fit to function. Nature , 505(7483): 318-326, 2014.

Gord Fishell and Adam Kepecs. Interneuron types as attractors and controllers. Annual Review of Neuroscience , 43:1-30, 2020.

Mattia Chini, Thomas Pfeffer, and Ileana Hanganu-Opatz. An increase of inhibition drives the developmental decorrelation of neural activity. Elife , 11:e78811, 2022.

Cristina Savin, Prashant Joshi, and Jochen Triesch. Independent component analysis in spiking neurons. PLoS Computational Biology , 6(4):e1000757, 2010.

Paul D King, Joel Zylberberg, and Michael R DeWeese. Inhibitory interneurons decorrelate excitatory cells to drive sparse code formation in a spiking model of V1. Journal of Neuroscience , 33(13):5475-5485, 2013.

Cengiz Pehlevan and Dmitri B Chklovskii. A normative theory of adaptive dimensionality reduction in neural networks. Advances in Neural Information Processing Systems , 28, 2015.

Nikolai M Chapochnikov, Cengiz Pehlevan, and Dmitri B Chklovskii. Normative and mechanistic model of an adaptive circuit for efficient encoding and feature extraction. Proceedings of the National Academy of Sciences , 120(29):e2117484120, 2023.

Lyndon R Duong, Eero P Simoncelli, Dmitri B Chklovskii, and David Lipshutz. Adaptive whitening with fast gain modulation and slow synaptic plasticity. Advances in Neural Information Processing systems , 2023.

David J. Heeger. Normalization of cell responses in cat striate cortex. Visual neuroscience , 9(2): 181-97, 1992. ISSN 0952-5238. doi: 10.1017/S0952523800009640.

Odelia Schwartz and Eero P. Simoncelli. Natural signal statistics and sensory gain control. Nature Neuroscience , 4(8):819-825, August 2001.

Matteo Carandini and David J Heeger. Normalization as a canonical neural computation. Nature Reviews Neuroscience , 13(1):51-62, 2012.

Peter J Huber. Projection pursuit. The Annals of Statistics , pages 435-475, 1985.

György Buzsáki and Kenji Mizuseki. The log-dynamic brain: how skewed distributions affect network operations. Nature Reviews Neuroscience , 15(4):264-278, 2014.

Cédric Villani. Optimal Transport: Old and New , volume 338. Springer, 2009.

Bharath K Sriperumbudur, Kenji Fukumizu, Arthur Gretton, Bernhard Schölkopf, and Gert RG Lanckriet. On integral probability metrics, ϕ -divergences and binary classification. arXiv preprint arXiv:0901.2698 , 2009.

Harald Cramér and Herman Wold. Some theorems on distribution functions. Journal of the London Mathematical Society , 1(4):290-294, 1936.

Julien Rabin, Gabriel Peyré, Julie Delon, and Marc Bernot. Wasserstein Barycenter and its application to texture mixing. In Scale Space and Variational Methods in Computer Vision: Third International Conference, SSVM 2011, Ein-Gedi, Israel, May 29-June 2, 2011, Revised Selected Papers 3 , pages 435-446. Springer, 2012.

Ishan Deshpande, Yuan-Ting Hu, Ruoyu Sun, Ayis Pyrros, Nasir Siddiqui, Sanmi Koyejo, Zhizhen Zhao, David Forsyth, and Alexander G Schwing. Max-sliced Wasserstein distance and its use for GANs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10648-10656, 2019.

Natalia Caporale and Yang Dan. Spike timing-dependent plasticity: a Hebbian learning rule. Annu. Rev. Neurosci. , 31:25-46, 2008.

Katie A Ferguson and Jessica A Cardin. Mechanisms underlying gain modulation in the cortex. Nature Reviews Neuroscience , 21(2):80-92, 2020.

Julijana Gjorgjieva, Guillaume Drion, and Eve Marder. Computational implications of biophysical diversity and multiple timescales in neurons and synapses for circuit performance. Current Opinion in Neurobiology , 37:44-52, 2016.

Alison I Weber, Kamesh Krishnamurthy, and Adrienne L Fairhall. Coding principles in adaptation. Annual Review of Vision Science , 5:427-449, 2019.

D O Hebb. The Organization of Behavior: A Neuropsychological Theory . Wiley & Sons, 1949.

Gina G Turrigiano. The self-tuning neuron: synaptic scaling of excitatory synapses. Cell , 135 (3):422-435, 2008.

David Lipshutz, Cengiz Pehlevan, and Dmitri B Chklovskii. Interneurons accelerate learning dynamics in recurrent neural networks for statistical adaptation. In The Eleventh International Conference on Learning Representations , 2023.

Lyndon R Duong, David Lipshutz, David J Heeger, Dmitri B Chklovskii, and Eero P Simoncelli. Adaptive whitening in neural populations with gain-modulating interneurons. Proceedings of the 40th International Conference on Machine Learning, PMLR , 202:8902-8921, 2023.

Yang Dan, Joseph J Atick, and R Clay Reid. Efficient coding of natural scenes in the lateral geniculate nucleus: experimental test of a computational theory. Journal of Neuroscience , 16 (10):3351-3362, 1996.

Xaq Pitkow and Markus Meister. Decorrelation and efficient coding by retinal ganglion cells. Nature Neuroscience , 15(4):628-635, 2012.

Michael S Lewicki. Efficient coding of natural sounds. Nature Neuroscience , 5(4):356-364, 2002.

Bernhard Wegmann and Christoph Zetzsche. Statistical dependence between orientation filter outputs used in a human-vision-based image code. In Visual Communications and Image Processing'90: Fifth in a Series , volume 1360, pages 909-923. SPIE, 1990.

Siwei Lyu. Dependency reduction with divisive normalization: Justification and effectiveness. Neural Computation , 23(11):2942-2973, 2011.

Trevor Hastie, Robert Tibshirani, Jerome H Friedman, and Jerome H Friedman. The Elements of Statistical Learning: Data Mining, Inference, and Prediction , volume 2. Springer, 2009.

Ralph Linsker. Self-organization in a perceptual network. Computer , 21(3):105-117, 1988.

Gal Chechik, Amir Globerson, Naftali Tishby, and Yair Weiss. Information bottleneck for Gaussian variables. Advances in Neural Information Processing Systems , 16, 2003.

JH Van Hateren. Processing of natural time series of intensities by the visual system of the blowfly. Vision Research , 37(23):3407-3416, 1997.

Sonya Giridhar, Brent Doiron, and Nathaniel N Urban. Timescale-dependent shaping of correlation by olfactory bulb lateral inhibition. Proceedings of the National Academy of Sciences , 108(14):5843-5848, 2011.

Andrea Benucci, Aman B Saleem, and Matteo Carandini. Adaptation maintains population homeostasis in primary visual cortex. Nature Neuroscience , 16(6):724-729, 2013.

Adrian A Wanner and Rainer W Friedrich. Whitening of odor representations by the wiring diagram of the olfactory bulb. Nature Neuroscience , 23(3):433-442, 2020.

Ariana R Andrei, Alan E Akil, Natasha Kharas, Robert Rosenbaum, Krešimir Josi´ c, and Valentin Dragoi. Rapid compensatory plasticity revealed by dynamic correlated activity in monkeys in vivo. Nature Neuroscience , 2023.

Eero P Simoncelli and William T Freeman. The steerable pyramid: A flexible architecture for multi-scale derivative computation. In Proceedings., International Conference on Image Processing , volume 3, pages 444-447. IEEE, 1995.

David J Field. Wavelets, vision and the statistics of natural scenes. Philosophical Transactions of the Royal Society of London. Series A: Mathematical, Physical and Engineering Sciences , 357(1760):2527-2542, 1999.

Eastman Kodak Company. True color Kodak images dataset, 1993. URL https://www.math. purdue.edu/~lucier/PHOTO_CD/ .

Eero P Simoncelli. Statistical models for images: Compression, restoration and synthesis. In Conference Record of the Thirty-First Asilomar Conference on Signals, Systems and Computers , volume 1, pages 673-678. IEEE, 1997.

Siwei Lyu and Eero P Simoncelli. Nonlinear extraction of independent components of natural images using radial gaussianization. Neural Computation , 21(6):1485-1519, 2009.

Robbe LT Goris, J Anthony Movshon, and Eero P Simoncelli. Partitioning neuronal variability. Nature Neuroscience , 17(6):858-865, 2014.

Daniel B Rubin, Stephen D Van Hooser, and Kenneth D Miller. The stabilized supralinear network: a unifying circuit motif underlying multi-input integration in sensory cortex. Neuron , 85(2):402-417, 2015.

Matthew Chalk, Paul Masset, Sophie Deneve, and Boris Gutkin. Sensory noise predicts divisive reshaping of receptive fields. PLoS Computational Biology , 13(6):e1005582, 2017.

Jesús Malo, José Juan Esteve-Taboada, and Marcelo Bertalmío. Cortical divisive normalization from Wilson-Cowan neural dynamics. Journal of Nonlinear Science , 34(2):1-36, 2024.

Veronika Koren and Sophie Denève. Computational account of spontaneous activity as a signature of predictive coding. PLoS computational biology , 13(1):e1005355, 2017.

Alireza Alemi, Christian Machens, Sophie Deneve, and Jean-Jacques Slotine. Learning nonlinear dynamics in efficient, balanced spiking networks using local plasticity rules. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 32, 2018.

Gabrielle J Gutierrez and Sophie Denève. Population adaptation in efficient balanced networks. eLife , 8:e46926, September 2019.

Vincent Stimper, David Liu, Andrew Campbell, Vincent Berenz, Lukas Ryll, Bernhard Schölkopf, and José Miguel Hernández-Lobato. normflows: A PyTorch package for normalizing flows. Journal of Open Source Software , 8(86):5361, 2023.

Andrea Apicella, Francesco Donnarumma, Francesco Isgrò, and Roberto Prevete. A survey on modern trainable activation functions. Neural Networks , 138:14-32, 2021.

Aapo Hyvärinen and Erkki Oja. Simple neuron models for independent component analysis. International Journal of Neural Systems , 7(06):671-687, 1996.

A Analysis of circuit transform

In this section, we analyze the circuit transform T ( · ) in the case that µ ≥ 0 (i.e., λ ≥ -1 ), g i ≥ 0 and ϕ ( θ i , · ) are convex so that L ( W , θ , g , s , · ) is convex and

When µ > 0 , we can solve for the equilibrium responses using the fixed point iteration:

where f ( θ , z ) := ( f ( θ 1 , z 1 ) , . . . , f ( θ K , z K )) denotes the vector obtained by applying the function f elementwise to the pairs ( θ i , z i ) and ' ◦ ' denotes the elementwise (Hadamard) product.

A.1 Invertibility of the transform

If the constraint functions are twice continuously differentiable, then T ( s ) is continuously differentiable and we can relate the response density p r to the signal density p s as follows:

where J T denotes the Jacobian of T with respect to s . In general, we don't have a closed form solution for T ; however, setting the update in eq. (4) to zero, we see that the inverse transform T -1 (when it is well-defined) satisfies

where we have used the fact that the partial derivative of ϕ ( θ, z ) with respect to z is equal to the partial derivative of h ( θ, z ) with respect to z . It follows that its Jacobian J T -1 satisfies

where a i ( r ) := ∂ 2 ϕ ( θ i , r · w i ) /∂z 2 . Provided that either (a) µ > 0 or (b) g i > 0 and ϕ ( θ i , · ) is strictly convex for all i and the column vectors of W span R N , then the Jacobian of T -1 is positive definite everywhere. This implies that the Jacobian of T is positive definite everywhere, and thus T is invertible everywhere.

A.2 Relation to function class

Equation (7) establishes a relationship between the set of parameterizable (inverse) transforms and the function class { h ( θ, · ) } . To better understand this relationship, consider the scalar setting N = K = 1 in which case the optimal transformation is given by T = F -1 marginal ◦ F s , where F target and F s are the cumulative distribution functions (cdfs) of p marginal and p s , respectively. When T = F -1 marginal ◦ F s , it follows from eq. (7) that h ( θ, z ) satisfies

The multidimensional setting is more complicated; however, in the case that the signal is an orthogonal mixture of N independent sources, we can derive a precise relationship between the signal density and ( W , θ , g ) Consider the case that the signal is of the form s = Au , where A is an N × N orthogonal mixing matrix and u = ( u 1 , . . . , u N ) has statistically independent coordinates. Suppose W = A ⊤ . After left multiplying eq. (7) on both sides by W ⊤ , we get

If r is Gaussian, then so is z = W ⊤ r and

Therefore, an optimal solution to eq. (2) is when { θ i } and { g i } satisfy

where F u i is the cumulative distribution function for u i .

B Activation functions

In this section, we discuss the activation functions used for the experiments carried out in Sec. 4 of the main text.

B.1 Choice of activation functions

How should we choose the family of activation functions { f ( θ, · ) } ? As stated in the main text, one approach is the choose a family that is well matched to the marginal statistics of the inputs. In this case, the input marginals are well approximated by generalized Gaussian distributions of the form

with varying scale α and shape β . Here Γ is the gamma function. Therefore, from eq. (8), we see that an optimal family of activation functions { f ( θ, · ) } is such that for each choice of scale α and shape β , there is a gain g and parameter θ such that

where F α,β and Φ denote the cdfs of the generalized Gaussian distribution and the standard Gaussian distribution. In general, defining the family of activation functions directly in terms of the above display leads to challenges implementing Alg. 1 since ϕ ( θ, z ) and ∇ θ ϕ ( θ, z ) are not readily computable.

We instead sought a simple algebraic expression that approximates F -1 α,β ◦ Φ( z ) . First, suppose the activation function takes the form f ( θ, z ) = sign ( z ) | z | θ . When µ = 0 , this activation function is optimal for a signal whose cdf F monomial satisfies

Rearranging and differentiating with respect to z , we find that the pdf of the signal p monomial satsifies

Substituting in with s = g sign ( z ) | z | θ , we find the following expression for the pdf

where q = 1 /θ . This closely resembles aspects of the pdf for the tails of the generalized Gaussian distribution. To reshape the distribution local when s ≈ 0 , we included a linear term, resulting in the activation function

Here a ( θ ) and b ( θ ) are nonnegative functions given by

The forms of a ( θ ) and b ( θ ) were chosen to approximately minimize min θ max z | F s ( gf ( θ, z )) -Φ( z ) | when F s is the cdf of a generalized Gaussian distribution with shape parameter β between 0.2 and 1.

B.2 Activation function updates

Here we derive the updates to the activation functions. Recall from Alg. 1 that the θ update is given by θ ← θ + η θ ∇ θ ϕ ( θ, z ) , where

and C ( p ) is the absolute p -moment of a scalar Gaussian random variable:

Note that when θ = 3 , the constraint function normalizes the kurtosis of the marginal and the resulting update bears similarity to ones used in neural models of ICA [82]. Differentiating with respect to θ , we obtain the updates

where

ψ (0) ( p ) is the polygamma function and we have used the fact that the derivative of the gamma function is Γ ′ ( p ) = Γ( p ) ψ (0) ( p ) .

C Experiments

C.1 Experimental set up

We ran our experiments on a cluster comprised of 40-core Intel Skylake nodes with 768GB of RAM. Each experiment shown in Fig. 2 (i.e., Gaussianization of the coefficients from 1 image) took less than 5 minutes to run. Each experiment shown in Fig. 3 took around 5 hours to run (we did not optimize the choice of hyper-parameters).

C.2 Additional experimental results

In Fig. 5, we provide additional examples of histograms of local filter responses and optimized responses for various images from the Kodak dataset [70] (see Fig. 2 of the main text for a detailed description). We note that when the shape parameter β of the fitted generalized Gaussian distribution is small, the distribution of responses has a characteristic dip near zero. This is likely due to the fact that when β is small, there is more probability mass concentrated near zero and so small discrepancies between the learned activation function gf ( θ, z ) and the optimal activation function F -1 α,β ◦ Φ( z ) when z ≈ 0 can lead to large discrepancies between the response distribution and N (0 , 1) near zero. This could potentially be resolved by choosing a parameterization of gf ( θ, z ) that better approximates the optimal activation function.

In Fig. 6, we provide additional examples of contour plots of local filter responses and optimized responses for three additional images from the Kodak dataset (corresponds to top three rows of Fig. 5, see Fig. 3 of the main text for a detailed description). In some of these examples we see that unlike the standard Gaussian distribution N ( 0 , I ) the learned response distribution is clearly non-monotone with respect to ∥ r ∥ . This non-monotonicity is likely inherited from non-optimality of the interneuron activation functions that leads to the discrepancies in the scalar case (see Fig. 5).

natural image

6

4

2

0

2

4

6

10

3

10

2

10

1

10 0

signal density

ps

= 0.36,

= 0.73

6

4

2

0

2

4

6

6

4

2

0

2

4

6

interneuron act.

gf

(

,

z

)

g

= 0.27,

= 2.2

6

4

2

0

2

4

6

6

4

2

0

2

4

6

circuit transform

r

=

T

(

s

)

6

4

2

0

2

4

6

10

3

10

2

10

1

10 0

response density

pr

6

4

2

0

2

4

6

10

3

10

2

10

1

10 0

= 0.17,

= 0.59

6

4

2

0

2

4

6

6

4

2

0

2

4

6

g

= 0.06,

= 2.42

6

4

2

0

2

4

6

6

4

2

0

2

4

6

6

4

2

0

2

4

6

10

3

10

2

10

1

10 0

6

4

2

0

2

4

6

10

3

10

2

10

1

10 0

= 0.33,

= 0.72

6

4

2

0

2

4

6

6

4

2

0

2

4

6

g

= 0.2,

= 2.24

6

4

2

0

2

4

6

6

4

2

0

2

4

6

6

4

2

0

2

4

6

10

3

10

2

10

1

10 0

6

4

2

0

2

4

6

10

3

10

2

10

1

10 0

= 0.08,

= 0.48

6

4

2

0

2

4

6

6

4

2

0

2

4

6

g

= 0.07,

= 2.39

6

4

2

0

2

4

6

6

4

2

0

2

4

6

6

4

2

0

2

4

6

10

3

10

2

10

1

10 0

6

4

2

0

2

4

6

10

3

10

2

10

1

10 0

= 0.05,

= 0.45

6

4

2

0

2

4

6

6

4

2

0

2

4

6

g

= 0.03,

= 2.54

6

4

2

0

2

4

6

6

4

2

0

2

4

6

6

4

2

0

2

4

6

10

3

10

2

10

1

10 0

6

4

2

0

2

4

6

10

3

10

2

10

1

10 0

= 0.21,

= 0.63

6

4

2

0

2

4

6

6

4

2

0

2

4

6

g

= 0.08,

= 2.38

6

4

2

0

2

4

6

6

4

2

0

2

4

6

6

4

2

0

2

4

6

10

3

10

2

10

1

10 0

6

4

2

0

2

4

6

10

3

10

2

10

1

10 0

= 0.13,

= 0.55

6

4

2

0

2

4

6

6

4

2

0

2

4

6

g

= 0.05,

= 2.44

6

4

2

0

2

4

6

6

4

2

0

2

4

6

6

4

2

0

2

4

6

10

3

10

2

10

1

10 0

10

3

10

2

10

1

10 0

= 0.0,

= 0.22

6

4

2

0

2

4

6

g

= 0.01,

= 2.63

6

4

2

0

2

4

6

10

3

10

2

10

1

10 0

EHE

6

4

2

0

2

4

6

6

4

2

0

2

4

6

6

4

2

0

2

4

6

6

4

2

0

2

4

6

A) signal

B)ZCA

MI=2.36e-1

MI=1.15e-1

MI=2.e-2

MI=1.76e-2

MI=9.91e-3

2

MI=9.e-2

MI=7.68e-2

MI=6.95e-2

Ml=1.15e-2

MI=1.32e-2

d=8

MI=3.39e-2

MI=2.89e-2

MI=7.83e-3

MI=7.97e-3

d=32

A) signal

B)ZCA

C)responses(K=2)

D)responses(K=3)E)responses(K=4)

MI=2.39e-1

MI=1.41e-1

MI=1.06e-1

MI=4.02e-2

MI=3.45e-2

d=2

MI=1.07e-1

MI=9.19e-2

MI=7.81e-2

MI=2.87e-2

MI=2.89e-2

=8

d

MI=4.65e-2

MI=4.65e-2

MI=3.35e-2

MI=2.59e-2

d=32

A) signal

B)ZCA

C)responses(K=2)D)responses(K=3)E)responses(K=4)

MI=1.81e-1

M1=1.11e-1

MI=4.81e-2

MI=1.94e-2

MI=1.75e-2

2

=p

MI=7.49e-2

MI=7.46e-2

MI=6.53e-2

MI=1.41e-2

MI=1.37e-2

d=8

MI=3.08e-2

MI=2.39e-2

MI=8.46e-3

MI=8.99e-3

d=32

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Our claims in the abstract and introduction match the theoretical and experimental results in the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We include limitations of our work and potential alternative approaches in the Discussion section.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: While our paper does not include proofs, we do include full details of our analytical results.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We describe the implementation of our algorithm and include a link to code for reproducing our results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We link to a GitHub repository with example code for reproducing the experi- ments described in the paper.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provide full details of our experiments.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We include error bars.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We include the amount of compute used and runtime for the experiments (see Appx. C).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Our research conforms with the NeurIPS Code of Ethics.

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: We do not foresee an immediate societal impact of our work.

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We cite the Kodak dataset and include a URL to the asset.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Peer reviews:
comments: SUMMARY:
This paper proposes a normative recurrent circuit to solve an optimal transport problem, focusing on the problem of Gaussianization of natural image statistics.

STRENGTHS:
This paper was a pleasure to read. The writing is clear, the formulation of the problem elegant, and the results represent a clear advance relative to past works on whitening circuits.

WEAKNESSES:
I have only a few small concerns regarding the biological realism of the model, but these are mostly already acknowledged by the authors. One point that I believe warrants further discussion is the plausibility of the activation functions (6). These are somewhat reminiscent of two-sided versions of the rectified-power law activations used in cortical models, no? I also have a handful of miscellaneous questions (see below), but on the whole these do not dampen my enthusiasm.

QUESTIONS:
- Along with the work of Van Hateren, the authors might consider citing Laughlin, "A simple coding procedure enhances a neuron's information capacity," and Juusola and Hardie, "Light Adaptation in Drosophila Photoreceptors." I leave it to their discretion whether or not to do so.

- In that vein, it might be nice to provide further evidence for adaptation of activation functions in biology. If additional references occur to me, I will update my review. 

- In Line 271, should "activations" be plural? I think this is a typo. 

- The legibility of Figure 3A could be improved; as it stands the text is a bit too small and the lines a bit too faint. 

- It could be nice to provide further support for the approximate form of the activation functions. Can you numerically evaluate the expression below Line 630?

LIMITATIONS:
As noted above, the authors clearly discuss the limitations of their work.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-06-14

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates a crucial question in neuroscience: how do neural circuits convert inputs into a target distribution, specifically focusing on how local interneurons transform natural image statistics into a spherical Gaussian distribution. The authors approach this problem through the lens of optimal transport. By using an integral probability metric, the task of transforming the input distribution into a spherical Gaussian distribution is framed as a minimax optimization problem. This constrained minimax optimization translates to recurrent neural dynamics (representing the minimization part) combined with gain modulation, activation function adaptation, and plasticity (representing the maximization part), with plasticity being local and Hebbian. Notably, the study's principled adjustment of activation functions is a unique contribution. The authors present experimental results on natural images using wavelet transformations.\n\nSTRENGTHS:\nNote: As I am not well-versed in wavelet transformations, which are extensively utilized in the experimental section, my review focuses primarily on the theoretical aspects prior to the experiments (Section 4).\n\n**Originality**: This paper tackles a significant question in neuroscience: how neural circuits transform input signals into a target distribution. The authors frame this problem as an optimal transport problem. Utilizing the integral probability metric, they demonstrate that this problem is equivalent to a constrained minimax optimization, which can be interpreted through neural circuit dynamics. The paper's theoretical contributions are both solid and original.\n\n**Quality**: The work is theoretically robust, with well-justified methods and experiments that effectively validate the theoretical model. The use of spherical Gaussian distributions, which might initially appear to be a limitation, is convincingly justified in Section 4.\n\n**Clarity**: The paper is clearly written overall.\n\n**Significance**: This paper introduces a normative model for normalizing inputs, offering potentially valuable insights for the theoretical neuroscience community.\n\nWEAKNESSES:\n- The authors should define $p_{target}$, $p_{marginal}$, and $p_r$ explicitly instead of using descriptive terms for better clarity.\n- The proposed model necessitates different $g$, $\\theta$, and $w$ for different stimuli. While neurons can potentially adjust their gain and activation function (i.e., $g$, $\\theta$) relatively quickly, the need for weight updates to handle varying inputs significantly limits the model's applicability and biological plausibility.\n- As noted in the limitations section, the weights are not sign-constrained. Therefore the distinguishing between excitatory and inhibitory neurons in this model makes no sense.\n\nQUESTIONS:\nThe constraint $g_i > 0$ is not included in Algorithm 1. Would imposing these constraints ($w > 0$, $g > 0$) negatively impact the experimental results?\n\nLIMITATIONS:\nThe authors have adequately addressed the limitation.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nAuthors propose an online algorithm that solves the problem of optimal transport. In particular, assuming a spherical distribution of stimuli, the goal of the algorithm is to generate neural responses such that their distribution best approximates the distribution of stimuli. Authors find that a neural network with excitatory and inhibitory neurons can solve such optimization problem, by learning a non-linear transformation that reduces dependency between neural responses. This is done through Hebbian learning on synapses and by adjusting activation functions of single neurons.\n\nSTRENGTHS:\nThe paper presents is a novel combination of well known techniques and provides an alternative to existing approaches. The work seems technically sound and authors describe some interesting relation to similar approaches. Proposed algorithm is a nonlinear extension of existing algorithms for data whitening using a neural network.\n\nWEAKNESSES:\nThe paper has the ambition of finding a neural implementation of generalized whitening such that it could be also implemented by biological neural circuits. However, a major concern is that inhibitory neurons do not obey Dale's law, and the model is therefore not biologically plausible. In general, it is not clear what the paper brings to understanding of biological or artificial neural networks.\nSince the model violates Dale's law, the discussion about the function of interneurons seems out of scope.\n\nMoreover, there are significant parts of the paper that are unclear. A number of details about the methods are given, but the main mechanism that supports learning remains unclear. Authors mention that the circuit \"learns directions that are least matched to the target distribution\", but it remained unclear to me how is this achieved and why this helps to Gaussianize the distribution of output firing rates of principal neurons. \n\nAuthors motivate their model by claiming that divisive normalization has not yet been solved with a neural network. This seems false (see for example Chalk, Masset, Deneve & Gutkin,  PLOS CB 2017). Gain control is also captured by available models. Efficient spiking models and normative models where changes in gain control can be captured by modulating the metabolic cost on spiking in the model's objective (see Guiterrez and Deneve, eLife 2019; Koren and Deneve, PLOS CB 2017). In general, the paper does not sufficiently take into account previous work on efficient coding with spikes that is closely related.  \n\nThere are typos on several places, for example: lines 75, 77, 100, 234\n\nQUESTIONS:\nAuthors motivate their modelling by claiming that it is currently unclear how neural networks could implement nonlinear transformations. Which non-linear transformations are not captured by the listed models?\nThe paper by Alemi et al. AAAI 2018, proposes efficient spiking models that implement non-linear transformation of stimuli at the population level. In Koren & Panzeri, NeurIPS 2022, neural populations perform linear transformation of stimuli on the level of neural populations, but on the level of single neurons these transformations are strongly non-linear. What do authors mean by non-linear transformations? Non-linear on what level? This should be clarified.\n\nAuthors say that approximating target distribution is useful because it facilitates efficient transmission of signals. It is however not entirely clear why making the distribution of responses Gaussian-like facilitates efficient transmission. Is this improvement in efficient transmission attributed to redundancy reduction?\n\nIs inhibition required to solve this optimization problem, e.g., could the same operation be achieved through plasticity of E neurons?\n\nLIMITATIONS:\nAuthors addressed the limitations adequately.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a method that utilizes multiple inhibitory neurons and the Hebbian learning rule to transform a signal into a representation that conforms to a specified target distribution. Specifically, the model incorporates Hebbian synaptic plasticity to establish connections that optimally match this target distribution. Interneurons within the model are adapted in terms of their gain and activation directions to respond most effectively to the input signals, mirroring the potential optimization mechanisms found in biological neural processing.\n\nUtilizing this approach, the study processes natural images from the Kodak dataset through a wavelet transform. This extracts 2-dimensional signals, specifically pairs of wavelet coefficients for images at fixed horizontal spatial offsets ranging from d = 2 to d = 64. The model then transforms the distribution of this two-dimensional information into a Gaussian distribution. This transformation demonstrates the model's ability to handle complex data structures and align them with statistically predictable patterns, enhancing both the analysis and interpretation of natural images.\n\nSTRENGTHS:\n1. Clear Presentation: The paper articulates its concepts with exceptional clarity, facilitating a deep understanding of complex models and their applications.\n\n2. Effective Transformation of Distributions: The model is highly effective at converting input signals into specific, targeted distributions, optimizing data for further processing and analysis.\n\n3. Online Operational Capability: One of the model's significant advantages is its ability to operate online. This feature significantly enhances its practicality for real-world applications, allowing for real-time data processing and continuous learning without the need for retraining.\n\nWEAKNESSES:\n1.  A main concern is that the model is not biologically realistic. In addition to the authors' admission that the feedforward synaptic weights $W^T$ and feedback weights $-W$ are symmetrically constrained, there is also a question about the biological plausibility of using pairs of wavelet coefficients as inputs. It's unclear how biological systems would naturally derive such information and whether the distribution of these inputs is something that biological systems need to transform.\n\n2. The setup of online learning with a batch size of 10 also raises questions about its biological feasibility. It's uncertain how biological systems could implement a similar mechanism.\n\n3. The paper only demonstrates the method's effectiveness on two-dimensional inputs, which might not sufficiently prove its efficacy. In real-world scenarios, we often deal with inputs that are high-dimensional ($N>>2$), and the model's performance in such conditions remains untested.\n\n4. The paper incorporates many specific settings, such as the choice of activation functions based on the Gaussianization of scalar signals. These particular choices may limit the generalizability and applicability of the model to different datasets or broader applications.\n\nQUESTIONS:\n1. As mentioned in the Weaknesses section, the model is not biologically realistic. Given this, my primary question is: What is the purpose of using neural networks to implement this function? Furthermore, if it does not realistically mimic biological processes, should it be compared with more engineering-oriented approaches?\n\n2. Scaling of Inhibitory Neurons with Input Dimensionality: The model currently demonstrates with two-dimensional inputs, requiring three inhibitory neurons. How does the number of required inhibitory neurons scale as the dimensionality of the input increases? This is crucial for understanding the feasibility and complexity of the model when applied to higher-dimensional data.\n\nLIMITATIONS:\nYes, the authors have addressed some of the limitations of their work in the Discussion section of the paper. They specifically mention the model’s lack of biological realism and its primary demonstrations within low-dimensional settings. These acknowledgments align with the checklist guidelines on discussing limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}