Brain-like Variational Inference
Authors: Hadi Vafaii
Dekel Galor
Jacob L. Yates
Venue: NeurIPS 2025 poster ()
Keywords: variational inference
free energy
ELBO
sparse coding
predictive coding
iterative inference
VAE
Primary area: probabilistic_methods
TL;DR: We derive brain-like inference as natural gradient descent on free energy (FOND). We apply FOND to derive a novel spiking network (iP-VAE) that outperforms amortized VAEs in reconstruction-sparsity trade-offs and out-of-distribution generalization.

Abstract: Inference in both brains and machines can be formalized by optimizing a shared objective: maximizing the evidence lower bound (ELBO) in machine learning, or minimizing variational free energy ($\mathcal{F}$) in neuroscience (ELBO = $-\mathcal{F}$). While this equivalence suggests a unifying framework, it leaves open how inference is implemented in neural systems. Here, we introduce FOND (*Free energy Online Natural-gradient Dynamics*), a framework that derives neural inference dynamics from three principles: (1) natural gradients on $\mathcal{F}$, (2) online belief updating, and (3) iterative refinement. We apply FOND to derive iP-VAE (*iterative Poisson variational autoencoder*), a recurrent spiking neural network that performs variational inference through membrane potential dynamics, replacing amortized encoders with iterative inference updates. Theoretically, iP-VAE yields several desirable features such as emergent normalization via lateral competition, and hardware-efficient integer spike count representations. Empirically, iP-VAE outperforms both standard VAEs and Gaussian-based predictive coding models in sparsity, reconstruction, and biological plausibility, and scales to complex color image datasets such as CelebA. iP-VAE also exhibits strong generalization to out-of-distribution inputs, exceeding hybrid iterative-amortized VAEs. These results demonstrate how deriving inference algorithms from first principles can yield concrete architectures that are simultaneously biologically plausible and empirically effective.

Brain-like Variational Inference
Hadi Vafaii
Dekel Galor
Jacob L. Yates!
co-first authors, correspondence to va wafaii@@erkelee..... edu
Redwood Center for Theoretical Neuroscience, UC Berkeley
Abstract
Inference both brains and machines can be formalized by optimizing a shared objective: maximizing the evidence lower bound (ELBO) in machine learning, minimizing variational free energy (F) in neuroscience (ELBO = F). While this equivalence suggests unifying framework, it leaves open how inference is implemented in neural systems. Here, we introduce FOND (Eree energy Online Natural- -raaient Dynamics), a framework that derives neural inference dynamics from three principles: (1) natural gradients on F, (2) online belief updating, and (3) iterative refinmment. We apply FOND to derive iP-VAE (iterative Poisson vari- ational autoencoder). recurrent spiking neural network that performs variational inference through membrane potential dynamics, replacing amortized encoders with iterative inference updates. Theoretically, ip-VAE yields several desirable features such as emergent normalization via lateral competition, and hardware- efficient integer spike count representations. Empirically, iP- outperforms both standard VAEs and Gaussian-basee predictive coding models in sparsity, re- construction, and biological plausibility, and scales to complex color image datasets such as CelebA. iP-VAE also exhibits strong generalization to out-of-distribution inputs, exceeding hybrid amartized VAEs. These results demonstrate how deriving inference algorithms from first principles can yield concrete architectures that simultaneously biologically plausible and empirically effective.
b bottom-up "descriptive" approach Manually add terms to equations of motion until the resulting dynamics matches observed data
u(t++1] 1) = u(t) + mi(t) &t
State space
W E R.K
Uk k
Governing Dynamies?
potentials
u == logr
(
top-down "prescriptive" approach
Instantaneous firing rates
dynamics
principles
v = u((-zz))
Inference: :
r(t) (x)()(t))
- Principle:
Mininize variatonal free energy, J
convergence to altractor
Prescriptions: (1) Poisson dist butions (2) Onlino inforence
=(1) ~ Pois(z;r(1)
(3) Natural gradient updatos
Comportte
Oomenia Octaa Ladee
"(v/V, Fin. a]
Neural Trajectory
integer spike counts
* Result
spkking neural network wi stochaslic integrate-annten neurons & lateral competition
wit)
x Tx TT)((t) B(u(t)
U
U1
feedforward
recurrent
homeostatic
"drive"
"explaining away"
Figure 1: Inferential and dynamical accounts of perception unified under variational inference. (a) Perception framed as a dynamical process convergence to attractors in a neural state space, where membrane potentials evolve and generate spikes along the way. (b) Our prescriptive approach derives neural dynamics by minimizing free energy via natural gradient descent, yielding a spiking network with lateral competition. The resulting architectures are principled and empirically effective. Code, data, and model checkpoints are available here:
VAE
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Introduction
Artificial intelligence and theoretical neuroscience share a foundational principle: understanding the world requires inferring hidden causes behind sensory observations (early work: [T3 modern interpretations: (4))). This principle is formalized through Bayesian posterior inference [10 12].
with variational inference 4 [3-17] offering an optimization- based approximation. Variational inference employs an identical objective function across both fields, albeit under different names. In machine learning, it is called evidence lower bound (ELBO), which forms the basis of variational autoencoders (VAEs; [1822T]; whereas, neuroscience refers to it as variational free energy (F), which subsumes predictive coding [22] [3] and is presented as unified theory of brain function (2
ELBO is exactly equal to negative free energy (ELBO = - F; [25). D. and this mathematical equivalence offers powerful common language for describing both artificial and biological intelligence (26).
Despite this promising connection, neither ELBO nor F have proven particularly prescriptive for developing specific algorithms architectures [2 28 Instead, the pattern often works in reverse:
effective methods are discovered empirically, then later recognized as instances of these principles.
In machine learning, diffusion models were originally motivated by non-equilibrium statistical mechanics (29]. and only later understood as a form of BIBOO maximization (30 31). Similarly, in
neuroscience, predictive coding (PC; (22) was first proposed as a heuristic model for minimizing prediction errors (32), before being reinterpreted as F minimization
[23 33). This pattern of post-hoc
theoretical justification, rather than theory-driven derivation, is common across both disciplines.
Then why pursue a general theory at all? A unifying framework offers several advantages: clarifies the design choices underlying successful models, explains why they work, and guides the development of new models. The recent Bayesian learning rule (BLR; (34) exemplifies this potential. It re- ci-interrrets broad class of learning algorithms instances of variational inference, optimized via natural gradient descent (35) over approximate posteriors 9x- By varying the form of qΛ and the associated approximations, BLR not only recovers wide range of algorithms-frrom SGD and Adam to Dropout- but, crucially, it also offers a principled recipe for designing new ones (34). Thus, BLR transforms post-hoc theoretical justification into a prescriptive framework for algorithm design.
In variational inference, the prescriptive choices are specifying Qx and how to optimize the variational parameters, A. The machine learning community has embraced amortized inference (18) 19 36-38),
where a neural network is trained to produce À in a single forward pass for each input sample. In contrast, neuroscience models have traditionally favored iterative inference methods [22] 39]. which align more naturally with the recurrent [40 and adaptive [45 49) processing observed in cortical circuits (but see Gershman and Goodman (36] for counterpoint). Despite this fundamental divide in methodology, systematic comparisons between iterative and amortized inference variants remain scarce, particularly when evaluated against biologically relevant metrics.
In this paper, we introduce Free energy Online Natural- -rauiillg Dynamics (FOND), a framework for deriving brain-like inference dynamics from F minimization. As concrete application of FOND, we derive a novel family of iterative VAE architectures that perform inference via natural gradient descent on F-including a spiking variant, the iterative Poisson VAE (PP-VAE)wild with distinct computational and biological advantages. The paper is organized as follows:
In section 2 and the associated appendix A we review and synthesize existing models from machine learning and neuroscience, showing how they can be unified under the variational inference framework via specific choices of distributions and inference methods (Fig. 2
In sections 3 and 4 and the associated appendix B we derive neural dynamics as natural gradient descent free energy. This yields novel family of iterative VAEs, including the iP-VAE, which performs online Bayesian inference through membrane potential dynamics, with emergent lateral competition, explaining away dynamics, and divisive normalization.
In section we show that iterative inference consistently outperforms amortized methods, with iP-VAE achieving the best reconstruction-spariity trade-off, while using 25x fewer parameters and integer-valued spike counts. Appendix C shows that iP-VAE: (i) exhibits emergent cortical response properties, (ii) generalizes better to out-of dist-iiiibttinn data, and (ii) scales to complex color image datasets such as ImageNet32, CIFAR- 10, and CelebA.
In section 6 and the associated appendix DD. we relate our results to recent advances in sequence modeling, and discuss how expressive nonlinearities, emergent normalization, and effective stochastic depth contribute to iP-VAE's computational strengths.

---

2 Background and Related Work
Notation and conventions. We start with a generative model that assigns probabilistic belief's to observations, æ E IRA (e.g.. images), through invoking K-dimensional latent variables, 2: po(=:) Po (2,, 2) f (z)) dz, where 0 are its adaptable parameters (e.g., neural network weights or synaptic connection strengths in the brain). Throughout this work, we color-code the generative and inference model components as blue and red, respectively.
Variational inference and the ELBO objective. Following the perception-ax-inieett 2]. we formalize perception Bayesian posterior inference. In appendix A..] we provide historical background, and in appendix [A.2] we review the challenges associated with posterior inference. We overcome these challenges by approximating the true but often intractable posterior, (=(æ)) using another distribution, go (=( referred to as the approximate (or variational) posterior.
framework
In appendix A.3] provide more details about how variational inference [5 17 approximates the
inference process, and derive the standard Evidence Lower BOund (ELBO) objective
13
pa (xx
log Pa
E (xla)) log
DEL (go || (z)
(z(zz)))
model evidence
Kullhack- (KI.) divergence
ELBO(2:9.0)
Importantly, maximizing the ELBO minimizes the intractable Kullback-Leible (KL) divergence between the approximate and true posterior. This can be seen by taking gradients of eq. (I) with respect to Φ. The model evidence on the left-hand side does not depend on o, making the gradients of
the two terms on the right additive inverses. Thus, ELBO maximization directly enhances the quality of posterior inference. In theoretical neuroscience, the negative ELBO referred to as variational free energy (FF), which is the mathematical quantity central to the free energy principle (24 25).
In the remainder of this section, we demonstrate how diverse models across machine learning and neuroscience emerge instances of F minimization through two fundamental design choices:
1. Choice of distributions (appendix A.4) approximate posterior qo (z(x)) (ii) prior ps (z), and (iii) likelihood (=(x)))
2. Choice of inference method (appendix (ii) iterative (e.g., gradient descent)
(i) amortized (e.g., learned neural network)
Variational Autoencoder (VAE) model family. Variational Autoencoders (VAEs) transform the abstract ELBO objective into practical deep learning architectures (18. The standard Gaussian VAE (G-VAE) exemplifies this approach by assuming factorized Gaussian distributions for all three distributions, with the approximate posterior do (2|zz) implemented as a neural network that maps
each input æ to posterior parameters: enc(æ; Φ) + (=(z),o²²x)) This amortization of inference using single network to approximate posteriors across the entire dataset-is a defining characteristic of VAEs. Alternative distribution choices are also possible; for instance, replacing both prior and posterior with Poisson distributions yields the Poisson VAE (P-VAE; [50]), which better aligns with neural spike-count statistics (31) We derive the VAE loss in appendix A.5 and discuss both
G-VAE and VAVAE extensively in appendices A.6 anddAndA A.7
Sparse coding and predictive coding as variational inference. Two major cornerstones of theoretical neuroscience, sparse coding (SC; (54]) and predictive coding (PC: [22). can also be derived as instances of ELBO maximization (or equivalently, minimization), given specific distributional choices 33 55 56). SC and PC share two key characteristics that distinguish them from standard VAES. First, they both use Dirac-delta distribution for the approximate posterior, effectively collapsing it to point estimate. But they differ in their prior assumptions: PC employs Gaussian prior, while SC uses a amortized inference as in VAžs, both PC and SC employ iterative inference, better aligning with the recurrent nature of neural computation (4). See appendix A.8 for an in-depth comparison, and appendix A.9 for pedagogical derivation of the Rao and Ballard 22 objective F minimization.
prior (e.g., Laplace; Fig. 2 Second, instead of
3

---

Variational Free Energy Minimization
approximate posterior?
prior?
approximate posterior?
Poisson
Gaussian
QA (z x))
Pe
QA
Poisson
Laplace
Gaussian
Dirac-Della
approximate posterior?
Predictive Coding
enooder?
(x(2(2)) x))
encoder?
(Rao & Ballard, 1999);
Iterative
Amortized
Dirac-Della
llerative
Amortized
iP-VAE
P-VAE (Vafali et 2024)
Sparse Coding
iG-VAE
Variational Autoencoder
(This paper)
(Olshausen Field, 1996)
(This paper)
(Kingma Welling, 2014)
Figure wide range of models across machine learning and theoretical neuroscience can be unified under free energy (F) minimization. Different distributional and optimization choices result in different models (appendix [A.10). Motivated by this unification potential, we introduce FOND, a framework for deriving brain-like inference algorithms from first principles (section 3). We apply FOND to derive a family of iterative VAE architectures, including the spiking rP-VAB (section 4).
Online inference through natural gradient descent. In a recent landmark paper, Khan and Rue (34] proposed the Bayesian learning rule (BLR), unifying seemingly disparate learning algorithms as instances of variational inference optimized via natural gradient descent [35 57] on Bayesian
objective which reduces to the ELBO when the likelihood is known [58] 591 BLR's generality extends naturally to sequential settings (60). where beliefs must be updated online as data arrives. In
such settings, rolling update scheme where the posterior at each step becomes the prior for the next (Fig. 7
provides a simple yet effective approach continual belief revision [60].
Summary so far. In this background section, and the corresponding appendix A we reviewed how seemingly different models across machine learning and neuroscience can be understood as instances of F minimization (Fig. 2) The differences among them arise from the choices we make on the distributions and the inference methods (appendix I0).
FOND: Free energy Online Natural-gradiee Dynamics
Building the unification potential shown above, we introduce FOND, framework for deriving inference dynamics from F minimization. Unlike post-hoc interpretations of existing models, FOND adopts top-down approach: it starts from theoretical principles and derives architectures, making explicit the connection between design choices and computational properties.
A two-level framework: flexible choices and fixed prescriptions. FOND operates two distinct levels. First, the modeler makes flexible choices about distributions and their parameterizations decisions that define the inference problem. Second, once these choices made, FOND applies fixed prescriptions that fully determine the inference dynamics.
This structure mirrors prescriptive theories in physics [61H64]. To derive dynamics from first principles, one must specify: (a) the dynamical variables of interest, and (b) the equations governing their evolution (see appendix B.T). In FOND, (a) corresponds to flexible choices, while (b) follows from fixed prescriptions.
Flexible choices what the modeler decides. The modeler must choose:
Distributions: Variational family and conditional likelihood (appendix A.4
Parameterization: Choice of dynamical variables that parameterize the variational distribution
These modeling decisions determine what is being optimized, but not how optimization proceeds.

---

- what FOND determines. Once the flexible choices are made, the inference dynamics are fully determined by FOND's three core prescriptions:
Fixed prescriptions- -
FOND prescriptions
Natural gradients: Dynamics follow natural gradients F
Online: Inference updates beliefs sequentially from streaming data
(3) Iterative: Inference refines estimates through recurrent updates
Why natural gradients? Natural gradient descent follows the steepest descent path in the space of
probability distributions [35]. achieving maximal free energy reduction per unit of distance (measured
via Fisher information metric 57 65 )5)). This provides the most efficient optimization trajectory.
Why online? Biological perception requires adaptive priors that evolve with experience. Static inference cannot account for phenomena like serial dependence, where recent stimuli systematically bias current perception [46], or learned helplessness, where beliefs about agency are updated by experience (66 677. Online inference enables time-evolving priors that capture such effects (Fig. 7
Why iterative? Perception follows an "unalysis-b--synteess
loop [68]: hypotheses generate predictions (synthesis). errors refine hypotheses (analysis). Iterative inference implements this closed-loop error-correction. adapting estimates when initial predictions fail. Amortized inference, by contrast, commits to a single forward pass-an an open-loop guess that cannot recover when emrs, making it brittle on novel or ambiguous inputs. The brain appears to implement this kind of iterative refinement: macaque visual cortex requires an additional ~30ms of recurrent processing to recognize challenging images [42], and neural codes for faces dynamically evolve over hundreds of milliseconds-a process termed "code switching" (44). Thus, iterative refinement motivated both computationally (enabling error correction) and empirically (matching neural dynamics).
Demonstrating FOND: iterative VAE architectures. To demonstrate FOND's utility, we derive three iterative VAE models by making different flexible choices while applying the same fixed prescriptions:
ip. -AE: Poisson distributions with log-rate parameterization
iG-VAF: Gaussian distributions with mean and log-std parameterization
iG, VAE: like Gaussian, but with post-sampling nonlincarity yp.))
See Table [ for complete specifications. In what follows, we present the theoretical derivations, evaluate empirical performance, and discuss implications for neuroscience and machine learning.
iP-VAE: Deriving a Spiking Inference Model from Variational Principles
In this section, we apply the FOND framework derive brain-like inference dynamics from first principles. We begin by identifying the key prescriptive choices required for deriving these dynamics. We then introduce Poisson assumptions and derive membrane potential updates as natural gradient descent variational free energy (F), leading to the iterative Poisson VAE (iP-VAE) architecture.
Three distributional choices. To fully specify F, we must choose approximate posterior, prior, and a conditional likelihood. We use Poisson distributions for both posterior and prior, as it leads to more brain-like integer-valued spike count representations (see appendix B.2 for discussion). Later the appendix, we show how this derivation extends to Gaussian posteriors with optional nonlinearities applied after sampling. illustrating the generality and flexibility of FOND.
For the likelihood, we assume factorized Gaussians throughout this work (a standard modeling choice in both sparse coding (54) and predictive coding (22). In the main text, for pedagogical clarity, we focus on linear decoders, (=( 2) N(x; 22, I). where is commonly referred to as dictionary in sparse coding literature. In the appendíx, we show how our results naturally extend to nonlinear decoders with learned variance [69], and we explore these extensions both theoretically and empirically.
5

---

Parameterization choices. For a neurally plausible model, we choose the dynamic variables to be real-valued membrane potentials, u R² where K is the number of neurons. To generate spike counts from u, we assume the canonical Poisson parameterization, defining firing rates as r= e== u) (Fig. [b) This choice is both mathematically convenient, since it avoids constrained optimization over strictly positive variable, and biologically motivated, as the spiking threshold of real neurons is well-approximated by expansive nonlinearities [70 71].
Under the assumption that conamiics,omppuuaioon and optimization are three expressions of the same underlying process [72-75] (i.e., inference <> dynamics), membrane potentials evolve to minimize variational free energy (F). To respect the curved geometry of distribution space, this optimization is (Fig. Ip).
implemented via natural gradient descent 34 57 76
Poisson variational free energy. Given Poisson posterior and prior, and a factorized Gaussian likelihood with linear decoder, the free energy (negative ELBO from eq. (T) takes the form:
K
F(x; 2 uo u) = E Bz=g(=(a)
2 x
22
2 uo) - (e - eu (2)
Lenoon reconstruction (distortion)
Cx KL term (coding rate)
where ui E R and u R are the prior and posterior membrane potentials, K is the latent dimensionality, O represents the element-wise (Hadamard) product, and a positive coefficient that controls the rate-distortion trade-off 77 78 See appendix B.3 for detailed derivation.
Next, we compute the gradient of F with respect the variational parameters, i.e., V. F.
Free energy gradient: the reconstruction term. Crecen. is the expected value of the prediction errors under the approximate posterior. Since this expectation is intractable in general, follow standard practice in variational inference and approximate it using a single Monte Carlo sample [34]
6 Lrevnn. ~ 2 || - (z(2))l2 where zuu) qu (=(x).
Since the reconstruction loss depends the sample z, which is stochastic function of the firing rate = exp(u), we propagate gradients through both the exponential nonlinearity and the sampling ).econ.). We apply the chain rule twice to get:
process (i.e.,
Ör Öz 0
Oz
Vu Leccon =
Crecon.
(-0
1 (x - @2())))
du Ōr 0z
Jr
We further simplify the expression by applying the straight-through gradient estimator 779.. treating 0z/ōr /Or = I for the purpose of deriving inference equations. During model training. however, we update the generative model parameters by utilizing the Poisson reparameterization algorithm (50).
This straight- through approximation of the sample gradient yields:
V.. Crecon. RJ (-0² (x (z(u)))
Free energy gradient: the KL term. The KL-term gradient is easily computed from eq.
(2)
Va CKL = E" O (u Uo). Combine it with the reconstruction-temn gradient from eq. (4) to get:
Flx; 2 uo, u) & о
t² (x .z(u)) + B(u un)]
Natural gradients: applying Fisher preconditioning. FOND prescribes that neural dynamics
follow natural gradients of the free energy, -G- V. F. where G the Fisher information matrix. For Poisson distributions in the canonical form, we have: G(u) = exp (u) (see appendix B.4 for derivation). Combining this result with eq. (5) yields:
X
To (T)
3(u - up
(6)
feodforward "drive"
recurrent "cxplaining away"
homcostatic "Jeak" term
This concludes the derivation of our core theoretical result underlying the iP-VAE architecture. Natural gradient descent on F yields circuit dynamics that share important characteristics with neural models developed since the early 1970s (80 while offering notable biological advantage over

---

standard predictive coding (PC). Namely, recurrent interactions in iP. VAE (eq. (6) oceur through discrete spikes 2, rather than continuous membrane potentials as in PC (eq. (T)), better aligning with how real neurons communicate (84). In appendix B.5 we interpret each term in the dynamics and explain the rationale behind its naming. appendix [6.6] we extend the derivation to nonlinear decoders; and in appendix [B.]] we extend to Gaussian posteriors with optional nonlinearities. Finally,
appendix B.8 explores this biological distinction with standard PC in more detail.
Online inference: adding time. Perception is an ongoing process that requires continually updating beliefs as data arrives, rather than reverting to a fixed prior in the distant past. To model this sequential process, we adopt rolling update scheme, where the current posterior becomes the next prior (Fig. 7 (60)) Combining this online structure with the dynamics from eq. 66). we arrive at the following discrete-time update rule:
U2++1 = Ut-
x
where u, and Ut-1 are interpreted as the prior and posterior membrane potentials at time (Fig.
Notably, the leak term in eq. -which stems from the KL term in eq. (2) -ddisappears in this discrete-time update. This is because we perform only a single step per time point in the online
setting, where the prior evolves continuously over time (Fig. 7. In appendix [B.9] we derive the
general form of free energy for sequences, and in appendix B.10 we explain why the KL. contribution vanishes in the single-update limit.
Lateral competition as a stabilizing mechanism. The recurrent connectivity matrix, W == oTT
has a stabilizing effect. To demonstrate this, we transform the discrete-time dynamics in eq. 7 from membrane potentials to firing rates, yielding the following multiplicative update for neuron
exp T ))
T+t1, Tt,i exp (Wa 21,3) 1 exp (Wij =(.))
(8)
where = =xp(ur) are firing rates, and 2, Pois(z;1r) are sampled spikes. The resulting denominator reveals form of multiplicative divisive normalization: co-active neurons suppress each other proportionally to their spike output.
As seen in eq- (8), the recurrent term W 2, acts stabilizing force in ways. First, its diagonal entries are positive (i.e., WH = || > 0); therefore, neurons that spike strongly receive self- suppression, dampening activity at the next time point. Second, the off-diagonal entries couple neurons with overlapping tuning. Due to this coupling. vigorously active neurons extend suppressive influence toward their similarly-tuned neighbors (i.c., when Wej > 0), preventing excess activity. In addition to stabilization, such competitive interactions also underlie the sparsification dynamics observed in cortical circuits 87). and are widely regarded as a hallmark of sparse coding (39 88
In sum, have shown that natural gradient descent on free energy, combined with biologically motivated assumptions, leads to principled and interpretable neural dynamics. As our next step, apply the theoretical derivations (eq. (6), appendix [B7] and Fig. (5) to design specific instances of iterative VAFS within the broader FOND framework. These include: the iterative Poisson VAE (iP-VAE), the iterative Gaussian VAE (iG-VAE), and the iterative Gaussian-relu VAE (iGreta -VBB).
5 Experiments
We evaluate the family of iterative VAE models introduced this work (iP-VAE, ig- VAE, iGret VAE; [.. These models share identical inference dynamics derived in section 4 and appendix B.7 differing only in their latent variable distributions (Poisson vs. Gaussian) and an optional nonlinearity (c.g., relu (89) applied after sampling from the posterior. This systematic comparison helps isolate the relative influence of each component on learning brain-like representations that generalize. To evaluate the impact of inference methods, we compare these iterative models to their amortized counterparts (A-, VAVA -reE-VVE), all implemented with identical convolutional encoders.
We also compare to classic normative models in neuroscience, including two predictive coding models, standard PC (22) and incremental PC (iPC; (90)), and the locally competitive algorithm (L.CA; (39), a sparse coding model that can be viewed as a deterministic, non-spiking precursor to iP-VAE. See Table for a summary of models, and Fig. 2 for visualization of the model tree.
Additional comparisons to hybrid iterative-amontt amortized VAEs 91 92 are presented in the appendix.
7

---

Poisson (o--AR)
0.8
Recesincten [80]
Gradient Nom
Gaussian ( VAL.
0.4
Apanch (araan
0.2.
(sp- VAE)
Poisson
Gaussian
Gaussion
109
10 Than 10a
10a
10
10
10a
10a
10a
10
10a
10a
Learned dictionary elements
Figure 3: All iterative VAEs converge beyond the training regime (Tuain 16). iP-VAE outperforms
iGuetu-VA VAE in sparsity, while ig- VAE achieves superior reconstruction but with dense representations. iP-VAE maintains reasonable reconstruction performance, despite using constrained representations (sparse, integer-valuee spike count). The sparsification dynamics of both (P-VAE and iG VAE resemble those observed in the mouse visual cortex [106] (appendix C.3) All models are trained on 16 16 natural image patches, and the traces averages the entire test set. The right panel displays representative dictionary elements (+); see Fig. for for the complete set of = 512 features.
Learning to infer. To learn the model parameters, gradients across the entire inference trajectory and applies a single update at the end. Specifically, perform Tiain inference steps per input batch and optimize model weights using the accumulated gradients across all steps (not just the final state). This procedure is analogous to backpropagation through time [93 94). and Timain can be interpreted
training scheme that accumulates
the effective model "depth" [43 95
(Fig.B) We find that the choice of Taain has a significant effect on quantitative metrics, making it an important hyperparameter to track. We explore various Train but report performance using the same Test = 1,000 steps.
Hyperparameters. In addition Tirain: we explore various values [78] which controls the reconstruction--I. trade-off (eq. (2). For iP-VAE and P-VAE (0). varying 3 traces out a reconstruction-spe sparsstyyyysssyyyyooooo landscape, loosely analogous
rate-distortion curve (77]. the main paper, all models have a latent dimensionality of K = 512, with linear decoders comprised of a single dictionary, & E MM x where M is the input dimensionality (i.c., number of pixels). In the appendix, explore more general nonlinear decoder architectures and different latent dimensionalities. See appendix C.I for additional implementation, hyperparameter, and training details.
Tasks and Datasets. We evaluate models in several ways, including convergence behavior, re- construction-spaasity trade-off, downstream classification, and out-of-distributtio (OOD; (98 (9]) generalization. We train and evaluate models on two datasets: (1) whitened 16 16 natural image patches from the van Hateren dataset [100]. used to assess convergence and reconstruction-spariiy trade-offs; (2) MNIST [O1],, used for reconstruction, classification, and OOD generalization tests. good portion of these results, including the OOD ones, are presented in the appendix, as our primary focus this paper is theoretical, and these results warrant their focused exploration in future work.
Performance metrics. We define model representations as samples drawn from the posterior at each time point (e.g., (t) in Fig. [T],, and quantifyy performance using the following metrics. For reconstruction, we compute the coelficient of determination (I²²) between the input image, æ, and its reconstruction, å = Φz R²(x:, á)) quantifies the proportion of input variance explained and yields score bounded from above (R² < 1, typically in [0, 1] for effective models). For sparsity, we use the proportion of zeros in the sampled latents: torch, mean (z This is simple but informative proxy for energy efficiency in hardware implementations [02 105
IP-VAE converges sparser states while maintaining competitive reconstruction performance. We frame inference as convergence to attractor states (Fig. [Tb) that faithfully represent inputs, achieving high-fidelity reconstruction with sparsity. To evaluate convergence quality, we train iterative VAEs using T Tram 16, and monitor inference process across Trss = 1,000 iterations, tracking three metrics averaged over the test set: (1) reconstruction fidelity via R22 (2) sparsity via the proportion of
zeros, and (3) proximity to the attractor via gradient norm | x |G-1 V., Fll calculated across the latent dimension (K 512). These metrics collectively characterize convergence dynamies and allow to evaluate attractor quality (Fig. 3).
8

---

We determine convergence by detecting when the R² trace flattens and remains stable (details in appendix (C2). For iP-VAE, ig -VEE, and VAE, respectively, we observe convergence times of 95, 75, and 69 iterations. At the final state (L = 1, 1,000), these models achieve R values of 0.83, 0.82, and 0.87, with latents exhibiting 77%, 58%, and 0% zeros (Fig. 3) ip- achieves superior sparsity compared to both Gaussian models, while ig- UAE achieves the best overall R². Throughout inference, the gradient norm | steadily decreases but plateaus around 0.5, 0.9, and 1.0, respectively, likely due to gradient variance induced by stochastic inference dynamics.
iP-VAE learns Vi-like features. Analysis of the learned features reveals that ip- -P-VAE develops VI- like Gabor filters [107-110] while igretu VAE learns localized pixel-like patterns and ig- VAE learns unstructured features (see right panel in Fig- 3] and Fig. [Tr complete dictionaries). Additionally, exhibits contrast-depender response latency characteristic of V1 neurons [111 [12] (Fig. 8 possibly a consequence of the normalization dynamics that emerge from our theoretical derivations (eq. (8)) We explore these cortex-like
when tested with drifting gratings of varying contrasts, ex-ibbx
properties further in appendix C.3 In summary, these results suggest iP-VAE learns brain-like representations while achieving the best reconstruuction-sparitty compromise.
iP-VAE achieves the best overall reconstruction-saarity trade-off. To assess the robustness of ip-VAE's performance across hyperparameter settings, we systematically explore combinations of Tiain E [8, 32]. and 3 values proportional to training iterations (ranging from 0.5x to 4.0 x T imain). To compare across inference methods, we include amortized counterparts with identical | selection criteria but using Timin = 1. Finally, we also include LCA [39] due to its theoretical similarity to iP-VAE (eq. (6) See appendices C.I.I to C.1.4 for architectural and training details for all models.
Figure 4u positions all models within unified sparsity- reconstruction landscape, enabling direct comparison of performance metries and revealing hyperparameter sensitivity. This representation can be interpreted through the lens of ditt-distottion theory (77). where R² corresponds to inverse distortion and sparsity to inverse coding rate. The optimal point represents perfect reconstruction (R² 1.0) using only zeros (sparsity- 1.0). This unachievable ideal is marked with a gold star.
For iP-VAE, varying Timin and 3 traces a characteristic curve where increased sparsity trades off against reconstruction quality at Train 8, with this trade-off improving at higher training iterations (Timin 16, 32). LCA and i exhibit a similar pattern, but iGrelu VAE consistently underperforms ip- -P-VAE in both metrics, confirming Fig. 3 iG-VAE achieves comparable Rt2 values to high-Tnain iP-VAE fits but with zero sparsity. These extensive evaluations confirm that both iP-VAE and LCA achieve the best overall reconstruction-spiie sparsity trade-offs among the tested models.
Finally, for fair comparison with LCA's maximum a posteriori (MAP) estimation, we evaluated VAE models with deterministic decoding, where iP-VAE slightly outperforms LCA (Fig. 9 We
also confirm that higher| 3 values increase iP-VAE VAE sparsity as theoretically predicted (50) (Fig. 10
and appendix C.4. For the effect of latent dimensionality on performance, see appendix
Iterative VA unanimously outperform their amortized counterparts. To quantify overall performance as single metric, compute the Euclidean distance from each model to the optimal point (the gold star at R² = 1.0, sparsity = 1.0), with lower distances indicating better performance. Figure shows that iterative VAE models VAE, i VAE, VAE) consistently outperform their amortized counterparts (g- (AE, Greu-VAE. and PAEAE), despite the latter employing deep convolutional encoders with orders of magnitude more parameters. As predicted by theory (com- paring eq. (6) with the LCA dynamics (39)), the performance of LCA and ip- VAE are statistically indistinguishable though iP-VAE exhibits lower variability across hyperparameter settings.
For clarity in the reconstruction-sparss analysis, we omit PC and iPC as their dense Gaussian latents are effectively represented by iG-VAE Extended experiments on MNIST (appendix C.6 show that iterative VAEs outperform PCNs in reconstruction metrics, with iP-VAE achieving the best reconstructionn-spasii
trade-off (Table 3) Interestingly, in downstream classification, P. P-VAE reaches ~ 98% accuracy, comparable to supervised PCNs 113 See appendix C.6 for more details.
Iterative VA match the inference runtime of their amortized counterparts. Despiterequiring ~ 2x training time (appendix C.1.4), iterative models achieve competitive inference speeds for large batches, matching amortized performance before continuing to superior trade-offs (appendix C.7

---

a
b
1.0
1.4
Euclidean distance from top-right corner
1.2
0.8
Reconstruction [R.
1.0
IP-VAE
83
0.6
0.8
12-VB
0.6
0.4
0.4
0.2
PE-VAE G-VAE
0.2
0.0
0.0
300 59
0.0
0.2
0.6
Sparsity [portion zeros]
Figure 4: Reconstruution-spaaiity trade-off across model families. (a) Performance landscape showing reconstruction quality (R²) versus sparsity (proportion of zeros). Different symbols indicate model variants (triangles: Train 8, crosses: Train 16, circles: Tirinn 32, empty squares: amortized VAEs, plus: LCA), with colors denoting model architectures. The gold star marks the theoretical optimum. (b) Overall performance measured as Fuclidean distance from the optimum point. Iterative models (right side) consistently outperform their amortized counterparts (left side), with iP-VAE and LCA achieving the best overall performance. See also related Figs. aadd
iP-VAE exhibits strong out-of-distribution (OOD) generalization. While the main results use linear decoders (á: 42), framework easily extends deep, nonlinear decoders. In appendix [B66) we develop the theory; and in appendix CC8 we train nonlinear iP-VAE models with multilayer
deep sparse coding-and compare them to hybrid iteratie-amootizedd VAES 9T 12,922. For both within-dataset perturbations (Fig. (4) and
perceptron and convolutional decoders interpretable
cross-dataset settings (Figs. 13 15 and 16p. iP-VAE consistently outperforms alternatives in OOD
reconstruction and classification accuracy (Fig. [4]. These results suggest that iP-VAE learns compositional code (Fig. l7p. a hypothesis we aim study more systematically in future work.
iP-VAB seales to complex color image datasets and generates realistic images. In appendix C.9 we demonstrate practical utility by scaling iP-VAE to high-dimensional color images (CelebA, CIFAR- 10, tiny ImageNet); and, in appendix C.10 we show that iP-VAE successfully generates realistic MNIST samples through iterative procedure.
Conclusions and Discussion
In this paper, we introduced FOND, a framework for deriving brain-like inference dynamics from variational principles. We then applied FOND to derive new family of iterative VAE models, including the iP-VAE. a spiking model that performs inference its membrane potential dynamics. iP-VAE's success likely stems from three key attributes: (1) exponential nonlinearities and (2) emergent normalization, similar to recent advances in sequence modeling [T14]; and (3) effective stochastic depth through temporal unrolling, which also explains the effectiveness of hierarchical VAEs [155). Further, iP-VAE's weight reuse and its sparse, integer-valued spike count representations enable efficient hardware implementation [166]. While we focused variational inference, equally important campling-basee approaches offer complementary perspectives [117 122). Future work should explore acceleration techniques for iterative inference [114) biologically plausible learning rules [23], predictive dynamics for non nta-stationary sequences [124], hierarchical extensions [15 126). and neural data applications 129 We extend upon these points in appendices D. D.5
In sum, this work connects the prescriptive model development philosophy of FOND with practical algorithms that exhibit bike behavior and deliver empirical benefits in machine learning.
10

---

7
Code and data
Our code, data, and model checkpoints available here: https//ghhhbbbcommmaaaaaal/llllr
8 Author contributions
All authors conceived of the project and developed the core theoretical framework. H.V. conducted the formal analysis, implemented the primary models, and generated figures and tables. D.G. imple- mented baseline comparisons. All authors contributed to writing the manuscript.
9 Acknowledgments
We would like to thank Bruno Olshausen, Richard D. Lange, and Ralf M. Haefner for fruitful discussions. This work supported by the National Institute of Health under award number NEI EY032179. This material is also based upon work supported by the National Science Foundation Graduate Research Fellowship Program under Grant No. DGB-1752814 (DG), Any opinions, findings, conclusions, or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation. Additionally, DG was funded by the Center for Innovation in Vision and Optics, We thank the developers of the software packages used in this project, including PyTorch [T30), NumPy [I31, SciPy (132). scikit-leam [33 pandas 134] matplotlib
135 and seaborn 136
References
[1] Alhazen. Book of optics (Kitab Al-Manazir). 1011-1021 AD. URL: https: archive
org http:///eeeeeeetaoeooooooaaaaaaa
[2] Hermann Von Helmholtz. Handbuch der physiologischer Optik. Vol. 9. Voss, 1867. URL:
[3] Kenneth J. W. Craik. The Nature Explanation. Cambridge, UK: Cambridge University
Press, 1943. URL: https: /ttpps:///rrchai archive. org details dettalls//ntuurrffeeeeaattttttto
page/n5/mode/ /2up
[4] Peter Dayan al. "The Helmholtz machine". In: Neural Computation 7.5 (1995), pp. 889-
904. Dol: 101:/10.1 1ee2111ccccceeee 1995. 7.5.8...888..
[5] Tai Sing Lee and David Mumford. "Hierarchical Bayesian inference in the visual cortex". In:
JOSA 20.7 (2003), pp. 1434-1448. DOI: [0. 1364/ JOSAA. 20. 001434
[6] Horace Barlow. "Redundancy reduction revisited". In: Network: computation in neural
systems 12.3 (2001), p. 241. DO1: 10. 1ww.0000/net... 12.3. 3. 241 241.253
[7] Bruno A. Olshausen. "Perception as an Inference Problem". In: The Cognitive Neurosciences
(5th edition) (2014). Ed. by Michael Gazzaniga and George R. Mangun. DOL: 10 7551/
mitpress 9904. 003 0037 URL: http://rcee org/ /ttp//ruo/////// pttpa///pppppeeeeept..oooo
as- is-inference. -pdf]
[8]
David Ha and Jürgen Schmidhuber. "World models". In: (2018). arXiv: 1803 10122v4
cs. LG]
[9]
Danijar Hafner et al. Action and Perception as Divergence Minimization. 2022. arXiv: 2009 01791 [cs.AI]
[10] Thomas Bayes "An essay towards solving a problem in the doctrine chances". In: Philosophical Transactions of the Royal Society of London 53 (1763), pp. 370-418, DO1:
10. 10-10998rst.... 1763. 0053
[11]
Pierre-Simon Laplace. A Philosophical Essay on Probabilities. Originally published in French in 1814 as Essai philosophique sur les probabilités. Courcier, 1812. URL: https
/ dn79000 ca archive org / 0 items philosppiicaless0llppp...
philosophicclees00llppppppp
pdf
[12] Kevin P Murphy, Machine Learning: Probabilistic Perspective. MIT Press, 2012. URL:
https: https://probml. www.obbm..ggtttut github.......... btccuu..oo/ppee wot.:oommmmmmmmmmmmmm book book/bookkkkkkkooooooooooooookkk html

---

[13] Radford M. Neal and Geoffrey E. Hinton. "A View of the Em Algorithm that Justifies Incremental, Sparse, and other Variants". In: Learning in Graphical Models. Ed. by Michael I. Jordan. Dordrecht: Springer Netherlands, 1998, pp. 355-368. ISRN: 0111994--111----- DOT:
10.
[14] Arnold Zellner and. "Optimal Information Processing and Bayes's Theorem". In: The Ameri- can Statistician 42.4 4 (1988), pp. 278-280, DO1: 10. 1080/ 0000000000333555 1988. 10475585
[15] Michael I Jordan et al. "An introduction to variational methods for graphical models". In: Machine learning 37 (1 (1999), pp. 183-233. DOT: 10. 1023/1:230:: 1007665977777
[16] Dimitris G Tzikas et al. "The variational approximation for Bayesian inference". In: IEEE
Signal Processing Magazine 25.6 (2008), pp. 131-146. DOI: 10. 1109/MSP 2008. 929620
[17] David M. Blei al. "Variational Inference: Review for Statisticians". In: Journal of the American Statistical Association 112.518 (2017), pp. 859-877. DOI: 10. 10w.1000012220 01681/1
2017.122 1285773
[18]
Diederik Kingma and Max Welling. "Auto-encoding variational bayes". In: (2014). arXiv:
1312. 6114v11 [stat. ML]
[19]
Danilo Jimenez Rezende al. "Stochastic backpropagation and approximate inference in deep generative models". In: International Conference on Machine Learning. PMLR. 2014, pp. 1178-1285 URL: https: https///ppooeddnns...oo mlr. wrwss/ hatess//22/reee http///eeenee...eeee html
[20] Diederik P Kingma and Max Welling. "An introduction to variational autoencoders". In: Foundations and Trends in Machine Learning 12.4 (2019), pp. 307-392, DOI: 15617 2200000056]
[21] Joseph Marino. "Predictive coding, variational autoencoders, and biological connections". In: Neural Computation 34.1 (2022). pp. 1-44. Do1: 10. 1 hatp:///eco_a____111111
[22] Rajesh PN Rao and Dana H Ballard. "Predictive coding in the visual cortex: functional interpretation of some extra-classical receptive-field effects". In: Nature Neuroscience 2.1
(1999), pp. 79-87. DOI: 10. 1038/4580
[23]
Karl Friston. "A theory of cortical responses". In: Philosophical transactions of the Royal Society B: Biological Sciences 360.1450 (2005), pp. 315-836. DOT: 10. 100.199/rrstb 2005
1622
[24]
Karl Friston. "The free-energy principle: a unified brain theory?" In: Nature Reviews Neuro- science 11.2 (2010). pp. 127-138. Do1:[10. 1038/008/nnia nr77
[25]
Rafal Bogacz. "A tutorial on the free-energy framework for modelling perception and leam- ing". In: Journal of Mathematical Psychology 76 (2017). Model-based Cognitive Neuro- science, pp. 198-211, ISSN: 0022-2496. DOI: https://doi. http://dooooorgg////// 1016// 1tt.11000/m.mm.211111.11
jmp. 2015
11.003
[26] Beren Millidge. Applications of the Free Energy Principle to Machine Learning and Neuro- science. 2021. arXiv: 00140 [cS.AI]
[27] Samuel Gershman. "What does the free energy principle tell us about the brain?" In: 1rXiv:1901.0790 (2019). URL: https: http:///aarrii.orrra////00//// .79445
[28] Mel Andrews. "The math is not the territory: navigating the free energy principle". In: Biology & Philosophy 36.3 (2021), p. 30. DOE: 10. 1007
[29] Jascha Sohl-Dickstein et al. "Deep unsupervised learning using nonequilibrium thermody- namics". In: International conference on machine learning. PMLR. 2015, pp. 2256-2265. [30] Diederik P Kingma and Ruiqi Gao. "Understanding Diffusion Objectives as the ELBO with Simple Data Augmentation".. In: Thirty-seventh Conference on Neural Information Processing Systems. 2023, URL: https: h /ptp//eepeeeeeeeeeeeeeeeeeeeeeee wwt.:ee/tffoooee pent://orummiddnnnaaaa NMMEadedyD
[31] Calvin Luo. Understanding Diffusioo Models: A Unified Perspective. 2022. arXiv: 2208
11970 LG]]
[32] Mandyam Veerambudi Srinivasan et al. "Predictive coding: fresh view of inhibition in the retina". In: Proceedings of the Royal Society of London. Series B. Biological Sciences
216. 1205 (1982), pp. 427-459. DOI: 10. 1098/rapb. 1982. 0085
[33]
Beren Millidge al. Predictive Coding: a Theoretical and Experimental Review. 2022. arXiv:
2107. 12979 [cs. [II]
[34]
Mohammad Emtiyaz. Khan and HĀWvard Rue. "The Bayesian Learning Rule". In: Journal of Machine Learning Research 24.281 (2023), pp- 1-46. URL: http:://m// org/ patpr//p////
v24/22 0291. htmll
12

---

[35] Shun-ichi Amari. "Natural Gradient Works Efficiently in Learning", In: Neural Computation 10.2 (Feb. 1998). pp. 251-276. ISSN: 0899-7667. DO1: 10. 11w/1112///00 11662089976993000177729
[36] Samuel Gershman and Noah Goodman. "Amortized inference in probabilistic reasoning", In: Proceedings of the annual meeting of the cognitive science sociery. Vol. 36. 36. 2014. URL: htttss:/esschlarsst arship org/ uc/ httem/44111 1
[37]
Brandon Amos. "Tutorial on Amortized Optimization". In: Foundations and Trends@ in Machine Learning 16.5 (2023), pp. 592-732. ISSN: 1935-8237. DO1: 10. 10..116612200000000
[38] Ankush Ganguly et al. " "Amortized Variational Inference: A Systematic Review". In: Journal of Artificial Intelligence Research 78 (2023), pp. 167-215. DOI: 10. 1613/jair 1. 1.14258 Christopher Rozell al. "Sparse coding via thresholding and local competition in neural circuits". In: Neural Computation 20.10 (2008), pp. 2526-2563. DOI: 10. 1162/neco..... 2008
[39]
03-07-486)
[40] Victor A.F. Lamme and Picter R. Roelfsema. "The distinct modes of vision offered by feedforward and recurrent processing". In: Trends in Neurosciences 23.11 (2000), pp. 571- 579. ISSN: 0166-2236. DOI: pesttps//ddot http://ti.o..o/////////.. org 10. 2006//00666 220///0666622260000000 (26 01657-X,
[41] Tim C Kietzmann et al. "Recurrence is required to capture the representational dynamics of the human visual system". In: Proceedings of the National Academy of Sciences 116.43 (2019), pp. 21 21854-21863. DOT: 10. 10/11077/ppaa pnas. 1905544116
[42] Kohitij Kar et al. "Evidence that recurrent circuits are critical to the ventral stream's 's execution of core object recognition behavior". In: Nature neuroscience 22.6 (2019), pp. 974-983. DOI:
10. 1038/ N0.1038/s41553 10008//41593-01-000000 0392-5
[43] Ruben S van Bergen and Nikolaus Kriegeskorte. "Going in circles is the way forward: the role of recurrence in visual inference". In: Current Opinion in Neurobiology 65 (2020). Whole- brain interactions between neural circuits, pp. 176-193. ISSN: 0959-4388. DO1: https
//doi. Prri.org/100 /ttg/10.1116.. 1016/j conb. 2020. 00.
[44] Yuelin Shi et al. "Rapid, concerted switching of the neural code in inferotemporal cortex". In:
bioRxiv (2023). DO1: 10. 1w1111111222... 12. 570344
[45] Adam Kohn. "Visual adaptation: physiology, mechanisms, and functional benefits". In: Journal of neuropyysiology 97.5 (2007), pp. 3155-3164. DOI: 10. 1152/jn. 00022//..0006..... 000000008.... 0008 Jason Fischer and David Whitney. "Serial dependence in visual perception". In: Nature neuroscience 17.5 (2014), pp. 738-743. DOt: 10. 1ww8/n0..n3333 3689
[46]
[47] Guido Marco Cicchini et al. "Serial dependence in perception". In: Annual Review of Psy- chology 75 (2024), pp. 129-154. DO1: 10. 1 annurev- paynurree-pychh.ho. 021523- 104939
[48] Jennifer Freyd and Ronald A Finke. "Representational momentum". In: Journal of Experi- mental Psychology: Learning, Memory, and Cognition 10.1 (1984), p. 126. DOI: 10. 1037/ /
0278-7393 1. 126
[49] Floris P De Lange et al. "How do expectations shape perception?" In: Trends in cognitive sciences 22.9 (2018), pp. 764-779. DO1: 10. 1.1.//..iis............ tics. 2018 002
[50] Hadi Vafaii et al. "Poisson Variational Autoencoder". In: The Thirty-eighth Annual Conference on Neural Information Processing Systems. 2024. URL: https // openreview net / Forum?id=eetPPccqdd..
[51] David J Tolhurst et al. "The statistical reliability of signals in single neurons in cat and monkey visual cortex". In: Vision research 23.8 (1983), pp. 775-785, DOr: 10. 1016//0/04 6989 (83) 90200-6
(88)990000-0
[52] AF Dean. "The variability of discharge of simple cells in the cat striate cortex". In: Experi-
mental Brain Research 44.4 (1981), pp. 437-440. DOI: 10. LO00/BFO0238000
[53] Alexandre Pouget et al. "Information processing with population codes". In: Nature Reviews
Neuroscience 1.2 (2000), pp. 125-132 132. DOI: 10. 1038/ 0088/3533902
[54] Bruno A Olshausen and David Field. "Emergence of simple-cell receptive field properties by learning sparse code for natural images". In: Nature 381.6583 (1996), pp. 607-609, DOI:
10. ND.1008388667aa
[55]
Bruno A Olshausen. "Learning linear, sparse, factorial codes". In: (1996). URL:: https //dspace. mit. adu/bitstream/ httpt//eeeeeeaaaeeee/aaa////////
[56] Peter Dayan and Laurence F Abbott. "Theoretical Neuroscience". In: (2001). URL: https
mit https//aaaaaaaaoe//.////////aaaa
13

---

[57]
Mohammad Emtiyaz Khan and Didrik Nielsen. "Fast yet simple nutural-gradion descent for variational inference in complex models". In: 2018 International Symposium on Information
Theory and Its Applications (ISITA). IEFE. 2018, pp. 31 31-35. DOT: 200.00/999IIIST...... 2018
8664326
[58]
Olivier Catoni. PAC-Bayesian Supervised Classification: The Thermodynamics of Statistical Learning. Institute of Mathematical Statisties Lecture Notes Monnorapnh Series. IMS, 2007. URL: htt::htts:////aae arxiv org/abs/ abs/ htt.s//771..0244 12.0248
DOL: 10t.1...11111 1211//7499221111 00777492177770007 707000000391
[59] P. G. Bissiri et al. "A general framework for updating belief distributions". In: Journal of the Royal Statistical Sociery: Series B (Statistical Methodology) 78.5 (2016), pp. 1103-1130.
DOI: httph::t///// :/dp:i oeroii.org//o.. 10. 1111/rssb. 12158
[60] Matt Jones et al. "Bayesian Online Natural Gradient (BONG)". In: The Thirty- Anmual Conference on Neural Information Processing Systems. 2024. URL: [tt[:::tttts::/ppprreee//
net/forumiice 7 enDDy02G
[61]
L.D. Landau and E.M. Lifshitz. The Classical Theory of Fields. Siheldd..hhhh 4th. Vol. 2. Course of Theo- retical Physics. Pergamon Press, 1975, ptt cra.sspssssoocacooooooooooooo wtwp://asrrrr///sllaaiirrroooo
pdf
[62] Steven Weinberg. "Effective field theory, past and future", In: Internationa Journal of Modern
Physics A 31.06 (2016), p. 1630007. DOI: 10. 1t1.1114227775111630000000
[63]
Jakob Schwichtenberg. No-Nonsense Quantum Field Field Theory: A Student- Introduction. No-Nonsense Books, 2020. ISBN: 9783948763015. URL.: http: http://wwwwtttuuee... ucla. edu/
yww/QFFO.ppol
[64]
Jakob Schwichtenberg. Physics from symmetry. Springer, 2018. DOt: 10.1007/978-3-33 66631-0
[65]
James Martens. "New Insights and Perspectives on the Natural Gradient Method". In: Journal of Machine Learning Research 21.146 (2020), pp. 1-76. URL: http:// http:///mll..or///ppess///
org/papers/
v21/17-678 html
[66]
Martin E Seligman and Steven F Maier. "Failure to escape traumatic shock." In: Journal of experimental psychology 74.1 (1967), p. DOI: 10. 1033/000075051
[67]
Charisse Nixon. Learned Helplessness Nov. 2007. URL: https // www. youtube. com/
watch?v=gFuFOngg
[68]
Alan Yuille and Daniel Kersten. "Vision Bayesian inference: analysis by synthesis?" In: Trends in Cognitive Sciences 10.7 (2006), pp. 301-308. DOI: 10.10166j..tcc 2006. 05
002
[69]
Oleh Rybkin et al. "Simple and Effective VAE Training with Calibrated Decoders". In: Proceedings of the 38th International Conference on Machine Learning. Ed. by Marina Meila and Tong Zhang. Vol. 139. Proceedings of Machine Learning Research. PMLR, June 2021, pp. 9179-9189, URL: http:::///proceedinng. mmr. paress/v139/e http////////yii1aaa.aa html [70] Nicholas J Priebe et al. "The contribution of spike threshold to the dichotomy of cortical simple and complex cells". In: Nature neuroscience 7.10 (2004), pp. 1113-1122. DOI: 10
1038/nn1310
[71] Nicolas Fourcaud-Troemé et al. "How spike generation mechanisms determine the neuronal response to fluctuating inputs". In: Journal of neuroscience 23.37 (2003), pp. 11628-11644
DOI: 10.1 JNEUROSCI. 23- 3 2003
[72] Alfred North Whitehead. Process and Reality: An Essay in Cosmology. 1929. URL: https
//archive org dwt.adstaass/eeeeeeeeeeeeeealtta httpp///ppoosssreeieyiiill00000
[73] J Hopfield. "Neural networks and physical systems with emergent collective computational
abilities." In: Proceedings of the National Academy of Sciences 79.8 (1982), pp. 2554-2558.
DO1: 10. 1073/ 2ww.3//aas.... 7....2...... 2554
[74] Karl Friston et al. " "Active Inference: A Process Theory". In: Neural Computation 29.1 (Jan.
2017), pp. 1-49. ISSN: 0899-7667. DOT: 10. 1162/NECO_a_0000 0_912
[75] Saurabh Vyas et al. "Computation Through Neural Population Dynamics". In: Annual Review
of Neuroscience 43. Volume 43, 2020 (2020), pp. 249-275,
ISSN: 1545-4126. DOI: https
//doi.org/10 10. wwt/////n///-///eeeee -ww.---------00000909999001-1010 1111///aaaaueeeeer0o0000001--000
[76] Shun-ichi Amari and Hiroshi Nagaoka. Methods of information
getteerrmmttiiiiigorrtrtot geometry. Vol. 191. American
Mathematical Soc., 2000. DOI: 10. (0900//mooo/199
14

---

[77]
Alexander Alemi et al. "Fixing Broken ELBO". . In: Proceedings the 35th International Conference on Machine Learning. Ed. by Jennifer Dy and Andreas Krause. Vol. 80. Pro- ceedings of Machine Learning Research. PMLR, July 2018, pp. 159-168. URL: https //proceedings.oe ma. www.s//r.rrree.///./. alemi 18a, html
the
[78]
Irina Higgins et al. "beta-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework" In: Internationnl Conference on Learning Representations. 2017. URL: https
/apenreview net/ http://ooummmdd--yyyaee
[79]
Yoshua Bengio et al. "Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation".. In: (2013). arXiv: 1308. 3432 [C]
[80]
Shun-Ichi Amari "Characteristics of random nets of analog neuron- like elements". " In: IEEE Transactions systems, man, and cybernetics 5 (1972), pp. 643-657. DO1: 10. 1109/11TSSSS
4309193
[81] Wulfram Gerstner and Werner M. Kistler. Spiking Neuron Models: Single Neurons, Popula- tions, Plasticity. Cambridge, UK: Cambridge University Press, 2002. ISBN: 978052 DO1: 10. 101110000970011111777777
[82] Wulfram Gerstner et al. Neuronal dynamics: From single newrons to networks and models of cogmition. Cambridge University Press, 2014. DOI: 1011 100.1010//8098811074477766
[83] Daniel B Rubin et al. "The stabilized supralinear network: a unifying circuit motif underlying multi-input integration in sensory cortex". In: Neuron 85.2 (2015), pp. 402-417. DOT: 10
1016/j neuron. 2014. 12.026
[84] Eric R. Kandel et al. Principles of Neural Science. 5th ed. New York: McGraw Hill, 2000. URL: htttt::///arciiyeea
org/ http:///ee////.accpaaeaaaaaaaaaa
[85]
Ho Ko et al. "The emergence of funcctional microcircuits in visual cortex". In: Nature 496. 7443 (2013), pp. 96-100. DO1: 10. 1038/ mt/8///aaure12011
[86] Selmaan N Chettih and Christopher D Harvey. "Single-neuron perturbations reveal feature- specific competition in V1". In: Nature 567.7748 (2019), pp. 334-340. DOI: 10. 1038 / -19-86-019-0997-6
[87] Zhuokun Ding et al. "Functional connectomics reveals general wiring rule in mouse visual cortex". In: Nature 640.8058 (2025), pp. 459 DOI: 10. 1038/ s415/441666681 01111156666-1--0111 025-08840-3
[88] Karol Gregor and Yann LeCun. "Learning fast approximations of sparse coding". In: Proceed- ings of the 27th international conference on international conference on machine learning. 2010, pp. 399-406. URL: http::httttccccto
cc/ Conf eronnesreeeee///eee 2010/ 2ww/20000000...... /449. -pff
[89] James C. R. Whittington al. "Disentanglement with Biological Constraints: Theory of Functional Cell Types" _ In: The Fleventh International Conference Learning Representa- tions. 2023. URL.: http:tttsss//// //tterevv// wwt.teettteeeeee htt///formmmddd/222 GfhZnGH
[90] Tommaso Salvatori et al. "A Stable, Fast, and Fully Automatic Learning Algorithm for Predictive Coding Networks". In: The Twelfth International Conference Learning Repre- sentations. 2024. URL: htttt::::tppoeeeettteee
net / for forummidd werrrmmmiidiii RyduddddaR
[91] Joe Marino et al. "Iterative Amortized Inference". In: Proceedings of the 35th International Conference on Machine Learning. Ed. by Jennifer Dy and Andreas Krause. Vol. 80. Pro- ceedings of Machine Learning Research. PMLR, July 2018, pp. wwwpiiins..m...........m.
URL: https
//proceedings.nie mlr pwwssreessssseoo ht////800/mmaaa wwto0/marrnoo.oo 18a. html
[92] Yoon Kim et al. International Conference Machine Learning. Ed. by Jennifer Dy and Andreas Krause. Vol. 80. Proceedings of Machine Learning Research. PMLR, July 2018, pp. 2678-266766 URl: https: //ppee/e/soccedngss. mlr. prwssrressssse v00///80////// www./e..m......//............11t html
Variational Autoencoders". In: Proceedings of the 35th
[93] Paul J Werbos. "Backpropagation through time: what it does and how to do it". In: Proceed- ings of the IEEE 78. 10 (1990), pp. 1550-1560. DOI: 10. 11397
[94] Timothy P Lillicrap and Adam Santoro. "Backpropagation through time and the brain". In: Current Opinion in Neurobiology 55 (2019). Machine Learning, Big Data, and Neu- roscience, pp. 82-89. ISSN: 0959-4388. DOI: https: /oi org 1016/] conb
2019. 01 011 URL: https:://// //www.
www. sciencedirect. com/scienee comt/science//rticcle articce/ patticle/pii//// pii/
50959438818302000
[95]
Stanislaw Jastrzębski al. Residual Connections Encourage Iterative Inference. 2018. arXiv: 1710. 04773
[V)
15

---

[96] Qianli Liao and Tomaso Poggio. Bridging the Gaps Between Residual Learning. Recurrent Neural Networks and Visual Cortex. 2020. arXiv: 1604. 03640 [cs. LG]
[97]
Avi Schwarzschild. "Deep Thinking Systems: Logical Extrapolation With Recurrent Neu- ral Networks". PhD thesis. UMD, 2023, p. 124. URL: https: // www. proquest com/ dissertations - theses / deep- thinking systems logical extrapolation se-2
[98]
Kaiyang Zhou al. "Domain generalization: A survey". In: JEEE Transactions on Pattern Analysis and Machine Intelligence 45.4 (2022), pp. 4396-4415, DOI: 10. 11111.1001 1119/ 2022
3195549
[99]
Jindong Wang et al. "Generalizing to unseen domains: A survey on domain generalization.. In: IEEE transactions knowledge and data engineering 35.8 (2022), pp. 8052-80722 DOI:
10. 11.1//TTTD..... 2022. .12..317811 3178128
[100]
J. H. van Hateren and A. van der Schaaf. "Independent component filters of natural images compared with simple cells in primary visual cortex". In: Proceedings of the Royal Society of London. Series B: Biological Sciences 265.1394 (1998), pp. 359-366. DO1: 10. 1098/ 1098/rspb
1998. 0303
[101]
Yann LeCun et al. "Gradient-based learning applied to document recognition". In: Proceed- ings of the IEEE 86.11 I (1998), pp. 2278-2324. DOI: [0. 1009/119.... 726791
[102] Song Han et al. "EIE: Efficient inference engine compressed deep neural network". In:
ACM SIGARCH Computer Architecture News 44.3 (2016), pp- 243-254. DOI: 10. 1145/
3007787 30011663
[103] Vivienne Sze et Efficient processing of deep neural networks. Springer, 2020. DOI: 10 0007/978-3-031-017558
[104]
Amir Gholami et al. "A survey of quantization methods for efficient neural network inference". In: Low-power computer vision. Chapman and Hal//CRC, 2022, pp. 291-326. DOL: 10. 1201/
9781003162810-13
[105] Itay Hubara Processing Systems. Ed. by D. Lee et al. Vol. 29, Curran Associates, Inc., 2016. URL: https : / proceedings neurips cc / /aperr files / paper / 2016 hash / Abt.rddd--bbsraacc.... html
"Binarized Neural Networks". In: Advances Neural Information
[106] S. Amin Moosavi et al. "Temporal dynamics of energy-efficient coding in mouse primary
visual cortex". " In: bioRxiv (2024). DO1: 10. 1a111100222422 12, 17..1.9..9889 628997
[107] S. Marčelja. "Mathematical description of the responses of simple cortical cells=". In: J. Opt. Soc. Am. 70.11 (Nov. 1980), pp. 1297-1300. DO1: 10. 1364/ 70. 001099
[108] David H. Hubel and Torsten N. Wiesel. "Receptive fields of single neurones in the cat's striate cortex". In: The Journal of Physiology 148 (1959). Dor: 10. www.111pppphsol.....mm 1959
sp006308
[109] David H. Hubel and Torsten N. Wiesel. "Receptive fields and functional architecture of monkey striate cortex". In: The Journal of Physiology 195 (1968). DOI: 10. 10.1113//physion
1968. sp008455
[110]
Jiakun Fu et al. "Heterogeneous orientation tuning in the primary visual cortex of mice diverges from Gabor- receptive fields in primates". In: Cell reports 43.8 (2024). DOI:
10. 1.11//e..eeeeee celrep.......... 2024. 114639
[111] Matteo Carandini et al. "Linearity and normalization in simple cells of the macaque primary visual cortex". In: Journal of Neuroscience 17.21 (1997), pp. 8621-8644. DOI: 10. 1523/
CIUUROSCI.172221 08621. 1997
[112] Duane G Albrecht et al. "Visual cortex neurons of monkeys and cats: temporal dynamics of the contrast response function". In: Journal of neurophysiology 88.2 (2002), pp. 888-913,
DO1: 10. 11w2/122jja 2002 28.22888.
[113] Luca Pinchetti et al. "Benchmarking Predictive Coding Networks Made Simple". In: The Thirteenth International Conference on Leurning Representations. 2025. URL: https
/ //operrvviww.o net/ wate/fooumee coromriiiii www.mddddaaaggssttttt.
[114] Maximilian Beck et al. "xLSTM: Extended Long Short-Term Memory". In: The Thirty- eighth Annual Conference on Neural Information Processing Systems. 2024. URL: https
16

---

[115] Rewon Child. "Very Deep [VE]}s Generalize Autoregressive Models and Can Outperform Them on Images" In: International Conference on Learning Representations. 2021. URL:
https://openreeviee
net rett/ftormmmic-BLLXcccEDDb
[116] C. D. Schuman et al. "Opportunities for neuromorphic computing algorithms and applica- tions". In: Nature Computational Science (2022). DOI: 10.
[117] Patrik Hoyer and Aapo Hyvärinen. "Interpreting Neural Response Variability as Monte Carlo Sampling of the Posterior". In: Advances Neural Information Processing Systems. Ed. by
S. Becker ct al. Vol. 15. MIT Press, 2002. URL: https://pprocceccingss neur cc/
html
[118] Gergó Orbán et al. "Neural variability and sampling-baseee probabillistic representations in the visual cortex". In: Neuron 92.2 (2016), pp. 530-543, D01: 1016/j nwwron.
09.038
[119] Paul Masset al. "Natural gradient enables fast sampling in spiking neural networks". In: Advances in Neural Information Processing Systems. Ed. by S. Koyejo et al. Vol. 35. Curran Associates, Inc., 2022, pp. -203444.... URL: / neurips cc/
paper
hwsh:aahhhaaaaaaaaaaaaaeaaa http:////00000/////////////////0 Abstract
Conference html
[120] Michael Y.-S. Fang et al. "Learning and Inference in Sparse Coding Models With Langevin Dynamies".. In: Neural Computation 34.8 (July 2022), pp. 1676-1700. ISSN: 0899-7667. DOI:
n0.1162/neco__. a_too_____1111
[121] Gaspard Oliviers et al. "Learning probability distributions of sensory inputs with Monte Carlo predictive coding". In: PLOS Computational Biology 20.10 (Oct. 2024), pp. 1-34. DOI:
10. N0.1371/journal pcbi. 1012532
[122] Eli Zachary Sennesh et al. SDivide-and-Conqueee Predictive Coding: a structured Bayesian inference algorithm"". In: The T'iitty-ighhth Annual Conference on Neural Information Pro-
cessing Systems. 2024. URL: htttps://operrewee
[123] Joel Zylberberg et al. "A Sparse Coding Model with Synaptically Local Plasticity and Spiking Neurons Can Account for the Diverse Shapes of V1 Simple Cell Receptive Fields".
In: PLOS Computational Biology 7.10 (Oct. 2011), 1), pp- 1-12. DOT: 10. 1371/ pcbi 1002250
[124] Gerardo AerannMartin et al. "A unifying framework for generalised Bayesian online learning in non-stationary environments". Transactions on Machine Leurning Research (2025).
ISSN: 2835-8856. URL:
[125] Hadi Vafaii al. "Hierarchical VAEs provide normative account of motion processing in
the primate brain".. " In: Thirty-seventth Conference on Neural Information Processing Systems.
2023. URL: http::tt///// http:/::///ppeeeeieiieeeooo 2023:///./ccceee///pp/ueooeeeeee
[126] Ferenc Csikor al. "Top-down perceptual inference shaping the activity of early visual cortex". In: bioRxiv (2025). Dot: 10. 1ew11110222.... 11. 29. 569262
[127] Martin Schrimpf et al. "Brain-score: Which artificial neural network for object recognition is
most brain-like?" In: Rio (2018), 407007. DO1: 10. 1101/407007
[128] Eric Y Wang et al. "Foundation model of neural activity predicts response to new stimulus
types". In: Nature 640.8058 (2025), pp. 470-477. D01: 1038/
[129] Chengrui Li et al. "A Differentiable Partially Observable Generalized Linear Model with Forward-Backward Message Passing". In: Forty-firsi International Conference on Machine h
Learning. 2024. URI.: hnt:httppp... http:p::////peeeeeeeeeeeeee wtwpreewwwweeeeeoooormmiieeeeeee
[130] Adam Paszke et al. "PyTorch: An Imperative Style, High-Performant Deep Learning Library". In: Advances in Neural Information
PropiiiiooPPPPooiioi Processing Systems. Vol. 32. Curran Associates,
paper / 2019 hash / Inc., 2019. URL: https::.... /wwp//pappraape nips cc /paper.ppp.... files /illes/paper/
wttp:/7777400sttaaaacec. html
[131] Charles Harris al. "Array programming with NumPy". In: Nature 585,7825 (Sept. 2020),
pp. 357-362. DOI: 10. 220///44186-000-22----
[132] Pauli Virtanen et al. "SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python".
In: Nature Methods 17 (2020), pp. 261-272. DO1: 10. 1038/ $4 1591192-000 019 0089-0686-2 2
[133] Fabian Pedregosa et al. "Scikit-leam: Machine learning in Python". " In: the Journal of machine
Learning research 12 (2011), 1), pp. 2825-2830. DO1: 10. 00155511953488.000
2078119
17

---

[134]
The pandas development team. "pandas-dew/pandas:: Pandas". Version latest. In: (Feb. 2020).
DOI: www.1.....//eoooo..oo 3509134 3559949
[135] John D Hunter. "Matplotib: A 2D graphics environment". In: Computing in science &
engineering 9.03 (2007), pp. 90-95. DOI: 10. 1109/MCSE. 2007 .55
[136] Michael I. Waskom. "Seaborn: statistical data visualization". In: Journal of Open Source
Software 6,60 (2021), 3021. DO1: 10. 21 21105/joss 03021
[137]
Immanuel Kant. Critique of Pure Reason. Translated by Paul Guyer and Allen W. Wood, Cambridge University Press. 1781. DO1: 10. 1017110111999 11w0111000000111000000
[138]
Aristotle. Posterior Analytics. Trans. by Jonathan Barnes. Originally written c. 350 BCE.
Oxford: Oxford University Press, 1994. URL.: https://classssss wit edu. wdw.t..dd/rriittteee..
posteriox: 1. 1.sterior...1.hmmll html]
[139] John Locke. An essay concerning human understanding. Ed. by Peter H. Nidditch. Oxford University Press, 1690. DOI: 10. 10910099/cccc /ttpa//aee//0/1902////8/////////
book
[140] Alison Gopnik et al. The Scientist the Crib: What Early Learning Telly Us About the Mind. HarperCollins, 2004. URL: htttp://alisonoopniiie cttpoppiikccmom//// wwwae//hheeeeeniiiieeeeeeeeeieie SwrisssIIInntibb htm
[141] Plato. Meno. Trans. by G.M.A. Grube. Originally written 380 BCE. Indianapolis: Hackett Publishing, 1976. URL: htttps://caassiee wit hwt.://mmiiteeeeee///////o/ao meno html
[142] Elizabeth S. Spelke and Katherine D. Kinzler. "Core knowledge". In: Developmental Science 10.1 (2007). DOI:
1111/j 1467-7687 2007 0007000069...
[143] Anthony M Zador. "A critique of pure learning and what artificial neural networks can learn from animal brains".. In: Nature Communications 10.1 (2019), p. 3770. DOI: 10. 1038 141467-019-11788
[144] Edwin Garrigues Boring. "Sensation and perception in the history of experimental psychol- ogy." In: (1942). URL: https: :/ttttpp://ia81100000 us archive org 12 items /nn ernet Sew.://.at222....tiiiin..... And- www.-eeeccpiioooo The -istoostt Experrhnentall-Pyyyollooo.y..ee
52372/252377 52372. Sensation
[145] Richard Langton Gregory. "Perceptions as hypotheses". In: Philosophical Transactions of the Royal Society of London. B. Biological Sciences 290.1038 (1980), pp. 181-197. DO1:
10. 10.1098/RSTB. 1980.
[146] Andy Clark. "Whatever next? Predictive brains, situated agents, and the future of cognitive science". In: Behavioral and brain sciences 36.3 (2013), pp. 181-204. DOI: 10. 1017
S0140525X12000477
[147] David C Knill and Alexandre Pouget. "The Bayesian brain: the role of uncertainty in neural coding and computation". In: Trends in Neurosciences 27.12 (2004), pp. 712-719. DOL: 10. t.i./...iis..............iiiii.t
[148) Karl Friston. "The free-energy principle: a rough guide to the brain?" ?" In: Trends in cognitive sciences 13.7 (2009), pp. 293-301. DO1: 10. 1016/j neuron. 2016. .20
[149] Karl Friston et al. "The free energy principle made simpler but not too simple". " In: Physics Reports 1024 (2023). The free energy principle made simpler but not too simple, pp. 1-29.
ISSN: 0370-1573, DOT: http:: //doi. org/10. 1016/j physrep.ppppppppp 2023. 001
[150] Bruno A. Olshausen and David J. Field. "Sparse coding with overcomplete basis set: A strategy employed by V1?" In: Vision Research 37.23 (1997), pp. 3311-3325. ISSN:
0042-6989. DOI: 10. 1016/500422 (9))
[151] Bruno Olshausen and David Field. "Sparse coding of sensory inputs". In: Current opinion in neurobiology 14.4 (2004), pp. 481-487. DOr: 10. 1016/j conb. 2004. 07 007
[152] William Lotter al. "Deep Predictive Coding Networks for Video Prediction and Unsu- pervised Learning. " In: International Conference on Learning Representations. 2017. URL: https://openrevve/
[153] Christopher L. Buckley et al. "The free energy principle for action and perception: A math- ematical review", In: Journal of Mathematical Psychology 81 (2017), pp. 55-79. ISSN: 0022-2496. DOL: https:t///// //doii///iii http://doo.oogg.../// 101.0/000000111116...... jmp.mmppp... 0.017.09.000
[154] Joy A. Thomas Thomas M. Cover. Flements of Information Theory. 2nd ed. Wiley- Interscience, 2006. DO1: 10.11000000/0777 10.01000/77774882X
[155] Naftali Tishby et al. The information bottleneck method. 2000. arXiv: physics/0004057
[physics. data-an]
18

---

[156] L. D. Landau and E. M. Lifshitz. Statistical Physics, Part 1. 3rd. Oxford: Pergamon Press.
1980. URL: https // /a902998 us archive org / items ost physics landaulifshitz pdf
[157]
E. T. Jaynes. Probability Theory: The Logic of Science. Cambridge: Cambridge University Press, 2003. URL: http:
med mcgill ca/ app.temil.ogg//aaaayy//ooooo (auusi/i//oda//yyyaooooaaattt/// htt.://gggg....////////oo//al/aa
[158]
Geoffrey E Hinton and Richard Zemel. "Autoencoders, Minimum Description Length and
Helmholtz Free Energy". In: Advances in Neural Information Processing Systems. Ed. by
neurips Cowan et al. Vol. 6. Morgan- Kaufmannn 1993. URL: https::/prrceedddnggg Seaper/1993/haae
9e3cf 0488ccf81a00576633563 1ateee3c99efffbbbo Abstract httttacct.htm]]]
[159]
Bin Dai and David Wipf. Diagnosing and Enhancing VAE Models. 2019. arXiv: 1903 05789
[cs.LG] LG]
[160]
Casper Kaae Sønderby et al. "Ladder Variational Autoencoders". In: Advances in Neural Infor-
mation Processing Systems. Vol. 29, Curran Associates, Inc., 2016. URL: httphuttstttttppppppearr http:///////aa/aa//a/a///////aaa
Obstract.htm)
[161] Arash Vahdat and Jan Kautz. "NVAE: A Deep Hierarchical Variational Autoencoder". In: Advances in Neural Information Processing Systems. Vol. 33, Curran Associates, Inc., 2020, pp. 19667-19679, URL: http::::ppp.. http:://pppppr.......o.e ctppspeccppppp...ppppppppp papapprrfllesse ptpppp/ppp00/////// 2t////2220/hhha/ 6B522666880ff733776666e66333717
[162]
Jakub Tomezak and Max Welling. "VAE with a VampPrior". In: Proceedings of the Twenty- First International Conference on Artificial Intelligence and Statistics. Ed. by Amos Storkey and Fernando Perez-Cruz. Vol. 84. Proceedings of Machine Learning Research. PMLR, Apr. 2018, pp. 1214-1223. URL: https://ppoceeddiiiiiiee mmr http:/ddiaaa......pprrrr//////// wwt.ess//v84tttccaaal8/3 html
[163]
Eric Nalisnick and Padhraic Smyth. "Stick-Breaking Variational Autoencoders". In: Interna- tional Conference on Learning Representations. 2017. URL: https: net/
forum?id-S1jmoon
[164] Gabriel Loaiza-
Loaiza-Ganem Ganem and John P Cunningham. "The continuous Bernoulli: fixing pervasive error variational autocncoders" In: Advances in Neural Information Pro- cessing Systems. Ed. by H. Wallach al. Vol. 32. Curran Associates, Inc., 2019. URL. https : proceedings neurips cc paper files / paper / 2019 file /
182798ac90012355677eee hate:/a55577bb6222222222-2------
pdf
[165]
Chethan Pandarinath et al. "Inferring single-trial trial neural population dynamics using sequential auto-encoders". In: Nature methods 15.10 (2018), pp. 805-815, DOI: 10. 1038/s41592
018-0109-9)
[166]
He Zhao et al. "Variational Autoencoders for Sparse and Overdispersed Discrete Data". In: Proceedings of the Twenry Third International Conference Artificial Intelligence and Statistics. Ed. by Silvia Chiappa and Roberto Calandra. Vol. 108. Proceedings of Machine Leaminn Research. PMLR, Aug. 2020, pp. 1684 1694. URL: https://procceeding.. mlr press/v108/zhao200 html
[167] Tim Salimans et al. "PixeICNN+:: Improving the PixeICNN with Discretized Logistic Mixture Likelihood and Other Modifications". In: International Conference on Learning Representations. 2017. URL:
net/forunt forffrommmiioooo AaddBJrFFGccgg
[168]
Danilo Rezende and Shakir Mohamed. "Variational Inference with Normalizing Flows". In: Proceedings of the 32nd International Conference on Machine Learning. Ed. by Francis Bach and David Blei. Vol. 37. Proceedings of Machine Learning Research. Lille, France: PMLR, July 2015, pp. 1530-1538. URL: https: htt::htts:://poottt h/tp://:/ppcceeiinsss. mlr. prwss v37/rezende1l html
[169]
Durk P Kingma al. "Improved Variational Inference with Inverse Autoregressive Flow". In: Advances in Neural Information Processing Systems. Ed. by D. Lee et al. Vol. 29, Curran Associates, Inc., 2016. URL: https / neurips cc paper_pppppppp ltpp:ffieeeeeeeeep//l files/
httml
[170]
Eric Jang et al. "Categorical Reparameterization with Gumbel--Softax"" In: International Conference on Learning Representations. 2017. URI.: htttts://tppereeete/e/
wttpr/ewwwwiieeeeefrurtreo net/forum?
id-rkE3y85ee
19

---

[171] Chris J. Maddison et al. "The Conerete Distribution: A Continuous Relaxation of Discrete Random Variables". In: International Conference on Learning Representations. 2017. URL: https://opeerrviier wttpaarrreeeeeeeeeorurrrdeeeeee net:/oorumm.ddsijj555.
[172] Jason Tyler Rolfe. "Discrete Variational Autoencoders". In: International Conference
on Learning Representations. 2017. URL: https : openreview net forum id
ryMxXPFex
[173] Arash Vahdat al. "DVAE++: Discrete Variational Autoencoders with Overlapping Transfor- mations". In: Proceedings of the 35th International Conference on Machine Learning. Ed. by Jennifer Dy and Andreas Krause. Vol. 80. Proceedings of Machine Learning Research. PMLR. July 2018, pp. 5035-5044. URL: httph: https://proceediinge html
mlr wrwss/ Satpss//oovvahaal1555 vahdat
[174] Yookoon Park et al. "Variational Laplace Autoencoders". In: Proceedings of the 36th In- ternational Conference Machine Learning. Ed. by Kamalika Chaudhuri and Ruslan Salakhutdinov. Vol. 97. Proceedings of Machine Learning Research. PMLR, June 2019, pp. 5032 5041. URL: https:: httpps://proceedings. mlr www.:........peeemme ppwss/v97/ http///.////... html
[175] Akash Srivastava and Charles Sutton. "Autoencoding Variational Inference For Topic Mod- els". In: International Conference on Learning Representations. 2017. URL: https //
openreview net aet//forum?dddBByooWkggg
[176] Emile Mathieu et al. "Continuous Hierarchical Representations with Poincaré Variational
Auto-Encoders" In: Advances in Neural Information Processing Systems. Vol. 32. Curran Associates, Inc., 2019. URL: https: : / proceedingseeseeeeeeee neurips cc paperr files/ /
paper/2019/ 1tt0://///////////////e///eeeeee http:///eeeeeee0000000000oo//-/0 0a00044b992201100774400071166683
Paper pdf
[177] Tim R. Davidson et al. /yyperspherical Variational Auto-Encoderss 2022. Xivi 1804. 00891
(stat.ML)
[178] Juno Kim et al. "1.3- -/rriaaittonnall Autoencoder: Learning Heavy-tailed Data with Student's 's t and Power Divergence". In: The Twelfth International Conference on Learning Representations.
2024. URL: httph:tttt///oo ptttps:://ppnreevee: wwtp:w.eeeeeeeeeeee
[179] Yixuan Zhang et al. Negative Binomial Variational Autoencoders for Overdispersed Latent Modeling. 2025. arXiv: 2508 2550205423 [cs.LG]
[180] Robbe LT Goris et al. "Partitioning neuronal variability". In: Nature neuroscience 17.6 (2014).
pp. 858 865. DOI: 10. 1038/ 3711
[181] Hayden R. Johnson et al. Inferring response times of perceptual decisions with Poisson
variational autoencoders. 2025. arXiv: 2511 11480 [q- ([-bio.NC)
[182] George Cybenko. "Approximation by superpositions of sigmoidal function". In: Mathemat- ics of control, signals and systems 2.4 (1989), pp. 303-314, DOI: 10. 10770000255512//0
[183] Chris Cremer et al. "Inference Suboptimality in Variational Autoencoders". In: Proceedings of the 35th International Conference on Machine Learning. Ed. by Jennifer Dy and Andreas Krause. Vol. 80. Proceedings of Machine Learning Research. PMLR, July 2018, pp. 1078- 1086. BRRL:https::/// httta::://proceeiiie
mlr. wwwpppeeeeeeeeeee 1::00//ceeee111e 1... html
[184] Charles O' et al. Compute Optimal Inference and Provable Amortisation Gap in Sparse Autoencoders. 2025. arXiv: 2411. 13117
[185] Rahul Krishnan et "On the challenges of learning with inference networks on sparse, -igh-dimensionl data". In: Proceedings of the Twenty-Firs, International Conference on Artificial Intelligence and Statistics. Ed. by Amos Storkey and Fernando Perez-Cruz. Vol. 84. Proceedings of Machine Learning Research. PMLR, Apr. 2018, pp. 143-151. URT.: https //proceedings mlr. ppesss/84//rrisaaantt
html
[186] Matthew D. Hoffman et al. "Stochastic Variational Inference". In: Journal of Machine Learning Research 14.40 (2013), pp. 1303-1347, URL: http://jmlr org wwwpii..rrg/aper//vi..
hoffman html
[187]
Arthur P Dempster et al. "Maximum likelihood from incomplete data via the EM algorithm". In: Journal of the royal statistical sociery: series B (methodological) 39.1 (1977), pp. 1-22. DOI: 10. 1111/j 2517 1977. tb01600.5
[188] Christopher M. Bishop. Partern Recognition and Machine Learning, New York: Springer. 2006. URL: http:: http://w.cooooffoooo com en
20

---

[189] Martin J. Wainwright and Michael I. Jordan. "Graphical Models, Exponential Families, and Variational Inference". In: Foundations and Trends in Machine Learning 1.1-2 (2008).
pp. 1-305. ISSN: 1935-8237. DOT: 10. 1561/2200000001
URT.: http::/dx doi. otto///.oooooooooo//
1561 1561/2200000001
[190]
Thomas P. Minka. Expectation Propagation for approximate Bayesian inference. 2013. arXiv:
1301. 2294 [cs.AI]
[191]
Timothy Hospedales et "Meta-learning in neural networks: A survey". In: IEEE trans- actions on pattern analysis and machine intelligence 44.9 (2021), pp. 5149-5169. DOI:
10. 10.1109/TPAMI, 2021 3079209
[192] Edgar Douglas Adrian and Yngve Zotterman. "The impulses produced by sensory nerve- endings: Part II. The response of Single End End-Organ", In: The Journal of Physiology (1926),
pp. 151-71. DO1: 100.1.1.1111 1113/ htt//pphysol... 1926. s2202220
[193] Malvin C Teich." "Fractal character of the auditory neural spike train". In: IEEE Transactions
on Biomedical Engineering 36.1 (1989), pp. 150-160. DOI: 10. 1109/10. 16460
[194] Michael N Shadlen and William T Newsome. "The variable discharge of cortical neurons: im- plications for connectivity, computation, and information coding" In: Journal of neuroscience
18. 10 (1998), pp. 3870-3896. DO1: 10. 1523/ JNEUROSCI. 18- 10- 03870. 1998
[195] Fred Rieke et al. Spikes: exploring the neural code. MIT press, 1999. URL.:
https //
mitpress wittesss..it.. wtt.:mmtt..u.rr/.mmmm (teddd/97702222881177
spikes/
[196] Wilson Truccolo al. "A point process framework for relating neural spiking activity spik- ing history, neural ensemble, and extrinsic covariate effects". In: Journal of neurophysiology
93.2 (2005), pp. 1074-1089, DOI: 10. 1152/jn. 00697 2004
[197] Zachary F Mainen and Terrence Sejnowski. "Reliability of spike timing in ncocortical neurons", In: Science 268.5216 (1995), pp. 1503-1506. DO1: /10 1 11/1 scwe/cciiccccceo/cc/ 7770778
[198] Daniel A Butts al. "Nonlinear computations shaping temporal processing of precortical vision". In: Journal of Neurophysiology 1 16..3 (2016), pp. 1344-1357.
DOI: 10 1152/jn
00878. 2015
[199] Alison I. Weber and Jonathan W. Pillow. "Capturing the Dynamical Repertoire of Single Neurons with Generalized Linear Models". In: Neural Compulation 29.12 (Dec. 2017),
pp. 3260-3289. ISSN: 0899-76677 DO1: 10. 1162/neco_a_01011
[200]
Shun-ichi Amari. Information geometry and its applications. Vol. 194. Springer, 2016. DOI:
10. 1007/978-44 17788-4-431--55- 451-55978-8
[201] S. M. Sherman and R. W. Guillery. "Functional organization of thalamocortical relays". In: Journal of Neurophysiology 76.3 (1996). PMID: 8890259, pp. 1367-1395. DOI: 10. 1152/
19969 76.3 1367
[202] P. Adams et al. "The role of the thalamus in the flow of information to the cortex". In: Philosophical Transactions of the Royal Society of London. Series B: Biological Sciences 357.1428 (2002), pp. 1695-1708. DOt: 10. 1098/rstb. 2002. 1161
[203]
Colin Blakemore and Elisabeth A Tobin. "Lateral inhibition between orientation detectors in the cat's visual cortex". In: Experimental brain research 15 (1972), pp. 439-440. DOI:
10. L.007BFF00234122
[204]
Arthur Szlam et al. "Structured sparse coding via lateral inhibition". In: Advances in Neural Information Processing Systems. Ed. by Shawe-Taylor et al. Vol. 24. Curran Associates, Inc., 2011. URL: https: :/ttttt:::////eeee p/wceeediooceee/eeoooooe neurips cc sc/paper__.. files/ hwtp://aper//// /01p1 Paper..da Hillel Adesnik and Massimo Scanziani. "Lateral competition for cortical space by layer- specific horizontal circuits". In: Nature 464.7292 (2010), pp. 1155-1160 DO1: 10. 1038/ nature08935
[205]
[206]
Liqun Luo. "Architectures of neuronal circuits". In: Science 373.6559 (2021). DO1: 10. 1126/ algennee.bbg288
[207]
Joseph Del Rosario et al. "Lateral inhibition in V1 controls neural and perceptual contrast sensitivity". In: Nature Neuroscience (2025). pp. 1-12. DOI: 10. 100.108///11113-02--01102
4
[208]
Jan Benda and Andreas V. M. Herz. "A Universal Model for Freke-Fre Adaptation". In: Neural Computation 15.1 11 (Nov. 2003), pp. 2523-2564. ISSN: 0899-7667, DOT: 10. 1162/ 0889976603522885063
21

---

[209] Arthur Hobson. "A new theorem of information theory". In: Journal of Statistical Physics 1 (1969), pp. 383-391. DOI: 10. L007/BF01106677
in Siguron [210] Christof Koch. Biophysics of Computation: Information Nennessonn Oxford University Press, Nov. 1998. ISBN: 9780195104912. DO1: 10. /1t..100/////00//000001010001000 000100001
[211] Kaiming He et al. "Deep Residual Learning for Image Recognition". In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). June 2016. URL: https: opwaaccesppccccccecee thecvf com/ wtte:onttttttteeeeeeeeeeeeeee cvpr 2016 html _He_Deep_
Residhal__eearang_ccce porpaaneg__..pe.22opppppaarprr ptt.rp..ppr.mmllll html
[212]
Angeliki Giannou et al. "Looped Transformers as Programmable Computers". In: Proceedings
of the 40th International Conference Machine Learning. Ed. by Andreas Krause et al. Vol. 202. Proceedings of Machine Learning Research. PMLR, July 2023. pp. 11398-11422.
URL:
mlr, pwwps/vv022...m. 11122/giaa gianoon33a....mmmmmmm html
[213] Liu Yang al. "Looped Transformers are Better Learning Learning Algorithms", In: The Twelfth International Conference on Learning Representations. 2024. URL: https
//openreview net/ wwt.:/ne//forruio http://oorrmmi---ttooo0SS0
[214] Ying Fan et al. "Looped Transformers for Length Generalization". In: The Thirteenth Interna- tional Conference Learning Representations, 2025. URL: htttps://ppereeee neeteeteeeieeeeeeeee/eeee
Forum?id=2ediggk8yol
[215] Trenton Bricken et al. "Emergence of Sparse Representations from Noise". In: Proceedings of the 40th International Conference Machine Learning. Ed. by Andreas Krause al. Vol. 202. Proceedings of Machine Learning Research. PMLR, July 2023, pp. 3148-3191. URL:
mll. wttp:ppss//2222/r.rreeem222/ /ttp////dd.aaa.a.....er///////// http://bbriiieeaaa.ahhccc html
[216] Michael Teti. I.CA J.C.PPTTrrc. [Computer Software https doi
11578/ URL. httts://doi.orr org/10
20230728 4 June 2023. DOT: 10. 11w78/dc..ccco 20w.d/d..222222....
11578/dc. 20230728.
[217] Diederik P Kingma and Jimmy Ba. "Adam: A method for stochastic optimization". In: (2014).
arXiv: 1412. 6980 LG]]
[218] Ilya Loshchilov and Frank Hutter. "SGDR: Stochastic Gradient Descent with Warm Restarts". In: International Conference on Learning Representations. 2017. URL: https
//openreview net wwwt/ffoouiiddaaaeeeeeeee Skg88Sxxx
[219] Samuel R. Bowman et al. "Generating Sentences from a Continuous Space". In: Proceedings of the 20th SIGNLL Conference Computational Natural Language Learning. Berlin, Germany: Association for Computational Linguistics, Aug. 2016, pp. 10-21. DOI: 10 18653/ v1/K16-1002
[220] Hao Fu al. "Cyclical Annealing Schedule: A Simple Approach Mitigating KL Vanishing". In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers). Minneapolis, Minnesota: Association for Computational Linguistics, June 2019, pp. 240-250. DOL:
[221] Y. Lecun et al. "Gradient- learning applied to document recognition". In: Proceedings of the IEEE 86.11 11 (1998), pp. 2278 2324.. DO1: 10. 1109/1 726791
[222] Mengehen Zhu and Christopher J Rozell. "Visual nonclassical receptive field effects emerge from sparse coding in dynamical system". In: PLoS computational biology 9.8 (2013), e1003191. DO1: 10. 1 httr1//jooaal...mm pcbi. 1003191
[223] Matteo Carandini and David J Heeger. "Normalization as a canonical neural computation".
In: Nature reviews neuroscience 13.1 (2012), pp. 51-62. DO1: 10. 1038 nrn3136
[224] Gregory Cohen et al. "EMNIST: Extending MNIST to handwritten letters". In: 2017 inter- national joint conference on neural networks (IJCNN. IEEE. 2017, pp. 2921-2926. DOI:
10. 1109/ IJCNN. 2017. .7966217
[225] Brenden M Lake al. "Human-level concept learning through probabilistic program induc- tion".. In: Science 350.6266 (2015), pp. 1332-13330 1312.133...11 DOI: 10. 1126/ 1126/science science...cccccccc/c.eo..rr aab3050 [226] Patryk Chrabaszez et al. A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasets. 2017. arxiv: 1707. 08819
[vs
22

---

[227] Muhammad Ghifary et al. "Domain Generalization for Object Recognition With Multi-Taa Autoencoders". In: Proceedings of the IEEE International Conference on Computer Vision (ICCV). Dec. 2015. URL: https: openaccess thecvf com content. iccv 2015/ html/Chifary Bawthhhaaaaaaaaaiiaaaaa DwwaDoomaaiiiieeee Generalization for ICCV 2015 ___l220111_________________11 .html
[228] Honglak Lee et al. "Sparse deep belief net model for visual area V2". In: Advances in neural information processing systems 20 (2007). URL: https: pwwprs.pppppssspee nips cc paper pdf Robin Rombach et al. "High- Image Synthesis With Latent Diffusion Models". In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion (CVPR). June 2022, pp. (0684-10695 URL: /ectps://opencee teeeessssteeeee thecvf com/ content/ CVtR:022/R22222//// html Rwwbacommahaaaaeeeeeeeeeeeeeeeee Highh Beiigh-Resoline www.eeeesinnnnnn. Image_ SarrieesSyttttsssssse With
[229]
Latent_Diffusi.or Models_ wwwpp__2222222pperrrr
[230] Ling Yang et al. Diffission Models: A Comprehensive Survey of Methods and Applications.
arg/abs//200 0tt.//299900000.666 [231] Horace B Barlow et al. "Possible principles underlying the transformation of sensory mes-
2024. arXiv: 2209.00796 [C] URL:
sages". In: Sensory communication 1.01 (1961), pp. 217-233.
[232] Robert Price. "A useful theorem for nonlinear devices having Gaussian inputs". In: IRE Transactions on Information Theory 4.2 (1958), pp. 69-72. DOI: 10. 1109 / TIT. 1958
1057444
[233] Georges Bonnet. "Transformations des signaux aléatoires a travers les systemes non linéaires sans mémoire". In: Anmales des Télécommunications. Vol. 19. 9. Springer. 1964, pp. 200-220. DO1: 10. 10010.0000////0077220
[234] Edgar Adrian "The activity of the nerve fibres". In: Nobel lecture (1932). URL: https
[235]
Donald H Perkel and Theodore H Bullock. "Neural coding". . In: Neurosciences Research
Program Bulletin (1968). URL: hhttss://ttts: nasa. gow/ http://tttooo/////022311111
[236]
Ehud Zohary et al. "Correlated neuronal discharge rate and its implicationn for psychophysical
performance". In: Nature 370.6485 (1994), pp. 140-143. Dor: 10. 1038/370140aa
[237]
Vivienne Sze et al. "Efficient processing of deep neural networks: A tutorial and survey". In: Pruceedings of the IEEE 105.12 (2017), pp. 2295-2329. DOI: 10. 1109//PROCC.. 2017
2761740
[238]
Yu-Hsin Chen et al. "Eyeriss: An energy-efficient reconfigurable accelerator for deep convo- lutional neural networks". In: IEEE journal of solid-state circuits 52.1 (2016), pp. 127-138. DOI: 10. 1109 JSSC. 2016 2616357
[239]
Xuexing Du al. "A generalized Spiking Locally Competitive Algorithm for multiple optimization paaimization roblemnssst problems".. In: Neurocomputing 624 (2025), p. 129392. ISSN: 0925-2332. DOT:
https://doi. http:://doi.or/10000 1016/] j neucom. 2025. 129392
[240]
W. K. Hastings. "Monte Carlo sampling methods using Markov chains and their applications".
In: Biometrika 57.1 (Apr. 1970), pp. 97- ISSN: 0006-3444. DOT: 10. 1093 bi /57
1.97
[241]
Alan E. Gelfand and Adrian F. M. Smith and. "Sampling- -aasee Approaches to Calculating Marginal Densities". In: Journal of the American Staristical Association 85.410 (1990), pp. 11.98--09 DOI: 10. 1080/01621459. 1990. 10476213
[242] Steve Brooks et al. Handbook of markov chain monte carlo. CRC press, 2011. DOI: 10 1201/b10905
[243] Christian A. Naesseth et al. "Elements of Sequential Monte Carlo". In: Foundations and
Trends@ in Machine Learning 12.3 (2019), pp. 307-392, ıSSN: 1935-8237. DOE: [00 1561/ 2200000074 URL: http: doi org/ 10. 1561 1561/2200000074
[244] J6zsef Fiser al. "Statistically optimal perception and learning: from behavior to neural
representations" " In: Trends in cognitive sciences 14.3 (2010), pp. 119-130. DOI: 10. 1016/
j.tics.207 2010 000010001.000 01.003]
[245] Lars Buesing et al. "Neural Dynamies as Sampling: A Model for Stochastic Computation in Recurrent Networks of Spiking Neurons". In: PLOS Computational Biology 7.11 1 (Nov. 2011), pp. 1-22. DOL: 10. 1371/journal. wcwpllpppppc..... 1002211
[246] Ralf M Haefner et al. "Perceptual decision-making as probabilistic inference by neural sampling". In: Newron 90.3 (2016), pp. 649-660. Dor: 10. 1016/j .neuron. 2016. 03. 020
23

---

[247]
Rodrigo Echeveste et al. "Cortical- dynamics in recurrent circuits optimized for sampling- based probabilistic inference". In: Nature neuroscience 23.9 (2020), pp. 1138-1149. DOI:
10.
[248]
Camille Rullán Buxó and Cristina Savin. "A sampling-based circuit for optimal decision making". In: Advances in Neural Information Processsig Systems. Vol. 34. Curran Associates,
Inc., 2021, pp. 14163-14175 URL: http:pt:/
neurips wwtp:ripppppppppp///p..aa cc/paper/2021/ http://ettt.--Atttttttt....e
[249] Cristina Savin and Sophie Deneve. "Spatio-temporal Representations of Uncertainty in Spiking Neural Networks". In: Advances in Neural Information Processing Sys- tems. Ed. by Z. Ghahramani et al. Vol. 27. Curran Associates, Inc., 2014. URL: https : proceedings neuripa cc / /aaprr files paper / 2014 file / 20a124443ee11944455CC33f22222222
Paper pdf
[250] Dylan Festa et al. "Neuronal variability reflects probabilistic inference tuned to natural image statistics". In: Nature communications 12.1 (2021), p. 3635. DOI: 10. 100114///44446 23838-x
[251] Suhas Shrinivasan et al. "Taking the neural sampling code very seriously: A data-driven approach for evaluating generative models of the visual system". In: Advances in Neural Information Processing Systems. Ed. by A. Oh et al. Vol. 36. Curran Associates, Inc., 2023. pp. 21922 219599. URL: https: / proceedingg neurips cc/ wwwppppp..... files/ filesspaperrrrrr 2023/ htsp/hahhhha (055dd9fdd5c7565550 60133000 436304c623333333 10 Abstract- Conference..cceeeoo htmm
[252]
Wen-Hao Zhang al. ""aappling Bayesian inference in recurrent circuits of stochastic spiking neurons". In: Nature communications 14.1 (2023), p. 7074. Do1: 10. 101.1..333/441455
023- 017433333-
[253]
Ralf M. Haefner et al. How does the brain compute with probabilities? 2024. arvi 2409 02709 [C]]
[254]
Yashar Ahmadian et al. "Analysis of the Stabilized Supralinear Network". In: Neural Compu- tation 25.8 (Aug. 2013), pp. 1994-2037. ISSN: 0899-7667. DO1: 10. http:/EECO_aa_0000
[255] Guillaume Hennequin et al. "The dynamical regime of sensory cortex: stable dynamics around a single stimulus-tuned attractor account for patterns of noise variability".. In: Neuron 98.4 (2018) 8), pp. 846-860. DOI: 10. 10w6/j.ij.... nwwron.e.nnnomm..oommmm 2018. 04.004..7
[256] Yi-An Ma et al. "A Complete Recipe for Stochastic Gradient MCMC". In: Advances in Neural Information Processing Systems. Ed. by Cortes et al. Vol. 28. Curran Associates,
Inc., 2015. URL: https: //poceeddgs//oceeinng neur iss cc paperr files/ walls/paper/////ee /00pr /015
paper
hash/9a4400501
Abteea--bbbteeee wwwstbacttt..... html
[257] Tim Salimans et al. "Markov Chain Monte Carlo and Variational Inference: Bridging the
Gap". In: Proceedings of the 32nd International Conference Machine Learning. Ed. by Francis Bach and David Blei. Vol. 37. Proceedings of Machine Learning Research. Lille,
France: PMLR, July 2015, pp. 1218 1226. URL.: https: https::/pprceeedii
mlr press/
337/salimans html
[258] Richard D. Lange et al. "Interpolating between sampling and variational inference with infinite stochastic mixtures". In: Proceedings of the Thirty-Eighth Conference on Uncer- tainty in Artificial Intelligence. Ed. by James Cussens and Kun Zhang. Vol. 180. Proceed- ings of Machine Learning Research. PMLR, Aug. 2022, pp. 1063-1073. URL: https //proceedings.oo mlr Vixres/v180/1anggee htmml
[259] Jason Wei et al. "Chain-of-Thouge Prompting Elicits Reasoning in Large Language Models". In: Advances in Neural Information Processing Systems. Ed. by Koyejo et al. Vol. 35. Cur- ran Associates, Inc., 2022, pp. 24824-24837 URL.: https: http::::s//ppceeeddinee mewriiinnnnnnnnnr.p.eee...ee neurips. cc/ paper_ wttrrffflessppppr///e http:///w/w///////.../.o/ooo//a. Conference html
Abstrratt
[260] Shunyu Yao et al. "Tree of Thoughts: Deliberate Problem Solving with Large Language Mod- els". In: Advances in Newral Information Processing Systems. Ed. by A. Oh et al. Vol. 36. Cur- Associates, Inc., 2023, pp. 11809-11822. URL: pestpa::/proceeediie Abstttatttbbtttt...oo......a
nwwr cc/
Conference html
[261] Aayush Karan and Yilun Du. Reasoning with Sampling: Your Base Model Smarter Than You Think. 2025. 2510. 14901 [Cs.LG]
[262] Guan Wang et al. Hierarchical Reasoning Model. 2025. arXiv:
21733 [Cs.AI]
24

---

[263]
Yeskendir Koishekenov et al. Encode, Think, Decode: Scaling test-time reasoning with recursive latent thoughts. 2025. arXiv: 07310.03358 [cs. [.G]
[264]
Dileep George et al. "A generative vision model that trains with high data efficiency and breaks text-based CAPTCHAS". In: Science 358.6368 (2017), eaag2612. DOI: 10. 1126/
science. aag2612
[265]
Dileep George et al. "A detailed theory of thalamic and cortical microcircuits for predictive visual inference". In: Science Advances 11.6 (2025), cadr6698. DOI: 10. 1126 sciadv
adr6698
[266]
Francois Chollet et al. ARC Prize 2024: Technical Report. 2025. arXiv: 2412 04604
[cs. AI]
[267]
Isaac Liao and Albert Gu. ARC-AGI Without Pretraining. Mar. 2025. URL: https : //
iliao2345. github httchuubio/bboo_ppottttttt aaattt//rrccee wtt.:/.....iioooopoeeeaaaaaaaaaa
arc_ agi_
without_prees pethouuppeerraiing... html
[268]
Yuxi Liu et al. The Serial Scaling Hypothesis. 2025. Xiv: 2507 12549 [cs. LC]
[269]
Tri Dao al. "FlashAtteenion: Fast and Memory-Efficient Exact Attention with Iw-awereess"". In: Advances in Neural Information Processing Systems. Ed. by S. Koyejo et al. Vol. 35. Curran Associates, Inc., 2022, pp. (6344-16356) URL: https : / proceedings neurips cc / paper files / paper / 2022 hash / .ttll
$7a57c32e20fa0ar/ htt.:/f00aaaa022222011ee666ee6e. 077
[270] Albert Gu and Tri Dao. "Mamba: Linear- Time Sequence Modeling with Selective State Spaces". In: First Conference on Language Modeling. 2024. URL: htttps://openrevien
hot/Forrum?io-tEEsswwl......
[271] Thomas Schmied et al. A Large Recurrent Action Model: xLSTM enables Fast Inference for Robotics Tasks. 2025, arXiv: 24100 22391 LG]
[272] Reiner Pope al. "Efficiently Scaling Transformer Inference", In: Proceedings of Ma- chine Learning and Systems. Ed. by D. Song et al. Vol. 5. Curan. 2023, pp. 606-624.
URL: https: / / pro.eediroeeeeeeeeeoe mlsys org paper files / paper 2023 / hash / -bte Abstract mlsys2023.c.mm html
[273]
Charlie Chen et al. Accelerating Large Language Model Decoding with Speculative Sampling.
2023. arXiv: 2302 01318 (cs.CL)
[274] Stephen Grossberg. "Competitive learning: From interactive activation to adaptive resonance". In: Cogmitive Science 11.1 (1987), pp. 23-63. ISSN: 0364-0213. DOI: https:://oo...... org/
10. 0216/S0034-0213(817 80025-3
[275]
Guillaume Bellec et al. "A solution to the learning dilemma for recurrent networks of spiking neurons". In: Nature communications 11.1 (2020), p. 3625. DOT: 10. 1088//447444400000 17236-y
[276] Alexandre Payeur et al. "Burst-dependent synaptic plasticity coordinate learning in hierarchical circuits". " In: Nature neuroscience 24.7 (2021), pp. 1010-1019. DO1: 10. 1038/
841593-021- 02100777700000
[277] James R. Whittington and Rafal Bogacz. "An Approximation of the Error Backpropagation Algorithm in Predictive Coding Network with Local Hebbian Synaptic Plasticity", In: Neural Computation 29.5 (May 2017), pp. 1229-1262. ISSN: 0899-7667. DO1: 10. 1162/
_000___00949
[278] Sen Song al. "Highly Nonrandom Features of Synaptic Connectivity in Local Cortical Circuits". In: PLOS Biology 3.3 (Mar. 2005). DOI: 10. 1371/Journal. pbii..ii.............. 0030068
[279]
James CR Whittington and Rafal Bogacz. "Theories of error back-propagation in the brain".
In: Trends in cognitive sciences 23.3 (2019), pp. 235 250. DO1: 10. 1016/j
(1001//..iis...... tics. 2018
12.005)
[280]
Timothy P Lillicrap et al. "Backpropagation and the brain".. In: Nature Reviews Nearoscience 21.6 (2020), pp. 335-346. DO1: 10. 100800034444 008//411/8---.0 000-1158880000077--21
[281]
Katharina Dobs et al. "Brain-like functional specialization emerges spontaneously in deep neural networks". In: Science Advances 8.11 (2022), eab18913. DOI: 10. 1126/sciady
ab18913
[282]
Wulfram Gerstner et al. "Eligibility traces and plasticity on behavioral time scales: experi- mental support of neohebbian fhree-factor learning rules". In: Frontiers in neural circuits 12 (201 8), p. 53. DOI: 3389/fncis 2018. 000530
25

---

[283]
Timothy P Lillicrap et al. "Random synaptic feedback weights support error backpropaga- tion for deep learning". In: Nature communications 7.1 (2016), p. 13276. DO1: 10. 1038/ ncomms 13276
[284]
Tahereh Toosi and Elias Issa. "Brain-like Flexible Visual Inference by Harnessing Feed- back Feedforward Alignment". In: Advances in Neural Information Processing Systems. Ed. by A. Oh et al. Vol. 36. Curran Associates, Inc., 2023, pp. 56979-56999 URL: https : / proceedings neurips cc / paper files / paper / 2023 file / waw.aaaaaaeeeeeo-appe-ooooeerree Conffrrr--oooeeeccee.e.. pdf
[285] Andre M Bastos et al. "Canonical microcircuits for predictive coding". In: Neuron 76.4
(2012), Pp. 695-711. DOI: 10. 1016// nww.on.ro... meuron..012.... 10.22.0.033
[286]
Anastasia Dimakou et al. "The predictive nature of spontancous brain activity across scales and species". In: Neuron (2025). DO1: 10. 1016/1 1t1./j.neuron neuron. 2025 009.
[287] Belle Liu et al. "Predictive encoding of motion begins in the primate retina". In: Nature neuroscience 24.9 (2021), pp. 1280-1291, DOI: 10. 1038//411993002-00889-3
[288] Georg B Keller and Thomas D) Mrsic-Flogel. "Predictive processing: canonical cortical computation". In: Neuron 100.2 (2018). DOI: 10. 1016/j neuron. 2018. 10.10000) 003
[289] Tsai- Wen Chen al. "A Map of Anticipatory Activity in Mouse Motor Cortex". In: Neuron 94 (2017), 866-879.e4. DOT: 10. 1016/] neuron. 2017
[290] Linxing Preston Jiang and Rajesh P. N. Rao. "Dynamic predictive coding: A model of hierarchical sequence learning and prediction the neocortex". In: PLOS Computational Biology 20.2 (Feb. 2024), pp. 1-30. DO1: 10. 10.11/ 10711/3orraaaa... pcbi 1011801
[291] David Ha et al. "HyperNetworks" In: Internattional Conference on Learning Representations. 2017. URL: https: btttpp:://openreviees http://wwwww.eeeetooooroddrdrraa
[292] Beren Millidge et al. "Predictive coding networks for temporal prediction"" In: PLOS Compu- lational Biology 20.4 (Apr. 2024). pp. 1-31. DOT: 10. 1371/ pcbi 10011811
[293]
Fierre-Etienae H Fiquet and Eero P Simoncelli. "A polar prediction model for learning to represent visual transformations" In: Thirty Conference on Neural Information Processing Systems. 2023, URL: http:st:
www.:eennfffrrnnaeeo idd hy-hyps
[294] Olivier J Hénaff et al. "Perceptual straightening of natural vidcos". In: Nature neuroscience 22.6 (2019), pp. 984-991. DO1: 10.
[295] Daniel Felleman and David C Van Essen. "Distributed hierarchical processing in the primate cerebral cortex". In: Cerebral Cortex 1.1 (1991), pp. 1-47. not: 10. COB03//EERRR/1.. 1.1
[296] Richard D Lange et al. "Bayesian encoding and decoding as distinct perspectives on neural coding". In: Nature Neuroscience 26.12 (2023), pp. 2063-2072. DOI: 10. 1038/ 023- 01458-6
[297] William Bechtel and Leonardo Bich. "Grounding cognition: heterarchical control mechanisms
in biology". In: Philosophical Transactions of the Royal Society B: Biological Sciences
376. 1820 (2021), p. 20190751. DO1: 10. w098/rstbb... 2019.
[298] Jeff Hawkins et al. Hierarchy or Heterarchy? ? A Theory of Lang-Range Connections for the Sensorimotor Brain. 2025. arviv: 2507 05888 [q- (q-bio.NN))
[299] Jay Hegde and Daniel Felleman. "Reappraising the functional implications of the primate visual anatomical hierarchy". In: The Neuroscientist 13.5 (2007), pp. 416-421. Do1: 10
1177/107385840736222
[300] Tommaso Salvatori et al. "Learning on Arbitrary Graph Topologies via Predictive Coding". In: Advances in Neural Information Processing Systems. Ed. by S. Koyejo et al. Vol. 35. Curran Associates, Inc., 2022, pp. 38232-38244. URL: h paper_files/pppor/222///ea
neurips cc/ Absttactt. org.a
Conference html
[301] Jacob A. Westerberg and Pieter R. Roelfsema. "Hierarchical interactions between sensory cortices defy predictive coding". In: Trends in Cognitive Sciences (2025). ISSN: 1364-6613.
DO1: 1016/ tiictii.... 2025.02200 09. 018
[302] Victor Boutin al. "Pooling strategies in V1 can account for the functional and structural diversity across species". In: PLOS Computational Biology 18.7 (July 2022), pp. 1-21. DOI: 10. 10.1371/journal. pcbii. pcbi 1010270
26

---

[303] Kumihiko Fukushima. "Neocognitron: A self organizigg neural network model for a mecha- nism of pattern recognition unaffected by shift in position". In: Biological cybernetics 36. 4 (1980), pp. 193-202. DOT: 10. 1007/BF00344251
[304]
Bugene M. Izhikevich. "Polychronization: Computation with Spikes". In: Neural Computation
18.2 (Feb. 2006). pp. 245- 2222. ISSN: 0899-7667. DO1: 10. 10e11162/0880 011116//8996666775333333
[305]
Camille E. Rullán Buxó and Jonathan W. Pillow. "Poisson balanced spiking networks". In:
PLOS Computational Biology 16.11 (Nov. 2020), pp. 1-27, Dol: 10. 1371/jounaal pcbi
1008261 URL: battss://dot org/ 10. / ptt.p..............
[306]
Yusi Chen et al. "Predictive sequence learning in the hippocampal formation". In: Neuron
112.15 (2024), pp. 2645-2658 Dor: 10. nner//nneuroon.....mm 20..200224..00000 02.024..0000000
[307] T. Anderson Keller et al. A Spacetime Perspective on Dynamical Computation in Neural
Information Processing Systems. 2024. arXiv: 2409 13669 [q-bio bN]
[308] Daniel LK Yamins al. "Peroonmance-ppimiized hierarchical models predict neural responses
in higher visual cortex". In: Proceedings of the National Academy of Sciences 111.23 (2014), pp. 8619-8624. DO1: 10. 1073//nas..... 1403112111
[309] Matteo Carandini et al. "Do we know what the early visual system does?" In: Journal of Neuroscience 25,46 (2005), pp. 10577-10597) Dou: 10. 1523/ JNEUROSCI 3726 05. 2005 [310] Michael C.-K. Wu al. "COMPLETE FUNCTIONAL CHARACTERIZATION OF SEN- SORY NEURONS BY SYSTEM IDENTIFICATION" In: Annual Review of Neuroscience 29. Volume 29, 2006 (2006). pp. 477-505 ISSN: 1545-4126. DOI: https:://oi..rrg
1146/ aatd/vnnurev..mm neuro. 29, 051605. 113024
[311] Bruno A. Olshausen and David J. Field. "How Close Are We to Understanding V1?" In: Neural Computation 17.8 (Aug. 2005), pp. 1665-1699, ISSN: 0899-7667. DO1: 10. 1162/
0899766054026639
[312] Nancy Kanwisher al. "Using artificial neural networks ask 'why' questions of minds and brains". In: Trends in Neurosciences (2023). D01: 10. 1016/j tins 2022. 12. 008 [313] Fabian H Sinz et al. "Engineering less artificial intelligence". In: Neuron 103.6 (2019). pp. 967-979, DOI: 10. 1016/j neuron 2019. 08.088.000 034
[314] Mackenzie Weygandt Mathis et al. "Decoding the brain: From neural representations to mechanistic models". In: Cell 187.21 (2024), pp. 5814-588 DOr: 10. 1016/ cell. 2024
088.011
27

---

Appendix: Brain-like Variational Inference
Extended Background: Philosophical, Historical, and Theoretical
29
A.I
Perception inference
29
A.2
Bayesian posterior inference
29
A.3
Variational inference and deriving the ELBO objective
29
Three distributions, one ELBO
30
A.5
Two ways carve up the ELBO
................................
31
A.6
Gaussian Variational Autoencoder (G- VAE
................................
A.7
Poisson Variational Autoencoder (P- VAE)
32
A.8
Amortized versus iterative variational inference
32
A.9
Predictive coding as variational inference
A. 10 Variational free energy minimization unifies machine learning and neuroscience
35
Extended Theory: Motivations, Derivations, and Discussions
36
B.1 What do we mean by "prescriptive"?
36
B
B.2
Why Poisson?
37
B.3
Poisson Kullback-Leine (KL) divergence
37
Fisher information matrix
38
B.5
Interpreting the terms in Poisson dynamics equations
41
B.6
Extension nonlinear decoders
42
B.7
Extension Gaussian posteriors
42
B.8
Biological interpretation of the standard predictive coding dynamics
45
B.9
Free energy for sequences
46
10
Static versus online inference
46
Extended Experiments: Methodological Details and Additional Results
48
Architecture, Hyperparameter, and Training details
48
C.2
Convergence detection procedure
49
iP-VAE exhibits cortex-like dynamics: contrast-dependent response latency
Reconstruction-spaartty with deterministic MAP decoding, and dependence on
51
C.5
Reconstructionnspaarity performance across latent dimensionalities)
53
C.6
Reconstruction performance and downstream classification accuracy on MNIST
55
Runtime comparison: iterative versus amortized inference.
56
C.8
iP. VAE cxhibits strong out-of -iitt---iittiittionn (OOD) generalization
58
1P. IP-VAE scales up to complex, high-dimensional color image datasets
63
1. VPE generates novel samples through an iterative procedure
64
D
Extended Discussion
65
summary of our prescriptive framework, and how it relates to BLR and BONG
65
D.2
Three possible explanations for the effectiveness of IP-
67
D.3
Hardware-cfficien implementation and real-world deployment potential of iP_ VAE
67
D.4
Relationship to sampling based inference methods
68
D.5
Limitations and future work
69
28

---

A Extended Background: Philosophical, Historical, and Theoretical
In this appendix, we present additional background information that complements the main text (section[2) 2 First, we explore the historical and philosophical perspectives underlying our work. We then connect these foundations to recent developments while providing pedagogical explanations of essential statistical concepts. Finally, we review existing machine learning and neuroscience models as instances of free energy minimization, culminating model taxonomy shown in Fig.
A.1 Perception as inference
The idea that perception is an inferential (rather than passive) process has been around for centuries. Early thinkers described it as process of "unconscious inference" about the hidden causes of sensory inputs [T] 2]. This inference relies on two key components: (i) incoming sensory data, and (ii)
priori knowledge about the environment [37]] Prior world knowledge can be acquired either through subjective experience [138 [10]. or innately built through evolutionary and developmental processes [141 (43). As recognized in early psychology (144), and later formalized in theoretical and computational frameworks, perception emerges when these two sources-Senson evidence and
prior expectations are integrated to infer the most likely state of the world. Thus, brains actively "construct" their perceptions 145
This framework remains major cornerstone of theoretical neuroscience, and has inspired many influential theories across the field 7 22 | 24 32 54| 146 149). Although these theories go by Of particular relevance
different names, they are unified by the core idea of pperception-a--infeeeeen
to our work are sparse coding [54]
55
150 151, predictive coding
[2 33 146
152]. and the free
energy principle 23 25
148
149
153
Ideas such as variational inference from statistics [4] 14 17 and variational autoencoders (VAEs) from machine learning (18 closely parallel those from theoretical neuroscience. While there has been increasing dialogue between these fields, opportunities remain for deeper integration. Our work speaks directly to this overlap by offering theoretical synthesis of these ideas (Fig. 2 as well as introducing a new family of brain-inspired machine learning architectures with practical benefits.
A.2 Bayesian posterior inference
In statistical terms, perception can be modeled as a problem of Bayesian posterior inference [1O We start with a generative model, po (a:), that assigns probabilistic beliefs to observations, x E (e.g., images), through invoking K K-dimensional latent variables,
ʃ po (2r dz
/ conditional likelibood
Po (xc)
po (o(2)) po(z) dz,
marginal likelihood (a.ka model evidence)
joint distribution
prior
where are the adaptable parameters of the generative model (e.g., neural network weights or synaptic connection strengths in the brain). Here, p=(z) represents prior beliefs over latent causes, and pa (æ/=2) the conditional likelihood of observations given latents.
Posterior inference inverts this generative process to determine which latent configurations are most likely given an observation. Apply Bayes' rule to get:
(m =) pe (x)
Po po (zol2))((2)) [ z ()) dz
(0(2)
(10)
(x)
posterior
This posterior distribution is analytically intractable in all but the simplest cases due to the integration over all latent configurations in the denominator. This necessitates approximate inference methods.
A.3 Variational inference and deriving the ELBO objective
Variational inference powerful approximate inference method that transforms the posterior inference problem, eq. (0), into an optimization task (15 Specifically, one introduces variational density, (= n. parameterized by á, which approximates the true posterior. The goal is to minimize the Lullback-Leible (KL) divergence between go (z|xx) and the true posterior pa (z ())
29

---

This optimization leads to the standard Evidence Lower BOund (ELBO) objective [5 17 which can be derived starting from the model evidence follows:
lo log (a:) (o(2)az)dz
/ (0(2)))) Po
(=|m)
log
log + log po log po (z|xz
log (ar )o (2)2))
Egrga(s(a)
Po
(ax, x)
(x
log
(11)
Po (2 (2) [z)
Ps (ar, go
(z=gu(=(m)
log
Po (z (x) g
Po (ar, 2)
2o 2 [z] ax)
[zlqs(=|a)
log
+ log
2: x
pa (= xx(x))
Po (ax x) log
9 (2 xx)
[z=gu(=|m)
+ [z~go(=(x)
log
2o (z) (x))
pe (z [x)
Po (ax x) log
(z
+ DKL. (z(2)) (o)
(=(=))))
ELBO(#;9.0)
This concludes our derivation of eq. ① All we did was multiply by one twice (once by inserting
[=o ddz and once by introducing the ratio (a(a)a) followed by a few algebraic rearrangements.
(=(m)
Importantly, eq. (I) holds for any (z(2(z) [z) that defines proper probability density. Compared to the typical derivation using Jensen's 's inequality [15), this derivation yields a stronger result [20]: An exact decomposition of the model evidence into the ELBO plus the KL divergence between the approximate and true posterior.
A direct consequence of this equality is that taking gradients of eq. T with respect to á shows that maximizing the ELBO minimizes the KL. divergence, since the modelevvdenm on the left-hand side is independent of Φb Thus, ELBO maximization improves posterior inference quality. corresponding to more accurate perception under the perception-as-inferen framework (appendix A.I
A.4 Three distributions, one ELBO
To fully specify the ELBO in eq. ( we must make choices about the approximate posterior (z(z)))) and the joint distribution po (ar, z), which further decomposes po (ax, =) po (z) (2)po((r z).
Therefore, we are faced with a total of three independent distributional choices: approximate posterior, prior, and likelihood!!
In both neuroscience and machine learning, it is common practice to assume all three distributions Gaussian. These critical assumptions are frequently adopted by convention rather than through theoretical justification. We emphasize that these distributions are entirely at the discretion of the modeler, and there are fundamental constraints requiring them to be Gaussian in all cases.
In the online inference setting, both the prior and approximate posterior (typically) become one family, reducing the independent choices into two: approximate posterior, and likelihood.
30

---

A.5
Two ways to carve up the ELBO
There are common ways of expressing and interpreting the ELBO. The first, more popular in the machine learning community, is the VAE loss decomposition (20]:
( po(
EL; ()
E (xloa)
log
4s (0(2 (2)
+ E Bzzg (2/x) log [z(z)
E (=
log Pogpa (z]lz)
(12)
go (z(x))))
E (x
log po
DxL (z=(z(a)) [P(
Resonstruction term (distortion)
KL term (coding rate)
This view emphasizes reconstruction fidelity and latent space regularization. The reconstruction term can be interpreted as distortion measure, quantifying how well the latent code 2 can explain information rate,
the input æ through the generative model. The KI. term, by contrast, acts measuring how much input-dependent information is encoded in the posterior qo (=(xz) beyond what already present the prior po (=).. In other words, the KL quantifies the coding cost of representing via 2, and reflects the capacity of the latent space to capture novel, stimulus-specific structure. This interpretation is closely related classical rate-distortion theory [54 155
and has been formalized
in the context of VAEs by Alemi et al. (77).
The second view, more aligned with theoretical neuroscience and physics, splits (negative) ELBO as: H. (=|xx]]
-ELBO(2; 0) Flx;0, 0) (g)gg(=)=) - log pa
x)
(13)
Energy
Entropy
where H[g]] - g log is the Shannon entropy.
This carving is analogous to the concept of Helmholtz free energy from statistical physics [156 158 where minimizing free energy involves reducing energy while preserving entropy (i.e., maittaning uncertainty). Below, we will use this decomposition to show that the predictive coding objective of Rao and Ballard [22] can be directly derived from eq. (13) under specific distributional assumptions.
A.6 Gaussian Variational Autoencoder (G-VAE)
The standard Gaussian VAE (G-VAE) represents foundational model in the VAE family, where all three distributions are factorized Gaussians [20). To simplify the model, the prior is typically fixed as standard normal distribution with zero mean and unit variance: pa pa(z) ~0 1).
The key innovation of VAEs lies in how they parameterize the approximate posterior and likelihood
distributions using neural networks:
Approximate posterior. The encoder neural network transforms each input into parameters for the posterior distribution. Specifically, outputs both a mean vector and a variance vector: enc(x; Φ) X(x) (μ(x), o² (x)). These parameters fully specify the approximate posterior: (z(2)²) = (2; (xx), 0 (z)). Typically, neural networks are made to output log a, which is exponentiated obtain the variance. This choice ensures the variance always remains positive.
Likelihood. The decoder neural network maps sampled latent variables back reconstructions in the input space, producing the mean of the likelihood distribution: μ(z) dec(z; 0).
Likelihood variance. There are several approaches for handling the variance of the likelihood:
Fixed variance: Setting constant variance for all dimensions (this is bad practice: (159). Input-dependent variance: Using separate neural net to predict variance as a function of Learned global variance: Learning set of global variance parameters o² for the entire dataset [69
In our implementation, we use learned global variance parameters for the decoder, meaning:
p=(x))) = =N(x;; d))
31

---

Training, A critical component enabling end-to-end training of VAEs is the reparameterizattio trick [18). This technique allows gradients to flow through the sampling operation by expressing a
sample from (=(=))) as deterministic function of x: and a noise variable e ~ (0, I):
x = {x) + (x) @
With this formulation, both encoder and decoder parameters can be jointly optimized by maximizing the ELBO objective shown in eq. 12
numerous proposals have extended or modified the standard Gaussian framework. These efforts can be broadly categorized into three main directions: (i) developing more expressive or learnable priors, (ii) replacing the likelihood function with non- Gaussaan alternatives, and (iii) altering the latent distribution.
Extensions and modifications. Since the advent of VAEs 18 19,
In terms of priors Pa (z), researchers have introduced hierarchical variants [115 160
161]. structured
priors such as VampPrior [162), and nonparametric approaches like the stick- breaking process (163)
For the likelihood function po (arl z), alternatives to the standard Gaussian have been proposed better accommodate binary, count, or highly structured data. Examples include the Bernoulli distribution for binary data [18] 164). Poisson [165] and negative binomial [166) for count data, and mixtures of discretized Logistic distributions for natural images [161] 167]
Finally, for inference, many have enhanced the expressiveness of the variational posterior qu (z(2) (z) by applying normalizing flows [168] or inverse autoregressive flows [169], thereby relaxing the mean-field assumption. Others have replaced the Gaussian latents altogether, exploring alternatiee
distributions such as categorical [170 171) Bernoulli [172 173]. Laplace (174), Dirichlet [175),
hyperbolic normal 176),, von Mises-Fisher 177], Student's 178 and negative binomial 179
A.7 Poisson Variational Autoencoder (P-VAE)
particularly relevant departure from Gaussian latent variables the Poisson VAE (P-VAE; (50). Motivated by findings in neuroscience that neural spike counts follow Poisson statistics over short temporal windows [51 180], Vafaii al. (50) introduced a VAE variant in which posterior inference is performed over discrete spike counts. In P- P-VAE, both the prior and approximate posterior distributions of the standard Gaussian VAE are replaced with Poisson distributions. The authors also developed a novel Poisson reparameterization trick and derived corresponding ELBO for -oisson-distributee VAEs.
An important theoretical insight arises from the P-VAE ELBO: the KL divergence term induces firing rate penalty that resembles sparse coding [54). fundamental and widely documented property of biological neural representations [15T]. When trained on natural image patches, P-VAE learns sparse latent representations with Gabor-like basis vectors [107 110), much like classic sparse coding [54.
P. P-VAE has also been extended to model psychophysica response times [181].
Despite these advances, P-VAE underperformed the locally competitive algorithm (LCCA; 39
classic iterative sparse coding algorithm from 2008-in both reconstruction accuracy and sparsity. The authors attributed this performance gap key difference in inference mechanisms: VAEs rely on amortized inference; whereas, sparse coding uses iterative optimization at test time.
A.8 Amortized versus iterative variational inference
Once we select our approximate posterior distribution--ucc as a Gaussian (G-VAE) or Poisson (P-VAE)-we must decide how to parameterize and optimize it. Let X denote the distributional parameters of the approximate posterior (distinct from neural network weights). For example, Gaussian posterior requires both mean jl and variance o², i.c., A (μi, o²); whereas, a Poisson posterior is specified by vector of rates, À We now describe three strategies for optimizing A: amortized, iterative, and hybrid derative-amortizer inference.
Amortized inference. In amortized variational inference, an encoder neural network enc(æ; Φ). parameterized by weights Φ maps each observation æ to its corresponding posterior parameters in a single forward pass: A(x)= enc(æ; æ;; Φ). This approach, central to VAEs, is efficient at test time, as the cost of inference is "amortized" over training, much like spreading out fixed expense over time in accounting (36).
32

---

The amortized approach is partially motivated by the universal approximation capability of neural networks [122: if there exists a mapping from samples their optimal posterior parameters. + X" (a)), then sufficiently large and deep neural network should theoretically be able to learn it.
However, despite this theoretical promise, amortized inference often suffers from amortization gap-a systematic mismatch between the inferred posterior parameters, A(x) = enc(2; ø), and the optimal variational parameters, A (az). This gap can significantly degrade model performance (183
184] and can be conceptualized in two equivalent ways.
First, within given family of distributions, the optimal variational parameters X (xx) are by definition those that minimize the KL divergence from the approximate posterior to the true posterior (the last term in eq. ① Second, due the equality in eq. I this KL divergence is mathematically equivalent to the difference between the ELBQ and the model evidence, log pa (xz). The closer the variational approximation is to the true posterior distribution in terms of KI., the smaller the gap between BLBO and log Po (x:) becomes, thereby increasing the "tightness" of the lower bound.
In summary, the amortization gap reveals that despite the promise of the universal approximation theorem [82], even highly parameterized deep neural networks often fail to find the optimal varia- tional parameters [83). This results in both poor inference quality, well as substantial discrepancy between the ELBO and the model evidence.
Iterative inference. In contrast to amortized inference, iterative variational inference optimizes the posterior parameters À separately for each data point by directly maximizing the ELBO, typically via gradient-based optimization. Although more computationally intensive at test time, this approach often achieves tighter variational bounds and more accurate posterior approximations [183 185).
canonical example is stochastic variational inference (SVI; (186) ) which updates local variational parameters via stochastic gradient ascent. Iterative optimization is also central to classical Bayesian approaches such as expectation maximization (EM; [13 [87)) ), coordinate ascent variational inference
[188), meen-fielld variational inference (17, 189]. and expectation propagation [190), all of which refnn posterior estimates through successive updates.
In neuroseience-inspired models, iterative inference emerges naturally in biologically plausible frameworks such the locally competitive algorithm (LCA; (39 for sparse coding (SC), and in predictive coding models (PC; [22 23 Both SC and PC implement inference through local, recurrent dynamics. While these models
always formulated explicitly in terms of ELBO maximization, as reviewed in the main text, appendix A.9 and illustrated in Fig. 2 both SC and PC can be interpreted as performing variational inference via iterative refinement of latent variables in response to sensory input.
In summary, iterative inference methods can substantially reduce the amortization gap by producing more accurate posterior approximations. But they incur increased computational cost at test time, often requiring multiple optimization steps per observation.
Hybrid iterative-moortized inference. Several hybrid approaches have been proposed to bridge the gap between amortized and iterative inference, aiming to combine the speed and scalability of amortized methods with the accuracy and flexibility of iterative refinement. The Helmholtz machine [4] is an early example, using a recognition network for approximate inference (similar to amortization) but trained through the wake-sleep algorithm. This approach alternates between updating the generative model in the "wake" phase and the recognition model in the "sleep" phase, combining aspects of both amortized and iterative approaches in EM-like fashion.
More recent examples include iterative amortized inference (1], which trains a neural network to predict parameter updates rather than the posterior parameters themselves. This approach draws inspiration from meta-learning [19T], where the inference procedure is learned as a dynamical process conditioned on data. By learning to iteratively improve posterior estimates, such models can adapt inference trajectories to individual data points while still leveraging amortization during training.
Another example is the semi-amonn amortized variational autoencoder (92), which uses standard encoder network to produce an initial guess for the approximate posterior, and then performs small number of gradient-based based updates, as in stochastic variational inference (SVI; (186)), to refine the estimate. This two-stage approach retains the test-time efficiency of amortization while reducing the amortization gap via local adaptation.
33

---

Our models are fully iterative, in the spirit of LCA, but remain conceptually related to these hybrid strategies. We compare them thoroughly in our extended results below.
Iterative versus amortized inference: which approach is more brain-like? Unlike one-shot
amortized inference, neural dynamics unfold over time (75), allowing for progressive refinement of perceptual representations [42] 44). This temporal unfolding is consistent with recurrent processing and resonance phenomena observed in neural circuits (40] 43 suggesting that the brain engages in form of iterative inference. This idea is reflected in biologically inspired algorithms such as sparse coding [39] and predictive coding (22 (3), where inference implemented through ongoing dynamical interactions that minimize error over time.
counterpoint comes from Gershman and Goodman (36] who argues that the brain may instead employ amortized inference, using fast, feedforward mappings (e.g.. learned neural networks) that approximate posterior beliefs without requiring iterative updates. This view emphasizes the efficiency of inference through learned function approximation, contrasting with the flexibility and adaptivity of iterative approaches.
However, this efficiency comes at the cost adaptability and biological realism. Given the strong evidence for recurrent, time-evolving neural dynamics, view iterative inference as the more plausible default. Though in practice, the brain may implement a hybrid of both strategies.
The jury is still out.
A.9 Predictive coding as variational inference
Having established the variational framework for perception, now demonstrate how the classic predictive coding model of Rao and Ballard [22] can also be understood as a form of variational inference with specific distributional choices. Here, focus single-layer predictive coding network (PCN) with a linear decoder. See Millidge et al. 33 for comprehensive review, covering more general cases including hierarchical PCNs.
The key insight is that predictive coding is mathematically identical to free energy minimization (or ELBO maximization), with three specific distributional choices: (1) factorized Gaussian prior, (2) factorized Gaussian likelihood, and (3) a Dirac-delta distribution for the approximate posterior, combined with iterative inference.
Consider images IR M and latent variables z E R K Starting from the free energy formulation in eq. [13). we substitute qa (2(x)) = 8(2 1)), where @ = are the encoding parameters. With this delta distribution, the entropy term vanishes, simplifying the free energy to:
F(x: 0 p)) Eg~ [- log po (o z)] =
log po (x. pt).
(14)
For the generative model, we adopt factorized Gaussians:
Gaussian prior: po (p( = N( Mo; I):
Gaussian likelihood: po (arl = N(x; 1,,,, I):
where 1s0 E R K are learnable prior parameters, and & E Mx is a linear decoder (i.e., the dictionary sparse coding). Therefore, the generative model parameters are (No, 4).
Substituting these Gaussians into eq. [1] yields:
F(a; 0, (1) log pa (a/ (4) log (14)
(15)
2 [x x2)² + (μ mo)3] .....
where the dots represent constant terms irrelevant optimization. The two squared terms capture prediction errors at different levels of the hierarchy.
In predictive coding, the neural dynamics emerge directly from gradient descent on F. Specifically, predictive coding assumes neural activity evolves according to ja - , F, which gives:
i
Top
140
(16)
feedlforward drive
lateral connections
term
34

---

Table Models considered in this paper. Variational autoencoders (VAEs), sparse coding (SC), and predictive coding networks (PCNs) are different instantiations of variational inference with specific "prescriptive" choices. Our theoretical synthesis explicitly identifies these critical choices-includdng distribution families for priors and approximate posteriors, as well as the inference optimization method (amortized versus iterative)-that define and distinguish each model. Iterative VAEs, SC, and PCNs optimize encoding parameters À through iteration, while amortized VAEs perform one-shot inference using neural networks with parameters Ø. The main paper (sections and Fig. 5 focuses on linear decoders (dictionary 4), with the general nonlinear decoder case explored in appendices B.6 and C.8 to
10 Figs. 6 and 13 to 17 and tables 4 and S For an illustration of this table as a model
tree, see Fig.
(+)): nonlinear activation function (e.g., relu), Nat. GD: natural gradient descent.
Model family
Posterior
Prior
Likelihood
Architecture
Model
Reference
9x(=(>)
pz(z)
pa (=]2)
Encoder
Decoder
iP-VAE
Poisson
Poisson
4 O dxp(z)
iterative
iG-VAE
Gaussian
Gaussian
Gaussian
Nat. GD
This paper
VAE
i9.- -VAE
Gaussian
Gaussian
P-VAE
Poisson
Poisson
dec( z;
Vafaii et 50
amortized
VAE
G-VAE
Gaussian
Gaussian
Gaussian
enc(2; の)
dec( z; 0)
Kingma and Walling
16
VAV
Gaussian
Gaussian
dec (p)=); 0)
Whittington
PCN
iPC
PC
delta
Dirac-
Gaussian
Gaussian
GD
8
Rao and Ballard
Salvatori al.
Sparse
LCA
Dirac- delta
Caplace
Gaussian
GD
Field
coding
Rozell et al.
A.10
Variational free energy minimization unifies machine learning and neuroscience
Throughout this extended background, we reviewed how variational free energy (F) minimization serves as unifying theoretical framework that connects scemingly disparate approaches across machine learning and theoretical neuroscience. This synthesis reveals that models as diverse as variational autoencoders (VAEs), predictive coding, and sparse coding all be derived from the same fundamental principle of minimization, differing only in their specific prescriptive choices.
Our synthesis emphasizes two critical dimensions of choice that determine model's characteristics:
1. Distributional choices: The selection of three distributions
approximate posterior: qx (2(2)))). prior: pa(z),
likelihood po(x))).
2. Inference optimization strategy:
The decision between
training a neural network to compute posterior parameters in a single forward pass, versus
amortized inference:
iterative inference: optimizing parameters through iterative (c.g., gradient-basedd refinement.
These choices define a rich taxonomy of models. illustrated in Fig. 2 For example, combining amortized inference with factorized Gaussian distributions yields the standard VAE (G-VAE; [18]). Replacing the Gaussian prior and posterior with Poisson distributions leads to the Poisson VAE (P-VAE; [50].
On the neuroscience side, models based on iterative inference with Dirac-delta posteriors and Gaussian likelihoods give rise to predictive coding (22) and sparse coding (54). Predictive coding typically assumes Gaussian prior, while sparse coding often uses Laplace prior to encourage sparsity (Fig. 2). See Friston (23) and Millidge et al. [33] for derivations of predictive coding as variational inference, and Olshausen [55 and Chapter 10 of Dayan and Abbott (56) for sparse coding.
35

---

Overall, this systematic mapping clarifies how models originally developed in separate domains and described with different terminology can be understood through a common theoretical lens.
Building on this synthesis, our contribution is introducing the FOND framework (Free energy Online caaurallgradden Dynamics), which derives brain-like inference dynamics as natural gradient descent on free energy with specific distributional and structural assumptions. Unlike post-hoc theoretical interpretations, FOND starts with principles and derives algorithms in top-down manner. We apply FOND to derive a new family of iterative VAE architectures: the iterative Poisson VAE (iP-VAE), the iterative Gaussian VAE (iG-VAE), and the iterative Gaussian-relu VAE (ig VAE)).
These models exemplify FOND's prescriptive power, while combining the strengths of two theoretical traditions: the probabilistic foundation of machine learning and the recurrent, adaptive processing characteristic of neuroscience models. By making our prescriptive choices explicit and theoretically grounded, FOND provides just synthesis of existing approaches, but principled framework for developing new algorithms that address limitations in both fields. See Table I for a summary of models considered this work.
B Extended Theory: Motivations, Derivations, and Discussions
In this appendix, we present additional motivations for our modeling choices, detailed derivations, and occasional commentary key outcomes, complementing the theoretical exposition in the main paper (sections 4).
We begin by clarifying what we mean by "prescriptive" using analogy from fundamental physics. then motivate the Poisson assumptions, which led us to the iP-VAE architecture. In the remainder of this appendix, we further develop the theoretical foundations and present generalizations of our framework beyond what covered the main paper, along with interpretive insights that clarify its connection to principles of neural computation.
B.1 What do we mean by "prescriptive"?
In physics, prescriptive theories derive dynamics from first principles rather than Rand-tuning to fit data. Notably, the Principle of Least Action [61] states that physical systems follow trajectories minimizing the Action (time integral of the Lagrangian). However, this principle alone is insufficient without addressing two fundamental questions:
Deriving dynamics from first principles
QQ1) Dynamical Objects: What are the mathematical objects of interest which exhibit the dynamics we seek to understand?
(Q2) Dynamical Equations: objects' time evolution?
What differential equations govern these
For instance, effective field theory 622 addresses these questions most elegantly, where symmetry principles alone resolve both questions (e.g, see Chapter 17 in Schwichtenberg (63)). Thus, in prescriptive theories, the arrow goes from principles - dynamics, and not the other way around.
While the clegant derivations of physics (where everything follows from symmetry alone [64] may not be fully achievable in the messy world of NeuroAI, can still apply this prescriptive philosophy.
In this work, we follow the same arrow arrow-from principles dynamics to derive brain-like inference dynamics from first principles. Specifically: (1) we choose real-valued neural membrane potentials as our dynamical objects (Fig. [). This because membrane potentials are the source of information
processing in the brain and they directly generate spiking activity (70) 84) which in turn causally drive perception and behavior [192]. and (2) we derive inference dynamics as natural gradient descent
on variational free energy with Poisson assumptions in online inference setting (Fig. Tp. FOND).
36

---

B.2
Why Poisson?
To fully specify the variational free energy objective, one must make concrete decisions about distributional assumptions. Our choice of Poisson distributions for both the approximate posterior and prior is motivated by both theoretical principles and empirical observations from neuroscience.
From a theoretical perspective, Poisson is natural choice for neural coding because spike counts are, by definition, discrete, non-negative integers. When implementing variational inference, distributional choices should reflect the natural constraints of the variables being modeled, and the Poisson distribution inherently respects these constraints for neural spike counts.
Empirically, the Poisson assumption aligns with extensive neurophysiological observations. Neurons exhibit variable responses across repeated presentations of identical stimuli, with spike-count variance proportional to the mean 53). For brief time windows, this proportionality approaches unity 195), approximately matching Poisson statistics. While longer windows and higher cortical areas often show super- super-Poisson
Poisson variability, this can be attributed to modulation of an underlying inhomogeneous Poisson process [180].
In other words, neurons can be modeled as conditionally Poisson, even not marginally so 196
The apparent precision observed in certain neuronal responses [197] is not inconsistent with Poisson processes. Modern models demonstrate that inhomogeneous Poisson rate codes can capture precise spike timing [198], and the high-precision examples often cited against rate coding are effectively captured by poisson-procest Generalized Linear Models [199].
Overall, the Poisson assumption offers mathematical convenience, and it is not too biologically implausible. Neurons are not literally Poisson all contexts, but this distributional choice provides reasonable approximation that respects the discrete nature of spike counts while capturing key statistical properties of neural activity. By grounding our theory in these biologically realistic assumptions, we derive dynamics that naturally reproduce important features of cortical computation.
B.3 Poisson Kullback-Leible (KL) divergence
We provide closed-form derivation of the KL divergence between two Poisson distributions. Recall that the Poisson distribution for single spike count variable z, with rate r E Ry given by:
Pois(z;;)
(17)
Now suppose p- Pois(2;;oo is the prior distribution for this single neuron, and g = Pois(2; r) is the approximate posterior. The KI. divergence is given by:
Dea (1llp) =
Bsng log
r e /2!
= Benu
log
r rje-r0/2!
=
Eznn
log
r
e (r-ra)
r
(18)
= Ezng
log
- (r - ro)
r
= Egng
log
r + TD
To
= log
- r + To
- -1)
= ro +
log
37

---

We now express the final result in terms of the membrane potentials, u = log to get:
DIL (Pois(2; exp u)) (ois(2;
e c 1).
(19)
This concludes our derivation of KL divergence for single Poisson variable. It's easy to see how this result would generalize to latents, which what we show in eq. (2.
B.4 Fisher information matrix
To derive inference dynamics natural gradient descent on free energy, we must account for the geometry of distributional parameter space. The Fisher information matrix defines this geometry by acting as Riemannian metric over the space of probability distributions, measuring how sensitively a distribution changes with its parameters.
When small parameter changes produce large shifts in the distribution, the Fisher information high, meaning those parameters are farther apart distribution space. Natural gradient descent leverages this structure, adjusting updates to reflect the true curvature of the space; whereas, standard gradient descent implicitly assumes flat, Euclidean geometry. These geometry-aware updates lead more
efficient and stable optimization, especially in ill-conditioned settings [35 57 76 200
In what follows, formalize this idea and derive explicit Fisher information matrices for Poisson and Gaussian distributions.
B.4.1 Fisher information matrix as a second-order approximation of the KL divergence
To derive the Fisher information matrix, we begin with local approximation of the KI. divergence between two nearby distributions. Consider distribution px(x) parameterized by A, and the same
distribution with slightly perturbed parameters, PA+ (z). The KL divergence between these distributions can be expanded using a second-order Taylor approximation:
Dac px PATAA) f Px(x) log
(x
de
PA+AA ANTG(AJAA O(IAA1)),
(20)
where G(A) is the Fisher information matrix defined as:
G(A) = (xxpx(x) V lo
(21)
This matrix appears as the Hessian in the second-order approximation of the KL divergence, revealing its role the natural metric for measuring distances between nearby probability distributions. From this perspective, natural gradient descent updates take the form:
A++1 = >; - G VA,,)
(22)
where L(A) the loss function (in our case, variational free energy), and r is the learning rate. By incorporating the inverse Fisher information matrix. the update accounts for the curvature of distribution space, leading to more effective optimization.
B.4.2 Poisson Fisher information matrix
In this paper, considered the canonical parameterization of the Poisson distribution in terms of log rate, u log r, interpreted as membrane potentials. We now derive the Fisher information matrix for single Poisson variable in this log rate parameterization, where the probability mass function is:
Pois(2;e") = eue
(23)
2!
The log-likelihood function of u is:
log Pois(z;e")
e" looo z!
(24)
38

---

e" Taking the derivative with respect to a:
log Pois(2;E") = 2- ca
(25)
Ou
Plug this expression in eq. (11 to get:
0
G(u)
(ewPois(x;e"))
log Pois(z;
Ou
(ewPols(zte")
(26)
(ewPois(zeeN)
2² 2ze e2u
22 2e E. Pempooissse + e
Ez=Pois(szen)
For a Poisson distribution with rate e", we know E[2] e" and Var[z] = e", which means E[2 Var[z] + B[2] e" e²" Substituting, get:
Guu) = (e" + e²2) - 2e e" + 20
= e" + 22 20 + n
(27)
The final result reveals remarkably simple form for the Fisher information matrix in the membrane potential parameterization:
Poisson:
G G(u)
e"
(28)
For the multivariate case with independent Poisson components, the Fisher information matrix becomes diagonal with entries (u) = e" or in vector notation, G(u) = diag(e").
The natural gradient update for the membrane potentials thus takes the form:
à GG
- ④ V.F.
(29)
This derivation leads directly the dynamic equations shown in eq. (6) providing principled foundation for our spiking neural inference model.
B.4.3 Gaussian Fisher information matrix
For the Gaussian distribution, we define the log standard deviation, & = log o, and derive the Fisher information matrix using the (H, parameterization. The probability density function is:
N(x; (1,655 = V2me2E exp (- (xr 11)
exp(-C
(30)
2c
And the log-likelihook is:
1 log log(2m)) - (x - -122
log NogN((; (2; -
(31)
2e
We compute the partial derivatives with respect the parameters:
log Nz:
0p
e2E
(32)
logNN(x: u4 25)
--14
+ e
p
0g
39

---

The Fisher information matrix components are:
27
Gyz (a
a
log (2: e25)
pp
(x u³²
et5
(33)
(x -1)2
e
e
e 25
And for the & component:
27
0
log (a; e (o))
Gęg
0g
22
-1
(34)
2
x -z 11 17 +
r 1
(x
1-2 2
e
e
For Gaussian, have E[(x-- - 11)²1 the expectations, we get the final result:
e and BB[( =) 3e After expansion and computing
GEe = 2.
(35)
Remarkably, when using the log standard deviation parameterization, the corresponding Fisher information metric becomes constant.
Finally, for the cross-terms have:
à log, (x;
log N(;;
(36)
I- (x==)² 2
(-1+
= -BxnN(rs
2
o
Since Eir- μ1] = [[(x 21) = 0 for Gaussian, this cross-term vanishes:
G = 0.
(37)
Therefore, the Fisher information matrix for univariate Gaussian with (14, {) parameterization is:
-2
Gaussian:
(w,
- 0
(38)
0
2
For the multivariate case with diagonal covariance matrix, the Fisher information becomes block- diagonal with entries corresponding to each dimension's parameters. This structure leads to separate natural gradient updates for means and log standard deviations.
Equation 338) shows that natural gradient descent Gaussian parameter space involves scaling the mean gradients by the variance, while the log standard deviation updates require simple constant scaling factor. In appendix B.7 we will use this result to derive the inference dynamics for the case of Gaussian posterior distributions.
40

---

B.5 Interpreting the terms in Poisson dynamics equations
In this section, interpret each term in the neural dynamics, eq. (6) We explain their biological significance and draw connections to canonical models in computational neuroscience.
B.5.1 Feedforward drive
The first term, 4 x, represents the feedforward excitatory input to neurons (i.e., "feedforward drive"). projects the incoming stimulus æ into the linear subspace spanned by each neuron's ' receptive field (the columns of 4). This operation loosely corresponds to the selective tuning of synaptic weights, with each neuron responding strongly to stimuli matching its preferred feature. In sensory cortices, for example, this feedforward drive could correspond to thalamocortical projections that relay sensory information to cortical neurons (201 202]
B.5.2 Recurrent explaining away
The second term, TT6 z(u), with spikes sampled from the posterior z(u) qu (z(a), captures recurrent interactions within the neural population. Crucially, this interaction occurs through spiking activity z(u), not membrane potentials, thereby preserving the biological principle that neurons communicate via discrete spikes rather than continuous voltages
[22
The recurrent connectivity matrix, TT allows lateral interactions among latent dimensions. This is known as "explaining away" in probabilistic inference contexts (7) or as "lateral competition" or "lateral inhibition" in neural circuit models [8 86
207).
Two distinct suppressive mechanisms emerge: (1) self-suppression via diagonal terms ([*"0] > 0). where neurons with high activity suppress their own future responses, acting as a form of spike- frequency adaptation [208); and (2) lateral inhibition through off-diagonal terms ([
0), where neurons with similar feature preferences compete 87). While most interactions are inhibitory when features are positively correlated [[074], >> 0). excitatory effects can also arise when features are anti-correlated ([o" < 0).
B.5.3 Homeostatic leak
The final term, ((u-wa), effectively functions as a homeostatic "leak" that pulls membrane potentials back toward their prior values. This establishes direct formal connection between fundamental biophysical property of neurons-tte leaky integrator dynamics [81]-am and the Hallback-Leibler divergence in variational inference, which measures the relative information (209) between posterior and prior distributions.
From biological perspective, this leak term reflects passive membrane properties that restore neural excitability to baseline in the absence of input, preventing hyperactivity and ensuring metabolic efficiency (100. From an iifommation-hhoreee standpoint, it implements regularization pressure that penalizes deviations from the prior, balancing the reconstruction objective against representational complexity 777 78
The strength of our framework lies in unifying these perspectives: the biophysical constraint of "don't 't deviate too much from your baseline state" aligns nicely with the cntrmatiion-hhoreed principle that "encoding more ciimulus-spcciff information incurs a coding cost." " This duality offers a principled explanation for why neural systems operate in sparse regimes: they naturally minimize free energy by balancing reconstruction fidelity against metabolic constraints (50).
Overall, the derived iP-VAE dynamics, eq. (6). reveals how neural circuits can implement Bayesian inference through local computations and spiking communication, in way that mimics established principles of neural physiology. The emergence of these biologically plausible mechanisms directly from variational principles suggests that evolution may have indeed optimized neural systems to perform approximate Bayesian inference
66
74 148
41

---

B.6 Extension to nonlinear decoders
In the main derivation (section 4) we made two simplifying assumptions enhance clarity: (1) we assumed a linear decoder, and (2) we ignored the variance of the decoder. In this section, we relax these assumptions and provide full derivation that reduces those in the main text.
Throughout this work, we assume a Gaussian likelihood function for all models considered. The mean is produced by decoder neural network, f=(z) dec(2; () fo(z), with parameters 0. For the main derivation, we assumed a unit covariance. But in fact, the code we have a set of global learnable parameters, o², that we learn end-to-end alongside other parameters [69]. Given these assumptions, the likelihood is given by:
p=(zz) N(x; fo(z),
And the negative log-likelihood is:
1 fo(z)
2
where the dots represent terms that are going to vanish after differentiating with respect to the variational parameters, so we didn't bother writing them out.
Similar to the main text, we approximate the reconstruction term using a single Monte Carlo sample:
x
f=()):
Creoon. (x; A)
where z(A) (x(2(2)),
æ). We see that to compute the gradient of the reconstruction term, we only need to compute the gradient of the sample and simply use chain rule 0/0>; = (0))))0))00)),
Oz
M
IJ
[fol,
Lroonn (x; 02
Jolis
ax;
ax
Jj
where we defined the decoder Jacobian as follows:
2 [f)
(39)
Oz.
In matrix form, the gradient of the reconstruction term reads:
0z OX o
x - 10(2)] 02
V Crocon 0, =
[z(z)
(40)
where 0z/0A is a vector of length K, the decoder Jacobian Ja is a K x M matrix, the expression inside the bracket is a vector of length M, and is the Hadamard (element wise) product. During inference, we evaluate eq. (40) at the particular sample drawn from the posterior, z =A)).
This concludes our extension of the results the case of nonlinear decoders with learned global variance. If the decoder is neural network with relu activations, then the Jacobian can be interpreted dynamic feedforward receptive field. In the of linear decoder, fz( z, get fixed
Jacobian: Jo DT, producing the main paper result: V frecon. x ( x + 02).
Finally, compare Fig. and Fig. for a ResNet-style view of the unrolled encoding algorithms, contrasting linear and nonlinear decoders.
B.7 Extension to Gaussian posteriors
In the main text, derived dynamics for the case of Poisson latent variables, resulting in the spiking inference dynamics shown eq. (6). This led to the corresponding iterative Poisson VAE (iP-VAE) architecture.
Here, we extend these results to Gaussian latents, with an optional nonlinearity, sp.)), applied after sampling, corresponding the ig- VAE and iG.- -VAB architectures, respectively.
42

---

t: lager
b
indes
Amorlized Encoder (a)
Mp41
Our Encoding Algorithm (unrolled in time)
414-1
Deterministic Me++11 + fo. (M.)
Standard ResNet
Φt
inceperrbet nnt parsesrtive per agart (o+)
fΦr
fo
Our Iterative Encoder (b)
Adaplive
1 1eg + folue, x)
Φ
Stochastic
Is(me, - 6- 0
Spiking
24
480 11
x
Ug
single not of sharec parumeters (dacoder-enr/ 0 @
Figure 5: Comparison between standard deep residual neural networks (ResNet; (211) and our inference algorithm. (a) Standard ResNets process information through a series of deterministic layers, each with independent parameters ét, where i is the layer index. (b) The iP-VAE encoding algorithm (eq. (7)) unrolled in time resembles ResNet with shared parameters, but incorporates adaptive, stochastic, and spiking dynamics. This recurrent architecture is conceptually similar to Looped Transformers 211) which use "input injection" injection"cooniiiiiie conditioning on the input æ at each step-as an effective heuristic to improve stability and performance. In our framework, this mechanism not a heuristic choice, but emerges directly from the first principles of online variational inference. Finally, unlike conventional networks [95). the iP-VAE uses only decoder parameters 6, enabling online Bayesian inference through iterative sampling and recurrent updates conditioned on both current state u, and input This figure illustrates the linear decoder implementation used in the main paper. For the general nonlinear decoder case, see Fig. 6
ui+1
fo
U++1 = fo((uz x))
solu,, x)) Jol Jo(ze] A.
MatMul
A.
JØ
=
x
[M[=]]
8 [deu(z; 0)} 0z.
Decoder Jacobian
dec(2;0 0)
A = x dec(2z; 0)
Prediction Errors
~ Pois(z;rr)
Integer Spike Counts
exp
Zt
Ti ex=
Instantaneous Firing Rates
Ut
Figure 6: Detailed view of the VAE iterative inference algorithm with a nonlinear decoder. The left panel illustrates the computational flow: membrane potentials (ut) are transformed into firing rates (Tr) through an exponential nonlinearity, then sampled to produce integer spike counts (zt). These spikes passed through decoder to generate predictions, with eπrors propagated back through the decoder Jacobian. The right panel shows the corresponding mathematical formulation derived in appendix B.6 For the simpler case of linear decoders used in the main paper, Fig. 5
Let's revisit the prescriptive choices needed for deriving dynamics from first principles (appendix B.1 For the Gaussian case, we select the mean, μ. and log standard deviation, & == log o, dynamical variables. This parameterization avoids constrained optimization that would be necessary with o directly, since must remain strictly positive while & can take any real value. Additionally, as shown in appendix B.4.3] the log standard deviation parameterization yields a constant Fisher scaling, which simplifies our derivations.
Next, we derive Gaussian inference dynamics from natural gradient on free energy. Since we use Gaussian likelihoods across all models in this paper, we' ve already addressed this component in the main text for linear decoders (eq. (4) and extended it to general nonlinear decoders in the appendix (eq. 44h). We now need to derive the KL component.
43

---

B.7.1 Gaussian KL divergence
Suppose the posterior and prior parameters for a single latent variable are given by (H, o²) and (Ho, o)), respectively. The Gaussian KL term reads:
(M= 140
+ ( (00)² ) log (00
DeL qn. (z|ax) (poo.o(2)) = 2
-
0
With our log standard deviation parameterization, Ç = log o, this becomes:
1 14--100 140 2
DeL qu.s (2 Pro. (z) = 2
+ 215- - 25 Eo) -
eco
B.7.2 Gradient the Gaussian KI. term
For the general factorized K-variate Gaussian, we compute derivatives with respect to the variational parameters to find:
22co V VeDx(allp)) = 2 (((E-Eo)
(41)
1.
The next step is to estimate the reconstruction gradient from eq. 440). This requires computing the partial derivative of the sample with respect to each parameter. Using the Gaussian reparameterization trick 20)). we have:
r + exp(E) Oc,
(42)
with 6 ~ (O, 1). Therefore, the sample derivatives are:
Oz
dz
1,
exp(E) O e.
(43)
0μ
B.7.3 The Gaussian inference dynamics
Following our prescriptive approach:
Á = -
(44)
VV Lreon. + v. LKI.]. (A)
Combining eqs.
38
(40).
and
yields the final dynamics for the Gaussian case:
# 26
[o(z))
x
fo(z)] 0²
2 (μ Ho)
(45)
1 c O
Jo(2)
3 (c2tE-E0) -1)
Remember that we had z = + exp(E) @ e, and 6 ~ ((,
This completes our derivation of the full Gaussian inference dynamics. Next, we examine how this general equation simplifies for linear decoders and when applying nonlincaritics after sampling.
B.7.4 The iterative Gaussian VAE (iG-VAE)
In the main paper, we focus on linear decoders, defining the iterative Gaussian VAE (iG-VAE) model.
For linear decoder, we have fo(z) o and Jo TT. Substituting these into eq. 45 gives us the VAE dynamics:
T x
T +z
i
28
3 2 EE- c (H))
x²
(46)
T x
TT
1
(22(6-E00) - -1)
& =
O
02
2
44

---

B.7.5 The iterative G VAE (ig.-VAA VAE)
We can further extend our framework by applying nonlinearity, (p-), immediately after sampling:
N = p (z) = yp ( ( + exp(E) )).
(47)
This modification affects the dynamics equations in two ways. First, we decode from 2 (z() rather than z directly: fz(z) + fz()) In the linear decoder case, this simplifies to: dz + @xp( (z) (see [. Second, we now have to apply the clement-wise product of the derivative of the nonlinearity, evaluated the sample point. This stems from the sample derivative term from eq. (40).
With these two changes applied, the ig VAE dynamics becomes:
fol((()))
i
28 O (z) o
Ja(((z))
2(E-EE) o (μ - 140), x - (e-fa(o(a))) o²
o²
(48)
Jo(((()) $ (x) O
3 (c2le-Eo) -
This approach provides general recipe that works with any differentiable function. In this paper, we specifically use (p.)) = relu(-) for two key reasons. First, Whittington et [89] demonstrated that applying relu to amortized Gaussian VAEs, combined with activity penalties, produces disentangled representations that can even outperform beta-VAEs (78). Second, Bricken et al. (215) showed that combining relu activation with noisy inputs is sufficient to approximate sparse coding.
In our implementation, we use the exact derivative of the relu function:
relu' (z) = ((2),
where ((-) is the Heaviside step funetion.
Finally, the igretu VAE dynamics under a linear decoder are given by:
Th x
T relu(z)]
j = 2 о O(2)
B 2(E-E0) o (μ- puo),
o²
(49)
? 2 e @c00(z)
O
T relu(z) T x
3 (e
-6) (2266-E0) 1).
B.8 Biological interpretation of the standard predictive coding dynamics
Due to Gaussian assumptions, the predictive coding inference dynamics (eq. (16), appendix A.9 reveal a rate model where the variables ls are real-valued and only loosely interpretable as neural activity. In contrast, biological neurons communicate through discrete spikes, not continuous values. One possible interpretation is that μ represents membrane potentials resulting from the summation of many stochastic synaptic inputs. By the central limit theorem, these could plausibly follow Gaussian distribution.
Additionally, the dynamics include recurrent term of the form TT μl, which can be interpreted as each neuron being directly influenced by the membrane potentials of other neurons, which is unrealistic. Membrane potentials of real neurons typically thought to be internal to each neuron and not accessible to others. Communication between neurons occurs primarily through spikes (84). One solution to this is to stay in the space of rate models, but apply nonlinearity to map from the membrane potentials to rates. In canonical circuit models -tartis-starting with Amari (80) (1972), and developed further in biologically inspired frameworks 83) it standard apply nonlincarities to neuronal activity before it influences other neurons, thereby maintaining biological plausibility.
While some predictive coding formulations incorporate nonlinearities, they typically apply them after the decoder weights [33]: du + sp( (.)).. This yields recurrent terms like T ( which are still at odds with principles of local computation and communication. A more biologically plausible formulation would instead apply nonlinearities before lateral communication: & (pp)).
Our framework addresses these limitations in two distinct ways. In the Poisson case, communication occurs through sampled spike counts, so each neuron only transmits spikes to its neighbors, preserving the privacy of membrane potentials (eq. (6). In the Gaussian case, apply a nonlincarity such as
45

---

relu after sampling from the posterior, allowing only rectified activity to propagate laterally (eq. (49)) Unlike predictive coding dynamics (eq. (I6), both of these formulations, membrane potentals remain local to each neuron, in line with biological constraints.
B.9 Free energy for sequences
Consider a sequence of input data, { where ot E RR- (e.g., frames of a video). For such sequential data, we must account for the temporal dependencies our inference process. After observing the first t frames, we form a prior belief:
po(2(0e1),
(50)
where de = ( $1,-.. 1 represents all previously observed frames. Upon receiving the next observation we update our beliefs by computing the posterior distribution (()))u).
This posterior update can be formulated through the minimization local-in-time free energy:
Fe(u)
(x po
DxL [=( po (zr
(51)
reconstraction term
KT. divergence from prior
This formulation captures the essence of Bayesian belief updating: the posterior balances fidelity to the current observation (reconstruction term) against deviation from prior beliefs shaped by all the previous observations (KL term).
Applying natural gradient descent this time-dependent free energy, derive the general update equation for sequential inference:
U6+1
U
TTT
Zt.
(52)
where 2, ~ ((2;/ This equation reflects a rolling update scheme in which the posterior at time serves the prior at time t 1. We examine this update rule in more detail in the next section.
For stationary sequences (i.e., repeated presentations of the same image, æt + x), this general update equation reduces to the simplified form presented in the main text (eq. (7). In this paper, we focus exclusively on such stationary sequences, leaving richer non-stationary dynamics for future work. Future extensions could incorporate more sophisticated predictive dynamics beyond the simple rolling update scheme, potentially allowing the model to anticipate future frames rather than just adapt current ones.
The overall free energy for the entire sequence is the sum of local-in time free energies:
T
Fe
(53)
During model training (i.c., learning the generative model parameters 0), we perform iterative inference updates, computing F, at each time point while accumulating gradients. After processing the entire sequence, a single gradient update is then applied minimize the overall F, similar to meta-learning approaches [191].
This sequential formulation reveals how encoding algorithms can be interpreted as unrolled residual networks (ResNets), where each time step corresponds to network layer with shared parameters (Figs. 6).
B.10 Static versus online inference
Traditional variational inference operates on static datasets, assuming samples are drawn indepen- dently from fixed distribution. In sharp contrast, biological perception involves continuous belief updating from streaming observations. This fundamental difference motivates our preference for online inference approaches with evolving prior.
This section contrasts static and online approaches to inference and their mathematical implications. A key distinction is that online inference features evolving prior, opposed to fixed prior in
46

---

a
t+1
b
t+1
KL(t +1)
+
t
t
KL(E)
RELO
XL(1
1)
t
t-1
t= 0
t ==0 0
Static iterative inference: fixed prior, multiple gradient updates
Online inference:
evolving prior, single gradient step per time point
Figure 7: Comparison of static and online inference frameworks. (a) In static iterative inference, all posterior distributions (points along the red curve) are regularized toward fixed prior at t O. Each blue line represents KL divergence (b) In online inference, the prior evolves over time, with each posterior becoming the prior for the next time step. KL divergence is computed only between adjacent time points (consecutive points on the red curve), representing rolling update scheme. This fundamentally different regularization structure leads the vanishing KI. term phenomenon when performing single gradient updates at each time step (e.g., compare eq. with
computed relative to this same unchanging reference point.
is always computed relative to the previous time step (Fig-7p). We show how this observation explains why single gradient update in online settings naturally leads to dropping the KL contribution to free energy, providing intuition for why BONG (60) advocates using only the expected log- oogliikelhood term in gradient updates.
static inference. This evolving prior means the KI. term (eq. 12
In static inference (Fig. (a), the time index t reflects gradient iterations, with each posterior being regularized toward the same fixed prior. As optimization progresses, the KL term consistently pulls the distribution back toward this initial prior point, preventing excessive drift regardless of how many iterations are performed.
In online inference (Fig. 7D), the time index represents sequential data points rather than optimization iterations. The crucial difference is that prior beliefs themselves get updated as new data arrives. When we follow the BONG approach [60) and perform single gradient update per time step, we only compute the KL divergence between adjacent time points, rather than regularizing all the way back to an initial prior.
Why does the KL regularization term effectively vanish in our discrete-ting online setting with single gradient updates? To understand this, we revisit the continuous differential equation from eq. 6
& TTx
T z z(u) 3(u uo)
When we discretize this equation for a single update step, we get the following general update rule:
Ue+ = Vimi + [ TH 2 (um)) B Uni up)]
where 5t is small time interval. We choose initialize the prior from the previous time point,
Vinit < ug. Consequently, the third term, B(umii uo), vanishes for this first gradient step:
Ut+5t = u0 + öt [ x o To =(uo)
uo]]
Because of the rolling update scheme, know the prior at time t is the same as the posterior from time - 1, denoted by u4- In other words, we have up = u4 resulting in:
Uc+51 = us + öt ih x To xwu))
Assuming 51 1, we recover the update rule shown in the main text (eq. 77. This explains why the leak term is absent eq. (7). also explains why BONG (60) advocates dropping the KI. term entirely when performing single-step online inference: has no effect on the first gradient step.
47

---

More generally, this analysis reveals fundamental difference between static and online inference paradigms: static approaches maintain stability through persistent regularization to a fixed prior; whereas, online methods continually update the reference point for regularization and are more adaptive as a result. This distinction particularly relevant for modeling biological perception, which must balance the preservation of accumulated knowledge with flexibility to new observations.
C Extended Experiments: Methodological Details and Additional Results
This appendix complements the main experimental results from section S
We provide detailed
information about model architectures, hyperparameters, and training protocols in appendix C.1 along with additional methods, analyses, and findings as we summarize below.
We describe our convergence detection methodology (appendix exhibits cortex-like properties such as contrast-dependent response latency (appendix C.3 extend our reconstruction-saan sparssty analysis with MAP decoding experiments (appendix CC.4. present MNIST results comparing iterative VAEs with predictive coding networks (appendix (C.6), and provide evidence for iP-VAE's strong out-of-distribution (OOD) generalization capabilities for both within- dataset perturbations and cross-dataset settings (appendix C.8)
demonstrate that iP-VAE
C.1 Architecture, Hyperparameter, and Training details
This appendix describes the implementation details of the iterative VAE models introduced in this paper. We cover architecture design, hyperparameter selection, and training procedures. We also provide details about all the other models we compare to, including amortized VAFs, predictive coding variants (PC, iPC), and I.A..
C.1.1 Architecture details
Our iterative VAE architectures consist of three key components:
Encoding mechanism. The encoding algorithm implements natural gradient descent on variational
free energy (eq. (7)). visualized in Figs. 5 and 6 Different distributional choices yield distinet
dynamics: Poisson encoding (eq. (7)) for iP- VAE versus Gaussian encoding (eq. (46)) for ig- VAE.
The encoder is fundamentally coupled to the decoder, performing mmalysis-by-syntheiss (68] We
derive results for linear decoders in the main paper and extend to nonlinear decoders in appendix
B.6
Key implementation details include: (1) learned priors (one parameter per latent dimension). and (2) a learned global step size, öt, interpretable as a time constant: Ut+1 = U + ôt ún with given by eq. (6) These parameters are learned end-to-end alongside decoder parameters 0.
Sampling mechanism. To approximate the reconstruction loss (eqs. and 40 ), we use differ- entiable sampling from the posterior distribution. For Gaussian models, we employ the standard reparameterization trick (18) (eq. (42) while for iP- we the Poisson reparameterization algorithm [50]. In - VAE models, we apply a nonlinearity gp.) post-sampling, which effectively modifies the latent distribution. For example, applying relu to Gaussian samples creates truncated Gaussian distribution, corresponding to the igrelu VAE architecture.
Decoding mechanism. Our main results use linear decoders (1) that map from latent to stimulus
space. This enables closed-form theoretical derivations (eqs. (6) (46) and (49)). ). In appendix B.6
extend our theory to cover nonlinear decoders; and, in appendix C.8 we extend our empirical analysis
to MLP and convolutional decoders. Throughout this paper, all models use Gaussian likelihoods
(yielding MSE reconstruction loss) with learned global variance parameters [69]
Implementation details for other models. For amortized VAEs, we utilized the implementation from 50]. including their lave-layer ResNet encoders. For LCA, we used the LCA-PyTorch library 216 (BSD-3 license). For PC and iPC models, we employed the PCX library [113) (Apache- 2.0 license) with their default MNIST-optimized MI.P architectures. But we matched the latent dimensionality to the other models, using consistent value of K 512 throughout.
48

---

C.1.2 Hyperparameter details
For the reconstruction-spariity analysis (Fig- we swept across Tainn E {8. 16, 32} and 3 values proportional to each Train with factors E (0.5, 0.75, 1.0, 1.25, 1.5, 2.0, 3.0, 4.0). This can be seen in the x- x-axis of Fig. I0 For amortized models, we used the same proportional values but with
Twain 1. For MNIST experiments, used 3 Tirain-
For the convergence experiment (Fig. B) we used Tiran = 16 with 24.0 for iP- and / 8.0 for ig-VAE and - based on optimal values identified in our hyperparameter sweep (Fig. 4 The contrast experiments (Fig. g used the same ip- configuration.
For LCA, we used a time constant T = 100 with 1000 inference iterations and learning rate of 0.01. We optimized the sparsity parameter through extensive sweep on the van Hateren dataset, finding À = 0.5 performed best.
C.1.3 Training details
We trained iterative VAEs using PyTorch with the Adamax optimizer [217 and cosine learning rate scheduling [218), but without warm restarts. We implemented KL term annealing during the first 10% of training following standard VAE practices [219] 220]. though we observed minimal impact since our models use single layer.
For ip- IP-VAE, adopted the Poisson reparameterization algorithm from (50). annealing the tempera- ture from temp 1.0 to tempstop 0.01 during the first half of training.
For the Hateren dataset, used learning rate of 0.002, batch size of 200, and trained for 300 epochs. MNIST [22T] models used identical hyperparameters but were trained for 400 epochs. These settings were consistent across both iterative and amortized VAEs.
For LCA, we adapted the training approach from (50, gradually increasing from 0 to the target value over 100 epochs. For PC and iPC models, we followed the PCX library examples [113] doubling the default epoch count to ensure convergence.
C.1.4 Training compute details
The computational requirements for iterative VAEs scale with the number of inference iterations (Tuan). For example, training the models presented in Fig. 3 with Train = 16 requires approximately three hours per model approximately linearly with T Timin as each additional inference iteration requires maintaining the computational graph for backpropagation.
NVIDIA A6000 GPU. Memory usage and training time both scale
Our complete set of experiments, including all hyperparameter sweeps and model variants, was conducted over the course of approximately one week using six NVIDIA A6000 GPUs (48GB VRAM each). To maximize GPU utilization, we often ran multiple smaller experiments concurrently on single GPU. Linear decoder models were the most memory-efficien nonlinear decoders for the OOD generalization tasks required more resources due to their additional parameters and the need to backpropagate through relatively more complex architectures.
while experiments with
C.2 Convergence detection procedure
In section 5 and Fig. 3
we determined convergence by finding the first time point where the R² trace flattens out and stays flat. Here, provide more details about this convergence detection algorithm.
The main intuition. We consider inference to be complete once the Rt² trace becomes essentially flat. This is because R² is monotonically related to the log-likel,hood and when the curve levels off, further improvements in log- likelihoodd are negligible.
To obtain a robust estimate the local trend, we fit a straight line every 60-step segment (a sliding window) of the trace and measure its slope. We call window flut when the absolute slope falls learning, we require at least five
below a fixed tolerance of 10-5, To protect against short pauses in successive windows to be flat before declaring convergence.
49

---

50
contrast
contrast 0.5
40
contrast 0.25
Spike Counts
Example neuron
contrast 0.15
30
20
-1
High contrast
contrast 0.25)
10
0
0.6
0.7
0.8
0.9
1.0
1.1
Relative Time [cycles]
Figure 8: iP-VAE neurons exhibit contrast-dependent response latency, mimicking well- documented property of V1 neurons. The figure shows trial-averaged firing rates (PSTHs) from representative model neuron in response to drifting gratings of varying contrast levels. Higher contrast stimuli (blue and orange) elicit responses with shorter onset latencies, earlier peak times, and larger firing rates compared lower contrast stimuli (green and red). Insets show the learned feature of the selected neuron (top) and the appearance of high contrast (left) versus low contrast (right) grating stimuli. The x-axis represents time in units of grating cycles, while the y-axis shows mean spike counts across N = 500 trials. Shaded regions indicate standard deviation. This hehavior likely emerges from the divisive normalization in our model dynamics (eq. (), even though the model was trained only on static images, suggesting that our theoretical framework captures key aspects of visual cortical computation
Details the convergence detection algorithm. Given & = (1.....,8777 ST,)): a one-dimensional array of R² scores in [0, 1]. let window 60, tol = 10-3 and consecutive 5.
1. Compute local slopes. For each start index O,... Tist window, define
T (0, 1,..., window 1), y (Si 8i+windog 1),
and compute the least-squares slope
B. E,(T, - - 9) Store |3:| in an array slopes.
Eg(T, F)2
2. Mark flat windows. Set flat (slopes < tol).
3. Require persistence. Find the smallest index & at which flat is true for consecutive=5 windows row. no such run exists, define teonverge = Test-
4. Report the convergence index. Otherwise return
(converee k + window.
the first sample that follows the verified flat period (at least 5 x 60 300 uninterrupted steps).
Please code for more info.
50

---

C.3 iP-VAE exhibits cortex-like dynamics: contrast-dependent response latency
The ip-VAD learns spatially localized, oriented, and bandpass (i.e., Gabor-like [077 110) features (Fig. I.. similar to those learned by classic sparse coding models (54) and LCA (39]. Previous studies have shown that LCA exhibits cortex-like properties, including contrast invariance of orientation tuning. cross-orientation suppression, and on (222). Given the theoretical (section and empirical (Fig. 4) similarities between iP-VAE and LCA, we investigated whether ip- (P-VAE also exhibits cortex- like dynamics. We focused on two key neural phenomena with strong experimental support: temporal sparsification dynamics and contrast-dependent response latency.
V1-like sparsification dynamics. Recent electrophysiological findings in mouse primary visual cortex (VI) by Moosavi et al. (T06) demonstrate that V1 neurons become progressively sparser over time, even after initial response convergence, while maintaining high mutual information with the input stimuli. Remarkably, both iP-VAE and iGretu VAE exhibit analogous dynamics in our experiments, continuing to sparsify their representations by an additional ~10% after functional convergence as measured by reconstruction quality (Fig. B). For iP-VAE this post-convergence sparsification coincides with distinctive dip in the gradient norm || (Fig- 3 suggesting a transition to phase of representational refinement. The emergence of this biological property across both
models provides evidence that our normative framework captures fundamental principles of neural computation, without being explicitly designed to do so.
V1-like contrast-dependers response latency. A significant theoretical outcome of our approach that multiplicatiie form of divisive normalization naturally emerges from the derivations (eq. (8). The resulting equation differs from canonical normalization that was developed to describe cortical circuits in that there is product instead of sum [223). However, in practice, captures features of normalization in the cortex. In particular, Carandini et al. (111) demonstrated that V1 neurons exhibit contrast-dependent response latency when presented with drifting gratings of varying contrast. Higher contrast stimuli consistently elicit faster responses, measured by both onset time and time-to-peak
To investigate whether our model reproduces this phenomenon, we conducted an analogous ex- periment on the spiking "neurons" of the iP-VAE We first characterized the tuning properties of individual model neurons to identify their preferred orientation and spatial frequency. We then presented drifting gratings matching these preferred parameters while systematically varying contrast levels E (15%, 25%. 50%, 100%) (a contrast level of 100% corresponds to maximum amplitude of the simulated gratings). We measured vPE neuron responses across N- = 500 trials, and computed trial-averaged firing rates (i.c., peristimulus time histograms, PSTHs) as shown in Fig. g
The results reveal response dynamics that are similar to those observed in biological VI neurons (Fig. () At high contrasts (50%, 100%), responses exhibit shorter onset latencies and higher peak firing rates compared to lower contrasts (15%, 25%). Response peaks also occur earlier in time at higher contrasts, with peak timing progressively delayed as contrast decreases. Quantitatively, we observe a mean latency difference of approximately 0.15 (in units of relative grating cycles) between the highest (100%) and lowest (15%) contrast stimuli, with intermediate contrasts showing progressively longer delays as contrast decreases.
This contrast-dependent latency shift arises naturally from ip- -AE'sss recurrent dynamics and is likely direct consequence of the divisive normalization mechanism eq. (8). Notably, the model was trained solely on static natural images, with no exposure to time-varying stimuli or explicit temporal structure.
C.4 Reconstruction-sparsiie with deterministic MAP decoding, and dependence on
In the main text (section 5). we explored the reconstructiion-saariiy trade-off, shown in Fig. We found that all iterative VAES outperformed their amortized counterparts, and that both LCA and iP-VAE achieved the best overall performance (Fig. 4p).
Here we note an important distinction in inference methods: LCA performs maximum a posteriori (MAP) estimation [39], while the VAE models maintain uncertainty in their posteriors by sampling. For more direct comparison, we also evaluated all VAE models using MAP decoding instead of sampling. Specifically, for the Poisson family, we used firing rates directly for reconstruction (instead
51

---

a
b
1.0
1.4
Euclidean distance from top-right corner
1.2
0.8
Reconstruction R
1.0
IP-YAE
83
0.6
0.8
0.6
0.4
0.4
0.2
PE-VAE G-VAE
0.2
0.0
0.0
S00 59
0.0
0.2
0.6
Sparsity [portion zeros]
Figure 9: Related Fig. but using maximum a posteriori (MAP) decoding instead of sampling for all VAE-based models. For Poisson models, this means using firing rates directly for reconstruction, while for Gaussian models, we use the posterior means. This approach provides a more direct comparison to LCA, which performs MAP estimation by design (39 Under these conditions, ip- VAE slightly outperforms LCC while maintaining the benefits of a probabilistic representation.
of spike-count samples), while for the Gaussian family, we used the posterior mean parameters. Note that for iterative VAEs, the inference dynamics was still fully stochastic (eqs. (6) (46) and (49)). ). The
difference here that for MAP decoding, simply used the final posterior parameters (instead of samples drawn from it) for the reconstruction.
MAP decoding. The results presented in Fig. 9 As expected, all VAE models improve their reconstruction 122 performance under MAP decoding. The best overall reconstruction achieved by iG-VAE and iP-VAE with a large Traan = 32, approaching R² 1.0. Notably, (P-VAE achieves this high reconstrucctio quality while maintaining considerable sparsity levels exceeding 60%, and slightly outperforms LCA in this MAP decoding setting. This result demonstrates that our probabilistic framework can match or exceed the performance of established sparse coding algorithms while maintaining the additional benefits of a fully probabilistic, integer spike-count representation.
The effects of varying 3. We also analyze the relationship between the & hyperparameter and iP-VAE performance in Fig. I0 The results confirm the thcoretical prediction that for Poisson models, the KL divergence term effectively functions as metabolic cost penalty (50), with higher values inducing greater sparsity. This relationship establishes a direct link between information rate (KL divergence) and our heuristic "proportion zeros" measure of sparsity, partially justifying the interpretation of the reconstruction-spariity landscape of Fig. 4 as a cnte-distortion map (77].
Interestingly, the relationship between 3 and model performance is significantly influenced by the training duration Timin- As shown Fig- [T models trained with longer inference trajectories (Tirin 32) achieve better reconstruction across all 3 values compared to those with shorter training iterations (Turan = 8), which tend to be sparser. The overall performance metric (distance from the optimal point, marked by gold star) reveals different patterns: Tirann 32 models show steady improvement that plateaus at higher values, while models with shorter training durations (Tirin = 8, 16) exhibit a U-shaped curve, initially improving but then degrading as 3 increases beyond an optimal point.
These results suggest that extended inference during training enhances the model's 's ability to find optimal representations, even when evaluated at fixed inference length during testing. Our approach bears similarities to meta-learning [91]. where the inner loop of optimization affects the outer loop performance. Future work should further explore this dependency on inner-loop iterations to better understand the mechanisms behind these performance differences across training regimes.
52

---

Overall performance [Euclidean distance from top-right corner]
T Trrain = 8
T Ttrain = 16
T train = 32
0.6
0.4
0.3
0.2
0.0
Reconstruction [R²]
0.9
0.7
0.5
sampled spikes
via firing rates
Sparsity [portion zeros]
0.90
0.85
0.80
0.75
0.70
0.65
x16
x32
-20
33.0
44
20
30
440
-20
-3.0
4.0
5
Normalized B B/Trrain train
Figure 10: Effect of 3 on ip- VAE performance metrics across different training durations. Each column represents different training duration (Train E {8, 16, 32}), with performance metrics shown as function of normalized 3 (scaled by Tuain). Top row: Overall performance improves with higher Timin with longer training allowing higher / values before performance degrades. Middle row: Reconstruction quality (R²) decreases with higher 3 across all training durations, but MAP decoding (using firing rates, gray dashed line) consistently outperforms sumpling-based based decoding (blue solid line). Bottom row: Sparsity monotonically increases with for all Tirain values, confirming the direct relationship between the KL penalty and representational sparsity for Poisson latents [50]. The gold star (defined in Fig. 4 indicates the theoretical optimum: perfect reconstruction (R 1.0) with maximum sparsity (portion zeros 1.0). See Figs. 4 and 9 for comparison with LCA and other iterative and amortized VAEs, embedded in a shared rrrrsstyyrcoonsttuuttto landscape.
C.5 Revonstruction-sparrity performance across latent dimensionalities
addition to 3 (Fig. 10), latent dimensionality also influences the reconstruction-spaaiiyy trade-off.
To systematically evaluate this effect, we trained iP-VAE models on 16 x 16 natural image patches (van Hateren dataset) with linear decoders, varying E {32, 64, 128, 256, 512, 768, 1024, 2048). All models used Timin 16 and 3 24, matching the configuration from the model shown in Fig.
53

---

Poisson (iP-VAE)
Gaussian + relu (igrelu-VAI
Gaussian (iG-VAE)
Figure 11: The full set of K 512 dictionary elements, corresponding to the models from Fig. Features are ordered in ascending order of their aggregate posterior membrane potentials (i.e., the final 1000000 for iP-VAE, and 10=1000 for Gaussian models, averaged over the entire test set).
54

---

Table Effect of latent dimensionality (K) on reconstruction and sparsity. The ip- VAE model from the main paper used K = 512 (highlighted), which balances performance and sparsity.
K
R²a f
Sparsity f
Overall perf. +
Active neurons
32
0.24
0.28
1.05
23
64
0.42
0.28
0.93
46
128
0.64
0.32
0.77
87
256
0.77
0.55
0.51
116
512
0.83
0.77
0.28
116
768
0.83
0.86
0.22
110
1024
0.83
0.90
0.20
107
2048
0.84
0.95
0.17
103
We report reconstruction quality (/t²,, sparsity (portion zeros), overall performance- Euclidean
distance from the ideal point: R² 1, sparsity = L, golden star in Fig. 4 and the average number of active neurons, defined as K (1 portion_ zeros). Results are shown in Table
We observe several key patterns:
Overcompleteness improves performance: Overall performance consistently improves K increases, with most gains occurring between K = 32 and K = 512. Performance saturates around 1024, with diminishing returns beyond this point.
Efficient coding emerges: The total number of active neurons plateaus around 1 110, even latent dimensionality grows to 2048. This suggests the model learns efficient sparse code that maintains consistent activity budget regardless of available capacity.
3. Reconstruction--parriyy trade-off: Beyond K = 512, further increases in dimensionality primarily improve sparsity (from 77% to 95% zeros) while reconstruction quality remains stable (R² 0.83-0.84). This validates our choice K = 512 for the main experiments reasonable balance point.
These results demonstrate that iP-VAE exhibits the expected overcomplete sparse coding behavior while maintaining computational efficiency through consistent use of approximately 100 active neurons across the natural image patch dataset.
C.6 Reconstruction performance and downstream classification accuracy MNIST
In this appendix, we explore the reconstruction fidelity and downstream classification accuracy of iterative and amortized VAEs MNIST, comparing them predictive coding networks (PCNs). Specifically, we compare to the original PC model of Rao and Ballard [22]. and recent enhanced version, incremental PC (iPC; Salvatori al. ). See Table for summary of model details.
C.6.1 Temporal scalability: iP-VAE scales consistently with training depth MNIST
As in the main text (section Fig. iterative VAE performance strongly depends Timin- Therefore, we first performed systematic search to select an appropriate Timin for different iterative VAE models on MNIST, exploring Timin E {4, 16, 32, 64, 128).
We evaluated these models using a consistent Tist = 1,000 and examined reconstruction performance. We observed markedly different behavior across variants. iG-VAE showed continuous improvement up until Tirain = 64, but plateaued thereafter with negligible gains from doubling to Train = 128. iG initially improved but began to deteriorate and diverge beyond Timin = 32. We suspect this instability stems from the absence of the divisive normalization mechanism that naturally emerges in iP-VAE theory (eq. (8)).
Notably, iP- -P-VAE was the only model that exhibited consistent improvements as we increased T Timin- It likely would continue improving with even longer training sequences, but the computational cost becomes prohibitive beyond what we tested (Timin can be conceptualized roughly as the number of layers in a deep ResNet, Fig. 5).
55

---

In conclusion, iP- demonstrated the most promising scalability characteristics (at least on MNIST). Based on these results, we selected T raann 128, 64, 32 for ii--iiP..... ig- VAE, -VE,, respectively, for the MNIST experiments reported below.
C.6.2 Iterative VAEs with an overcomplete latent space outperform PCNs on MNIST
We trained unsupervised models on MNIST and evaluated reconstruction quality using normalized per-dimension MSE, following prior work [113) Additionally, as the main text, we report Rt, dimeensionalityiiddeeeddn metric bounded between O and 1. To evaluate downstream classification performance, we extracted latent representations and trained a logistic regression classifier. Results are shown in Table 3
Iterative VAEs substantially outperform both PCNs and amortized VAEs in reconstruction fidelity. iP-VAR achieves the lowest per-dim MSE (4.39 x 10-$) and highest Rº (0.951) among all models, significantly surpassing both PC (MSE: 9.24 x 10-3 R2: 0.896) and iPC (MSE: 9.08 x 10-3 R2: 0.898). Remarkably, iP-VAE accomplishes this favorable reconstruction while maintaining high sparsity (84% zeros), whereas PCNs produce entirely dense representations.
For classification, P- P-VAE achieved the highest accuracy of 97.89% on downstream unsupervised classification, approaching the performance of supervised PCNs reported in ref. (T13). ip- iP-VAE achieves the second- best classification performance at 96.46%, substantially outperforming both PC (90.74%) and iPC (91.60%)
Limitations of the PCN comparisons. We acknowledge several important limitations in our comparative analysis. All iterative and amortized VAE models in this experiment used a single linear decoder, while amortized models employed convolutional encoders. In contrast, PCNs utilize hierarchical architectures with fundamentally different design principles. Rather than conducting comprehensive architectural optimization for PCNs, we used default settings from the official implementations released by Pinchetti et [T13), with only minor adjustments (e.g., doubling the number of epochs to ensure convergence, or matching latent dimensionality to other models see appendix C.1 for more details).
These substantial architectural differences limit the conclusiveness of our PCN versus iterative VAE comparisons presented in Table more rigorous comparison would require carefully controlling for network capacity and depth. Additionally, our evaluation metrics may inherently favor certain architectural choices. For instance, reconstruction metrics advantage models with overcomplete latent representations with direct pathways between the latents and reconstruction spaces.
Despite these methodological limitations, results provide valuable initial evidence that iterative VAFS can achieve comparable or superior performance to PCNs in both reconstruction quality and downstream classification tasks. Particularly noteworthy is that iP-VAE accomplishes this while maintaining high sparsity (84% zeros) and using integer-valued spike count representations that better align with biological neural computation. Future work should explore more controlled comparisons across these model families, including matching parameter counts, architectural depth, and computation budgets to establish more definitive performance relationships.
C.7 Runtime comparison: iterative versus amortized inference
A primary motivation for amortized inference is computational efficiency: a single forward pass through an encoder network should be faster than iterative optimization. However, as we saw in Fig. 4 this potential speed advantage comes at the cost of inferior reconstruction-sparne we quantify the actual runtime trade-off to determine whether iterative models' performance gains justify their computational costs and whether they are even slower at all.
performance. Here
Experimental setup. We compared the iterative VAE models from Fig. 3 (Tuzan 16) against amortized counterparts trained with standard ELBO (8 = 1). All models used linear decoders trained on 16 x 16 natural image patches (van Hateren dataset), with amortized models employing 6-layer convolutional encoders.
We measured inference time using CUDA events for accurate GPU synchronization:
56

---

Table 3: Comparison reconstruction performance, representational sparsity, and downstream classification accuracy on MNIST across model families. Iterative VAEs consistently outperform both PCNs and amortized VAEs in reconstruction metrics while using significantly fewer parameters than their amortized counterparts. Bold values indicate best performance in each category. Amortized VAE models used deep ResNet convolutional encoders, and all VAE models (both iterative and amortized) used single linear decoder for consistent comparison with the main text results. Iterative VAE models were trained with optimized iteration counts: Train = 128 for iP-VAE, Train = 64 for iG-VAE, and T Train = 32 for - based on convergence properties. Mean squared error (MSE) values are normalized per dimension for consistent comparison with previous benchmarks [13 Classification accuracy is reported using linear classifier trained unsupervised features. Values are mean = 99% confidence interval, estimated from 5 random initializations. See appendix for analysis of how Timin affects performance, and appendix C.6.2 for detailed interpretation.
C.6.1
Model family
Reconstruction
Portion Zeros
Classification Accuracy t
Num
Model
MSEx10-1 +
R.2
Params 4
iterative
iP- iP-VAE
4.39
.951
000
844
96.46_
0.40 M
iG-VAE
7.63
_914..000
2.0..0
92.73
0.40 M
VAE
19 VAE
8.45+.02
.906#.011
.78..oc
95.74=.4s
0.40 M
P-VAE
25.48
.713
84:..0
97.89 20.04
6.79 M
amortized
G-VAE
14.40
.841 2.022
.00s..o
95.75=2..0
6.93 M
VAE
Greta VAE
17.13
.809 2.022
.50..0o
95.41 21.51
6.93 M
PC
22
9.24
.896..000
.00+..o
90.74
0.13
PCN
iPC
900
9.08+.00
.898..omm
.00+..o
91.60
0.13 M
start torch.
(enable_ timbble--tiiingTrrre
end torch. cuda. cuda. Event (enable_ timing-True)
start. starr.record(
output model (x)
record))
torch. cuda. www.:cud.......oooocoo synchronizee duration = start. start. elapsed_time (end)
Since iterative methods distribute computation across many timesteps, they may underutilize available parallelism for small batches but benefit substantially from large-batch parallelization We therefore evaluated two extreme scenarios: single-sample inference (batch size = 1) and large-batch inference (batch size 10,000).
C.7.1 Inference runtime results: iterative models match exceed amortized speed.
Figure 12 shows reconstruction quality, sparsity, and overall performance as functions of wall- clock time. The key question is: how long does it take iterative models to reach the same overall performance as amortized models? The answer challenges conventional assumptions:
Small batches (batch size = 1): Amortized models complete inference ~ 3ms (marked by X's). Iterative models reach equivalent performance (distance from optimal point) within 3-15ms. Iterative models then continue improving, converging to superior final performance by 4000ss.
Large batches (batch size 10,000): Parallel computation reveals an unexpected advantage for iterative inference. Amortized models complete inference in ~ 50ms, but iterative models achieve the same performance (distance from optimal point) in approximately the same duration. After matching amortized performance, iterative models continue refining their representations, converging substantially better reconstructioon-paariiy trade-offss by ~ 2000ms.
This pattern holds across all three iterative variants: iP-VAE, iG-VAE, and iG VAE reach amortized-level performance within comparable speed to their amortized counterparts when batch size
57

---

is large. The iterative models then leverage additional compute to achieve superior final performance that amortized models cannot match.
Training time considerations. While inference speed is competitive between the two VAE types, training iterative models requires more time. As noted in appendix C.1.4 training iterative VAEs with Tuain 16 takes ~ 3 hours hours-approximiin approximately twice as long amortized models with 6-layer encoders. This reflects the effective depth of iterative models: backpropagation through time across Timin steps creates computational graphs comparable to deep ResNets (Fig. 5). Trrinning costs scale with both irain and dataset size, though inference remains efficient.
Implications. These results challenge the assumption that iterative inference is prohibitively slow. For small batches, iterative models match amortized performance at slightly slower speeds. For large batches, iterative models are actually comparable reaching equivalent performance levels, then continue improving to achieve superior reconstruction-saarity trade-offs. Combined with their better out-of-distribution generalization (appendix C.8) and 25x parameter efficiency, iterative models offer a compelling alternative to amortized inference. The increased training time appears to be the only meaningful trade-off, making iterative models particularly attractive for applications where inference performance matters more than training speed.
1.0
1.0
Distance optimal point
Sparsity (Portion Zeros)
20c
Reconstruction (R-)
0.6
0.6
0.4
VAE
0.4
Seee VAE
P-VAE
TL
GVAA
VAE
0.0
0.0
10-1
1.0
1.0
Sparsity (Portion Zeros)
X
Distance optimal point
30F
X
Reconstruction
0.6
IPVAE
10,000 = 221
0.4
x XXX
0.4
igVAE
lGru VAE
GAV
Gve-VVE
11 (rong m1
7700
10-1
100g
10a
Elapsed time (ms)
Elapsed time (ms)
Elapsed time (ms)
Figure 12: Runtime comparison between iterative and amortized VAEs for batch sizes of 1 (top) and 10,000 (bottom). Iterative models (solid lines) progressively improve over time, while amortized models (X markers) produce immediate but inferior results. Left: Reconstruction quality (IR²). Middle: Sparsity (portion zeros). Right: Overall performance (distance from optimal point, lower is better; see Fig. 4.. For large batches, iterative models reach amortized-level performance at comparable speeds to amortized models. Iterative models then continue improving to achieve superior final performance across all metrics by 2-200-300ms
C.8 iP-VAE exhibits strong out-of-distributiin (OOD) generalization
In the main paper, we compared our novel iterative VAE architectures, including the ip- -AEVE, their
amortized counterparts, as well as neuroscience models such sparse coding (39 and predictive
coding networks (PCNs: [22 90)). With exception of PCNs, all other models we tested in the
main paper employed single linear layer decoder (ár Φz, Fig. 5). In appendix B.6 we extended
our theoretical framework to nonlinear decoders by deriving inference algorithms that incorporate the
decoder's 's Jacobian (eq. 440 Fig. 6).
58

---

This appendix presents experiments with nonlinear decoders, where we replace the linear & with a deep neural network: a dec(2; 0), parameterized by 6. We compare ip-VAE P-VAE and hybrid heeaatve-amortized VAEs across various generalization tasks.
Tasks and Datasets. We evaluate performance primarily based on out-of-distribution (OOD; [98 (9]) generalization. measured through both reconstruction fidelity and downstream classification accuracy. We use MNIST as our primary training dataset, and test generalization on rotated MNIST, extended MNIST (EMNIST; 224)). ) Omniglot [225]. and ImageNer32 [226), with the latter two resized to 28 x 28 for consistent comparisons.
Architecture notation. We experimented with both convolutional and multi-layer perceptron (MLP) architectures, denoted using the (encIdec) convention. For example. (mlp (mpp) indicates both encoder and decoder networks are MLPs. We use (grad/mlp) to denote our fully iterative (non-amortized) ip- -PE,, which replaces the encoder neural network with natural gradient descent on variational free energy (section 4 and appendix B.6). We designed symmetrical architectures, ensuring that (mlp/m1p) has exactly twice many parameters as (gradmmpp)
Alternative models. To isolate the impact of inference method in OOD settings, we compare iP-VAE to the amortized P-VAE and state-of-the-art hybrid methods that combine iterative with amortized inference: iterative amortized VAE (ia- VAE; (91) both hierarchical (h) and single-level (s) variants, and comi-amortized VAE (sa- VAE; (92).
Inference iterations. For iP-VAE, we explored different training iteration counts, Timin E (4, 16,32, 64). Recall that during training, we differentiate through the entire sequence of iterations (learning to infer), which can lead to qualitatively different dynamics. For most OOD experiments, we use (grad|mlp) iP-VAE with main 64. At test time, we evaluate with Tist 1,000 iterations. For the hybrid models, we use their default iteration counts unless otherwise specified (sa- VAE: Tiain Tist 20, VAEVAE: Tiain Tiett 5).
C.8.1 Stability beyond the training regime and convergence for nonlinear decoders
An algorithm with strong generalization potential should learn inference dynamics that extend beyond the training regime. In the main paper (section 5). we found that all iterative VAEs converge when trained on natural image patches with linear decoders (Fig. 3 Here, investigate whether this convergence holds for the more challenging case of nonlinear decoders.
We trained models MNIST using different numbers of training iterations (Train E {4, 16, 32, 64)) with both (grad/mlp and (grad)com) conv) architectures, then tested whether they continue to improve beyond Tuain As shown in Fig. 13,, even iP-VAE models trained with few as Train = 4 iterations learn to keep improving during test time, eventually converging to stable solutions. Interestingly, models trained with more iterations initially show worse performance but ultimately converge to better solutions, suggesting that iP- VAE learns temporally-exter extendde inference dynamics that depend on the training sequence length but generalize well beyond it.
In contrast, hybrid models (sa-VAE and ia- VAE) begin with strong amortized initial guesses but plateau rapidly (Fig. 13p, right), converging to substantially higher MSE than (P-VAE models despite having more parameters. We note that sa- VAE's s authors acknowledged issues with the dominance of the iterative component Omniglot and reported using techniques like gradient clipping to mitigate this (see footnote in [92),, which may explain our observations on MNIST. More concerning, ia- VAE (single-level) begins to diverge outside its training regime and frequently resulted in numerical instabilities at test time when exceeding Turain
Overall, VP-VAE demonstrates better reconstruction performance and continues to improve outside the training regime, unlike other models. This extends our convergence results from linear decoders (Fig. 3) to the more general nonlinear case and reveals iP-VAE's first form of OOD generalization: temporal generalization beyond the training horizon. Next, we examine whether iP-VAR can generalize across visual domains.
59

---

(a)
Reconstruction loss (MSE)
(grod/nlp)
(grad coav)
Tont 16
Tran
Tran
Tha 64
16
sa-VAE
Tran 64
la-VAE (N)
VAEE (S)
10t
10
10a
Trest
10a
10
10a
T
10a
10
10
Test
10a
Tan
8
8
8 $ 8
sa-VAE
6
2
8 4 5 6
8
Tue 10
0
1
2
4
6
0
2
5
6
7 7
A
Tuo
1
2
7 8
8 A
la-VAE (h)
0
3
6
Tue 1,000
/
2
7
2
2
6
A
la-vAE
6
4
Ground truth
0
/
2
5
6 8
0 1
28
5 8 (grad conv)
4
(b)
6 10
(gradinlp) Timin
Reconstruction loss (MSE)
Than 16
16
4
Timen 64
x 10
ARAME
VSB
2 10a
LETE
L LEud
10"
10
T
10
10
10
10
T
10
10
10a
10
T
10
10
Test
Test.
test
AR-VAE
Ton
9
3
8
8
2
4
8
0
2 3 4
S7
7
To Ba 34
4
Tua 10
19
4
6
2
4
1
4 4
3 76
7
1
4
6
2
2
3
ia-VAE (h)
Tea 100
13
6,
2
25
10
12
4
4 4
2
3
4
10
3
33
10
76 2
99
la-
54 #
Ground truth
10
10f'
6
35
13
4
4
16 1of
4
13 7 A
op
Mapp
7 A
Figure 13: VP- generalizes beyond its training regime and across datasets. All models shown here are trained MNIST and tested either MNIST (panel a) or Omniglot (panel b). (a) iPE models trained with as few as Tizain = 4 inference steps continue to improve well beyond the training regime (Tist Timin). This temporal generalization holds across architectures: (grad/mlp) (left) and (grad/conv) (middle). In contrast, hy brid
models either plateau or diverge outside their training regime (right). (b) ip- -P-VAE trained exclusively on MNIST successfully generalizes to the structurally different Omniglot dataset, showing substantial improvement in reconstruction quality inference iterations increase. With higher T Tirain iP-VAE models show initially worse but ultimately better performance, suggesting they learn more complex but effective inference dynamics.
C.8.2 Out-of-distributiio generalization
We evaluate two levels of generalization for MNIST-trained models: (1) within-dataset perturbations, and (2) transfer across related datasets.
OOD generalization to within-dataset perturbation. We tested whether models trained on standard MNIST could generalize to rotated MNIST digits [98 227]. At test time, rotate digits between 0 and 180 in increments of 15". We evaluated (a) reconstruction quality of the rotated digits, and (b) whether the resulting latent representations support downstream classification (Fig. I4].
iP-VAE and sa- VAE maintained relatively consistent performance across rotation angles, though (P-VAE demonstrated superior performance in both reconstruction and classification. The amortized P- VAE showed substantially worse reconstruction than iterative models, but its classification accuracy remained remarkably stable across angles, outperforming all models except iP-VAE.
In contrast, both ia-VAN VAB variants showed significant degradation in performance as the rotation angle increased, with substantial drops in both reconstruction fidelity and classification accuracy. Overall, iP- VP-VAE maintained the most stable and superior performance across all rotation angles, suggesting leams rotation invariiat features that generalize effectively unseen transformations.
60

---

Geourd
2 8
4
5
6
7
8
Reconstructio (MSE)
2
3
7
8
PIMAE WA-ONE 01)
IPAVAE
2 8
6
8
INAVIE (SI VAE
PAVAE
2
6
7
S8AAE
0 22244
1.0
2
8
4
5
6
10h
90
0
PPVVE
0
7
9
4
Classiiication
PAVAE
0500
000001111
1 7
6
3
3
A
0 1
2
6
3
2 88
6
2
sa-VAE
2 8
15
30
45 75 135 120
135
150
165
180
Figure 14: Robustness to rotational transformations of MNIST digits. Left: Visual comparison of reconstructions for moderately rotated (0 15t) and severely rotated 90%) digits across models. iP-VAE maintains high reconstruction fidelity across rotation angles, while other models show progressive degradation. Right: Quantitative evaluation of reconstruction MSE (top) and downstream classification accuracy (bottom) across rotation angles from 0' to 180 ip- iP-VAE demonstrates remarkable stability in both metrics, with minimal performance degradation even at extreme angles. The solid lines represent means, while shaded regions show standard deviations across test samples. All models were trained only on standard (unrotated) MNIST, making this a challenging test of generalization to unseen transformations.
Graadd
T Ad V
10
A
Recomannziion (MSS)
A Q l t 1 V
e
W
6
2
3 d
PUNE
DAVE
0 3 Q 8 8 2
0 / 5
L 4 4
4
7
9 8
#0R0
6
T
G
2
4
3 Q 8 &
F.W
2
A A Q 1
e
F
W
2
13
Orrigot
Figure 15: Cross-dataset generalization from MNIST novel character datasets. Left panels: Visual comparison of original samples and their reconstructions for EMNIST (left) and Omniglot (center-left) across different models. All models were trained exclusively on MNIST. Center-right panel: Reconstruction MSE for EMNIST and Omniglot. Lower is better. iP-VAE consistently achieves the lowest error on both datasets despite never seeing them during training. Right panel: Classification accuracy on EMNIST using linear classifier trained on latent representations. iP-VAE representations support significantly higher accuracy, suggesting they capture more discriminative features that transfer well to new character classes.
OOD generalization across similar datasets. A model that learns compositional features with an effective inference algorithm should generalize datasets structurally related to, but distinct from, the training distribution. To test this, we evaluated MNIST-traine models on EMNIST (Fig. [5] and Omniglot (Fig. L3p. Fig. I5}. reporting both reconstruction MSE and classification accuracy (except for Omniglot, which has over 1 1,000 classes, making classification challenging).
iP-VAE consistently demonstrated superior generalization, achieving lower reconstruction error and higher classification accuracy than all other models. The visual quality its reconstructions on both EMNIST and Omniglot noticeably better, preserving key structural elements of characters it had never seen during training. This suggests that iP-VAE learns more general and compositional latent space that captures fundamental structural elements shared across different character datasets.
Extreme cross-domain generalization with iP- from MNIST to natural images. Most remarkably, (grad|mlp) iP-VAE trained exclusively on MNIST still performs well when tested cropped, grayscaled natural images from tiny ImageNet (Fig. 16]. domain vastly different
61

---

32
Figure 16: Extreme cross-domain generalization: MNIST-trained models tested on cropped, grayscaled natural images. Left panels: Ground truth grayscale ImageNet32 samples (resized to 28 x 28) alongside their reconstructions from various models. Despite being trained exclusively on MNIST digits, iP-VAE captures sigmificant semantic information and structural details of natural images. Right panel: Reconstruction MSE over inference iterations. iP-VAE continues to improve with more iterations, while hybrid models plateau quickly. The ia VAE variants fail adapt to the new domain, while sa- aA-V shows moderate adaptation. This challenging generalization task demonstrates iP-VAE's ability to represent visual concepts far outside its training distribution. This capability is likely enabled by its Gabor-like primitive features that can be recombined to represent novel visual structures (Fig.
iP- -P-V
(grad|milp)
P- P-VAE
(mlpinlp)
Al festures (test layer of MLP deoode
Examples zoomed in
Figure 17: Visualization of learned features in the final decoder layer for iP-VAE with (grad|mlg architecture (left), versus an mnlp/mmp) P--AEE (right), both trained on MNIST. iP-VAE learns Gabor-like features (fundamental visual primitives found in primate early visual cortex [107] rather than digit-specific features. In contrast, P-VAE learns more class-specific features resembling complete digits digit fragments. Features ordered by increasing kurtosis of their weight distributions to highlight the sparse nature of TAE'' representations. The more general, primitive features learned by iP-VA PP-EEE likely contribute its superior generalization capability across datasets, as these features can be recombined to represent novel visual structures not seen during training.
from handwritten digits. Despite never seeing natural images during training. iP-VAE continues to improve its reconstruction quality with increasing inference iterations, capturing significant structural details; whereas, hybrid models like (conv/conv) sa-VAE and mmlp/mmp) ia- VAE plateau quickly or even diverge (Fig. 16] right panel). This extreme cross- mooss-domain transfer challenges common assumptions about generalization boundaries in representation learning and suggests that iP- captures something fundamental about visual structure that transcends specific datasets.
Primitive visual codes enable generalization. How does ip- VA-V achieve this unexpected gener- alization capability? We visualized the 512 learned features from the final layer of the (grad ml iP-VAE's decoder and compared them to those learned by amortized (m1p (mlp|mlp) P-VAE (Fig. 17
The contrast is striking: iP-VAE learns Gabor-lik features fundamental visual primitives similar to those found in primary visual cortex white P-VAR VAE learns features that resemble entire digits or digit fragments. Previous work has identified strokes as compositional sub-components of digits (228) but iP-VAE learns even more general and primitive visual code. These elementary Gabor-like features, combined with iP-VAE's adaptive inference process, create a powerful compositional system that can flcxibly recombine primitives to represent novel visual structures far outside the training distribution.
62

---

C.8.3 A discussion of the out-of-distribution (OOD) generalization experiments
Empirically, iP-VAE demonstrates promising adaptability and robustness to OOD samples while dy- namically balaneing computational cost and performance. consistently outperforms both amortized models and hybrid iterative-amortized VAEs on the tasks we evaluated, despite using substantially fewer parameters.
These results are especially promising given the growing emphasis on OOD generalization in machine learning [98] 99 227]. ip- (P-VAE' 's ability learn primitive, compositional features that transfer across domains suggests its potential value for representation learning, whether as a pretraining method for downstream tasks as a backbone for diffusion models 229 230].
That said, we acknowledge that more systematic evaluation is needed to fully characterize the compositional capacity of iterative VAEs. While our experiments focus primarily reconstruction and classification, future work should explore more diverse tasks and evaluation metrics. The strong performance demonstrated here motivates a deeper investigation into the generalization capabilities of iP-VAE and its potential applications across a broader range of domains.
C.8.4 Methodological details for the OOD experiments
For comparisons, utilized the official implementations of sa- VAE [22 ia- VAE 11], and P-VAE [50]. Across models, we maintained consistent train/validation splits and hyperparameters unless otherwise specified.
Since sa- VAE's original code used Bernoulli observation model, we adapted it for Gaussian compatibility by removing the sigmoid in the decoder and replacing its reconstruction loss with MSE. For sa- sa-VAr VAE, we used the default configurations provided for Omniglot (batch size 50, 100 epochs), and adjusted appropriately for other datasets: MNIST (batch size 50, 32 epochs), and EMNIST (batch size 50, 16 epochs).
For ia- VAE, we used the authors' provided configurations for both Bernoulli and Gaussian observation models, as appropriate. The MNIST configuration was applied across our MNIST, EMNIST, and Omniglot experiments. Although the codebase hard-codes training to 2000 epochs, we halted training earlier, once the loss had converged (between 780 and 2000 epochs). In some runs, we encountered numerical instabilities that required restarting training from checkpoints.
Overall, the consistently strong performance of iP-VAE across these generalization tasks suggests significant potential for applications in representation learning and transfer learning. We hope these results motivate further exploration of iterative inference models for representation learning.
C.9 iP-VAE scales up complex, high-dimensional color image datasets
Theoretical neuroscience has produced many normative models, including predictive coding (22). sparse coding 39 54). efficient coding (231). and the free-energy principle (24). However, from machine learning standpoint, existing implementations rarely meet modern data complexity require- ments. Sparse coding, despite decades of development and widespread recognition of its advantages, still predominantly uses linear decoders and operates on small, grayscale image patches. While we previously demonstrated that VPE can perform nonlinear deep sparse coding (appendices) B.6 C.8 and C.10P. a critical question remains: can it scale to realistic, high-dimensional datasets?
Here we show that VA- successfully scales to modern color image datasets, including CIFAR-10 (3 x 32 x 32), tiny ImageNet (3 x 32 x 32), and CelebA (3 x 128 128).
CelebA (cropped to 128 x 128). We trained four models on CelebA to systematically compare iterative versus amortized inference and deep versus linear decoders:
1. iP-VAE, (grad) conv): iterative encoding + 8-layer convolutional decoder
2. iP-VAE, (grad]ling lin): iterative encoding + single-layer convolutional layer decoder
3. P-VAE, (conv) conv): convolutional encoder 8-layer convolutional decoder (amortized counterpart of #1)
4. P-VAE, (conv|lin): convolutional encoder single-layer convolutional decoder (amortized counterpart of #2)
63

---

All models used latent dimensionality 256 x 16 x 16, with linear decoders using kemel size 38. We trained with | = 1 and evaluated iroonstructionnspasstyy performance. Overall performance is measured as Euclidean distance from the ideal point (R 1, sparsity = 1, marked by the gold star Fig. 4 lower is better. Results are shown in Table
Table 4: Reconstruction
performance on CelebA (cropped to 128 x 128). The (grad conv) ip- -AE with deep decoder (model #1) achieves the best overall performance while using 63% fewer parameters than its amortized counterpart (model #3).
Model
Encoder
Decoder
R² f
Sparsity f
Performance +
Params
(1) PriiP-VA VAE
iterative
8-layer conv
0.95
0.93
0.09
0.7 M
(2) PP---AAE
iterative
linear conv
0.90
0.91
0.13
1.1 M
(3) P-VAE
conv
8-layer conv
0.86
0.96
0.15
1.9 M
(4) P-V
conv
linear conv
0.74
0.99
0.26
2.3 M
CIFAR-10 and Tiny ImageNet (32 x 32). To verify scalability across datasets, we trained (grad]conv) iP-VAE with 6 convolutional layers and latent dimensionality 256 x 8 x on both CIFAR-10 and tiny ImageNet (5 = 1). Results are shown Table 5
Table 5:
Recoosttacttonn-saarsee performance CIFAR-I0 10 and tiny ImageNet (32 x 32). The iP-VAE consistently achieves strong performance across diverse natural image datasets.
Dataset
R t
Sparsity
Performance
CIFAR-10
0.93
0.97
0.08
Tiny ImageNet
0.95
0.94
0.08
Across all three datasets, iP-VAE achieves overall performance scores of 0.08-0.09, demonstrating consistent scalability complex, high-dimensional color images. These results provide strong evidence that normative, brain-inspire models need not be limited to toy datasets: iP-.AE bridges theoretical neuroscience and modern machine learning, achieving superior connnstuucionsspassiii trade-offs with fewer parameters than amortized alternatives.
C.10 VP-VAE generates novel samples through an iterative procedure
The primary focus of this work (and of FOND more broadly) is on brain-like inference dynamics, where generation serves means to this end (i.e., "analysis-by by- (68)). Nevertheless, assessing the model's I's generative behavior provides useful insights into its learned representations.
To this end, we performed qualitative evaluation of (grad|conv) iP-VAE trained on the MNIST
dataset with T Train 16. The generation procedure is as follows:
Algorithm Iterative Generation Procedure for 1P-VAE
Sample from initial prior: Zg Poiss(z;; exp(( (u))
2: for f.= = 1 to T do
3: Generate image: क; <- fo(z(-1)
decoder output
4: 'Treat generated image input: H; + 2;-1
5: Run one inference step: update u, via eq. 77) where x xt
6: Sample new latent:
7:
end for
The model successfully produced coherent and recognizable digits (Fig. 18) Notably, during the iterative generation process, some initially ambiguous samples (e.g., resembling both '3' and '6') would converge to well-defined digits after several refinement steps. This progressive clarification mirrors the convergence behavior observed during inference (Fig. 3) suggesting a unified dynamical principle underlying both processes.
64

---

a Sample generation process across time (using (grad) conv) iP-VAB
x
g
8
f
d
8
# R 4 $
5
8
8 &
0
4
A
$
3
3
3
3
8
8
8
3
3
3
2
8
8
8
8
2
2
2
2 2 2 2
2
2
2
2 2
9
9
S
3
5
8
6
O
6
O
6
Ó
2
5
S
5
S
9
7
8
7
7
4
4 4
(b)
Generated samples at 100 iterations (using the same architecture as a)
/
8
8
8
8
6
3
8
8 6
9
4
7
7 4 9 3
3
0
4
5
6
9
7
1
5
0
6
)
3
3
7
9
7
7
4
9
7
3
+
3
4
7
7
9
3
8
7
6
0
3
0
9
8
6
8
7
3
1
8
8
3
7
6
7
6
7
9
f
7
7
9
4
5
4
2
7
4
3
7
9
4
8
/
3
4
G
3
5
3
/
9
7
7
9
2
2
7
9
4
8
5
7
7
0
6
6
9
2
/
7
4
5
2
5
L
8
9
3
3
7
3
3
8
3
0
5
8
5
8
4
9
6
4
8
1
9
1
8
7 9
8
3
9
3
4
/
7
4
3
9
3
2
3 9
3
9
2
1
7
4
6
8
7
L
6
8
6
7
3
4
5
5
5
7
7
2
1
4
4
6
5
4
9
f
5
2
6
7
7
3
7
4
7
3
4
3
3
0
8
6
5
Figure 18: Generation results from a (grad/cony) -P- trained on MNIST, sampled using
Algorithm [ (a) Temporal dynamics of generation: horizontal axis shows iteration steps, vertical axis shows different sampled trajectories. (b) Final generated samples after 100 iterations.
However, we observed modest class imbalance in generated samples: certain digits (e.g., '8') appeared more frequently than their 10% training prevalence. This bias likely reflects the learned prior distribution exp(uo) and could be mitigated through more expressive hierarchical priors.
D Extended Discussion
This appendix expands on key topics introduced the main paper's discussion (section 6). We begin by summarizing FOND and discuss how it relates to existing literature (appendix [.I]. We then analyze possible factors underlying ip. VAE's effectiveness (appendix [D.2], assess its potential for hardware implementation (appendix D.3), and explore relationships to sampling-based inference methods (appendix D.4).
We conclude with discussion of limitations and future directions (appendix D.5). These include:
accelerating iterative inference (appendix D.5.I). biologically plausible learning (appendix D.5.2).
predictive dynamics (appendix (D5.3), hierarchical extensions (appendix D.5.4). and applications to explain neural data variance, thereby bringing theory and experiment closer together (appendix (
D.1 A summary of prescriptive framework, and how it relates to BLR and BONG
In this work, we started by reviewing and synthesizing existing models- VAEs in machine learning to sparse coding and predictive coding in neuroscience-att instances of variational free energy (F) minimization. Drawing on this unification potential, we introduced FOND (Free energy Online natural-gradient Dynamics): framework that prescribes a clear sequence of choices to derive brain-like inference algorithms and architectures from first principles.
FOND establishes three-step guideline for developing inference models:
1. Select the distributions that fully specify the variational free energy, F
2. Identify the dynamic variables that parameterize the approximate posterior
3. Apply online natural gradient descent on J to derive the dynamics
65

---

The first two steps are flexible design choices, while the third is fixed: it is an algorithmic prescription that determines the equations governing the temporal evolution of the dynamic variables. Equivalently. this also fully determines the dynamics of the posterior distribution.
To fulfill the "online" component (the O in FOND), we require that inference is performed through continual belief updates, reflecting the adaptive 445 and recurrent [40H44] nature of neural processing. This iterative, sequential approach departs from the one-shot amortized inference typical
in VAEs 37 38 217 The "natural gradient" component [34 35 57] (the N in FOND) ensures that
optimization respects the intrinsic geometry of probability distribution space 65
200
Situating FOND within existing literature: FOND vs. BONG BLR. Both FOND and BONG
[60] are specialized applications of the Bayesian Learning Rule (BLR; (34)) in sequential settings, but they address fundamentally different problems through distinct technical approaches. We clarify these relationships here.
BLR provides general mathematical framework for deriving learning algorithms as natural gradient descent on variational objectives. It unifies diverse optimization methods from SGD to adam-underr single principled lens. Both FOND and BONG instantiate this framework but diverge in their goals, scope, and technical implementation.
Bayesian Learning Rule (BLR, 34
BONG (Bayesian Online Natural Gradient, 660) BONG focuses learning the parameters
of Bayesian neural networks in supervised settings. Given labeled data (xx, y). BONG updates network weights online to improve predictive performance. Technically, BONG restricts itself to Gaussian approximate posteriors, which enables the use of analytical results- sprlissaaii Price's [232] and Bonnet's [233] theorems-to compute gradients of expectations (their Eq. 3). This Gaussian-specific approach yields closed-form update rules (their Eqs. 9-10) that BONG then approximates in various ways to achieve computational efficiency.
FOND (Free energy Online Natural-gradies Dynamics) - FOND, by contrast, focuses on deriving inference dynamics for variational parameters (e.g., membrane potentials u) in unsupervised settings. FOND's goal is to specify how neural activity evolves over time to perform posterior inference, not to learn network weights.
This fundamental difference in goals necessitates different technical approach: FOND must be distribution-agnosiic to derive models like iP- (Poisson) alongside iG-VAE (Gaussian). Because Price's and Bonnet's theorems do not apply beyond Gaussians, FOND follows the standard VAE approach: approximate expectations via Monte Carlo sampling, then compute gradients of the resulting stochastic objective. This generality allows FOND derive the spiking dynamies of iP-VAE, which is impossible within BONG's Gaussian-restricted framework.
BONG vs. FOND: Key Differences
Goal: BONG learns network parameters (6); FOND derives neural dynamics (t)
Setting: BONG primarily targets supervised learning (though applicable to unsupervised settings via likelihood choice); FOND targets unsupervised inference
Scope: BONG is Gaussian-specific; FOND is distribution-agnosiic
Technical approach: BONG uses analytical gradient computation via Price's and Bonnet's theorems; FOND uses Monte Carlo sampling then differentiation
These are not merely different "flavors" of the same algorithm but distinct mathematical procedures addressing complementary problems. BONG focuses on online supervised learning with Gaussian uncertainty. while FOND derives brain-inspired inference dynamics across diverse distributional assumptions. Both enrich the BLR framework by demonstrating its versatility in different domains.
FOND's utility: application to derive iterative VAEs. To demonstrate the viability and utility of FOND, we developed a novel iterative VAE model family, including a spiking variant, the iP-VAE.
For this specific implementation, made two biologically motivated choices:
66

---

1. Modeling neural activity as spike counts [800] 192 193 195 234 236 via Poisson distributions [50], departing from the standard Gaussian assumptions in predictive coding [22 33
2. Assigning log-rate as the dynamic variable for each neuron. Log- rate is the canonical parameter of the Poisson distribution, and it is interpretable as the membrane potential of the neuron 70
With these choices fixed, the natural gradient descent on unambiguously determined the form of the spiking neural dynamics in iP-VAE.
The resulting architecture, ip- PA-,,,, shares theoretical and empirical similarities with L.C. (39] effectively functioning as spiking, stochastic extension of this classic model. Our empirical results showed that all iterative VAEs (iP-VAE, ig- i consistently outperformed their amortized counterparts in balancing reconstruction and sparsity, with iP-VAE and LCA achieving the best overall trade-offs. Additionally, when evaluated against hybrid iterative VAEs in out-of-distribution generalization tasks, iP-VAE showed strong performance.
Why is -PE effective? We explore this next by drawing connections to recent advances in sequence modeling and hierarchical VAE theory.
D.2 Three possible explanations for the effectiveness of iP-VAE
Several complementary factors contribute to iP-VAE's effectiveness. First, we note interesting parallels with recent advances in sequence modeling. Recently, Beck et al. (T14) introduced extended LSTM (XLSTM) that performs competitively with state-of-the-art transformers and state space models. xLSTM achieves this mainly through two key modifications: (1) replacing sigmoid gates with exponential nonlinearities, and (2) implementing a normalization scheme that stabilizes training. Similarly, ip- PP-V employs the exp(-) nonlinearity map membrane potentials to firing rates, providing a highly expressive activation function (Figs. 5 and 6 Furthermore, the theoretical derivations naturally gave rise multiplicative divisive normalization that stabilizes network dynamics and promotes convergence (eq. (8)..
Second, all our iterative VAES, including iP-VAE benefit from effective stochastic depth. Child [15] demonstrated both theoretically and empirically that hierarchical VAEs gain their effectiveness from stochastic depth. When unroll the iterative VAE encoding algorithms in time (e.g.. Figs.
and they resemble specialized hierarchical VAEs, but without the spatial coarse-graiee grrsiiin found in typical hierarchical architectures [115 125 161]. Based the arguments put forth by Child (113), we speculate that this temporal depth in ip. VAE's stochastic representations contributes significantly model performance.
In summary, iP- PP-VsE's success likely stems from least three key elements: (1) the exponential nonlinearity providing expressiveness, (2) emergent normalization ensuring stability and convergence, and (3) effective stochastic depth through temporal unrolling.
A promising direction for future work would be extending iP-VAE into full hierarchical architecture with spatial coarse seare-grriniing potentially further enhancing its already strong performance, which we discuss in detail below.
D.3 Hardware-efficient implementation and real-world deployment potential of iP-VAF
The computational characteristics of iP-VAE make it particularly well-suited for efficient hardware implementation, especially specialized accelerators and resource-constrained devices. This efficiency stems from three key properties: (1) sparsity, (2) weight reuse, and (3) integer-valued representations, as detailed below.
First, VAE generates sparse, integer- valued spike counts as representations. This sparse coding strategy aligns with emerging sparse tensor accelerators and neuromorphic computing architectures, which can achieve substantial improvements in energy efficiency compared to dense matrix operations 116). The predominance of zeros in these representations (60%-95% in natural image patch experiments, Fig. and up to 84% in MNIST experiments, Table 3 enables both memory and
computation savings through sparse storage formats [122 and siippig optimizations [03
67

---

Second, iP-'sE's inference algorithm reuses the same set of weights across all iterations (Fig. 5 potentially reducing memory bandwidth requirements compared deep neural networks with unique parameters at each layer. This is particularly significant for hardware deployment, as memory transfers often dominate energy consumption in neural network inference [T03] 237 238]. Where traditional deep networks must continuously load new parameters from memory, ip- -AA''s weight-sharing minimizes costly memory accesses and allows for efficient on-chip caching of the entire model.
Third, the discrete, integer-valued nature of spike counts eliminates the need for high-precision floating-poits arithmetic [103 [05], enabling deployment on simpler fixed-point hardware with reduced power consumption. Combined with the inherent sparsity, this positions ip-VAE as a promising candidate for edge computing applications where energy and computational resources are severely constrained.
These properties are particularly valuable for real-world autonomous systems and robotics applica- tions, where power budgets are limited and attaching power- GPUs is impractical [103]. For instance, drones, mobile robots, and wearable devices could leverage iP-VAE for efficient on-device perception without requiring cloud connectivity or massive batteries.
As noted above, iP-VAE and LCA share key theoretical, computational, and empirical characteristies. Given that LCA has been successfully implemented both as spiking neural network [123] and on neuromorphic hardware (239). we expect ip. VAE to be similarly deployable such platforms, potentially with even greater efficiency due to its principled stochastic dynamics.
Overall, iP-VAR's VAE's combination of sparse integer representations, weight reuse, and alignment with neuromorphic computing principles offers a promising path toward deploying sophisticated perception capabilities in resource-constraiie environments, bridging the gap between theoretical neuroscience models and practical applications.
D.4 Relationship to sampling-based inference methods
Throughout this work, we focused variational inference (VI) our framework for approximating Bayesian posteriors. However, campling-based methods, particularly Markov Chain Monte Carlo (MCMC; (43)) represent equally important alternative approach. Like VI, sampling-basee
methods have been extensively explored in theoretical neuroscience 117 122
244 252].
See Hacfner
et al. 253 for balanced review.
Both VI and sampling address key limitation of maximum a posteriori (MAP) inference by providing access to the full posterior distribution rather than just point estimate. The iP- VAE extends LCA
(39] from a deterministic point estimate to probabilistic representation with full posterior uncertainty.
Similarly, recent campiing-based approaches have enhanced both sparse coding (20 and predictive
coding [12T] 122 by incorporating stochastic dynamics that explore the full posterior landscape rather
than converging to single point. These parallel developments in both VI and sampling frameworks reveal their complementary roles in addressing the limitations of traditional MAP estimation.
Sampling-based models all perform Bayesian posterior inference, but differ in whether they adopt a "prescriptive" approach, as outlined in appendix B.1 For a non-prescriptive example, Echeveste et al. [247] began with a well-established biophysical circuit- the stochastic stabilized supralinear network (SSN;
83 254 255)) and adapted to perform posterior inference. Their optimized network exhibited key biological properties and achieved substantially faster sampling than standard Langevin samplers. These results compellingly demonstrate that biologically realistic circuits can perform Bayesian inference. However, this represents a bottom-up, descriptive from an existing circuit and adapting it for inference; whereas, other sampling- based approaches align more closely with the top-down, principle-driven methodology we advocate.
starting
particularly relevant example is the work of Masset et al. [T19), who demonstrated that incorporat- ing natural gradients into Langevin sampling significantly accelerates convergence. Their approach directly parallels ours: they derive spiking neural network algorithm prescriptively from first princi- ples, drawing inspiration from Ma et al. [256], who proposed a "complete recipe" for constructing MCMC samplers. This "complete recipe" serves a role analogous to the Bayesian learning rule (BLR; (34)) in our work, they both provide unifying framework for their respective domains. While BLR unifies learning algorithms under variational inference with natural gradient updates, the "complete recipe" provides corresponding unifying framework for sampling- based MCMC methods.
68

---

Interestingly, our iterative VAE models, while fundamentally grounded in VI principles, incorporate sampling elements during inference by approximating the reconstruction loss through Monte Carlo sampling. This hybrid character hints at an opportunity for theoretical unification between VI and MCMC. Prior work has explored this direction (257] 258], but further research is needed to clarify the relationships between VI and sampling. This unification potential is particularly interesting in the
context of brain-like inference models [23 117 253].
D.5
Limitations and future work
In this final appendix, we discuss the limitations of our work and how future work can address them.
D.5.1
Iterative inference: Generalization advantages with manageable computational costs
Our experiments consistently suggest that iterative inference outperforms amortized approaches in cronstruction-spariiyy trade-offs (Fig. 4 Additionally, the fully iterative iP-VAE also outperformed hybrid
VAEs in -uitr--distribution (OOD) generalization experiments, including both reconstruction and downstream classification OOD tasks (Figs. 13 [6].
Furthermore, we observed clear pattern: the less amortized the inference process, the better the generalization capability. Semi-amortized VAE (sa-VAE; (92), which relies partially on iterative
updates, outperformed the more heavily amortized iterative amortized VAE (ia-VARE VAE; (9I), suggesting that reducing dependence on learned amortization functions enhances adaptability to novel inputs.
Iterative reasoning other domains. Similar advantages of iterative inference have been observed in other domains well. Recent advances in large language models employ iterative reasoning at test time, either explicitly through prompting techniques like chain-of-thour (259) and (260). through more sophisticated inference-time algorithms and architectures. These approaches generally fall into two categories: some focus on the inference algorithm, like the MCMC-based sampler from Karan and Du (261). which elicits latent reasoning from a base model. Others modify the architecture itself to be inherently more iterative. For instance, the prain-inspired Hierarchical Reasoning Model (HRM; (262]) introduces a novel recurrent structure for latent reasoning, while Koishekenov et al. 2663] train a model to recursively apply its own reasoning-critiin critical layers.
This principle has earlier roots in computer vision, where the brain-inspired recursive cortical network (RCN: (264[265) showed that a generativv model using iterative pessage-passing equipped with both feedback and lateral recurrent connections, could achieve remarkable data efficiency and generalization on challenging visual reasoning tasks, such as breaking text-based CAPTCHАS (264).
These diverse approaches-spaniine similar principle: dedicating more test-time computational steps problem enables superior performance on complex reasoming tasks, despite the additional overhead.
prompting, sampling, and architecture-al converge on a
The power of iterative inference is particularly evident on benchmarks designed test the limits of abstract reasoning. On the challenging ARC-AGI benchmark [266). Liao and Gu (267] achieved strong performance 73%.75% on training set, 20% evaluation set) by replacing encoder networks entirely with expectation- matiiiioamaiiiiiiiionn (EM) iterative updates. This is conceptually similar to approach, where iterative optimization replaces learned encoders. Remarkably, Liao and Gu (267) achieved these strong results without any pretraining, further supporting the idea that iterative inference may offer inherent advantages for generalization on difficult, abstract problems.
The serial scaling hypothesis. These results align with the emerging serial scaling hypothesis 268). which posits that many challenging problems in machine learning, particularly those involving reasoning, planning, sequential decision maaiin.. physical simulation, are "inherently serial." Such problems possess computational dependencies that resist parallelization, meaning that simply scaling parallel compute (e.g., model width) is insufficient.
Therefore, progress fundamentally requires scaling serial computation: increasing the number of sequential processing steps (e.g., recurrent iterations or model depth). This perspective suggests the benefits of iterative methods might stem not just from mimicking biological processes, but from fulfilling fundamental computational requirement for certain task classes.
69

---

Re-evaluating the computational trade-off. Our runtime analysis (appendix C.7 challenges the assumption that iterative inference is prohibitively slow test time. Iterative models reached amortized-level performance within comparable wall-clock times (especially with large-batch paral- lelization), then continued refining to achieve substantially better reconstruction-spariit trade-offs. The primary cost is training time T 2x longer than amortized models (appendix (.1.4) which is understandable given their increased effective depth, interpretable as training deeper ResNets with shared parameters (Fig. 5). This suggests that iterative models are particularly well-suited for applications where inference quality and generalization are central, and the one-time cost of longer training is acceptable.
Despite these encouraging runtime results, the general efficiency of iterative methods remains an active area of research. The growing adoption of iterative techniques in large-scale commercial models has spurred optimization efforts. Recent advances in hardware acceleration [269] 270].
specialized architectures for recurrent computation [114] 271]. and algorithmic improvements for iterative processes [222] 273] suggest that the efficiency gap may narrow in the coming years.
The fundamental question remains whether the generalization advantages of iterative inference, increasingly supported by both empirical results and theoretical arguments, will continue to justify any remaining computational costs as both approaches evolve.
D.5.2 Biologically plausible learning rules
In this paper, we presented prescriptive framework for deriving inference dynamics from first principles, but we did not address this question for learning the generative model. In other words, our work is prescriptive about út but not 0.
We optimized model parameters through a procedure that can be interpreted as backpropagation through time 93 94). This is not biologically plausible, because it still suffers from the weight transport problem [274). It's even worse for temporal credit assignment because neurons lack a mechanism to store activation histories needed for precise backward passes through time (275) 276)
Other models, like predictive coding (PC), are prescriptive about both inference and learning aspects. Specifically, PC learning is effectively Hebbian [33 277]. Biologically plausible synaptic learning rules exist for sparse coding as well. For instance, Zylberberg et al. [123) introduced SAILnet, a spiking network that performs sparse coding using only synaptically local information. Interestingly, the distribution of inhibitory connections learned by SAlL.ett is lognormal, matching experimental
observations in cortex (278).
In general, we consider inference dynamics as more fundamental than learning, for several reasons. First, inference is far more experimentally accessible than synaptic weight dynamics. We can record from large populations of neurons while animals receive sensory stimulation, but we lack complete electron microscopy reconstructions of synapses before, during, and after learning. Second, what we call "learning" in machine learning likely corresponds to a complex mixture of cvolution, development, and experience-dependenn plasticity in biology [[43). Given this complexity. it is reasonable to achieve learning in neural networks using powerful but biologically implausible methods such backpropagation 281]
With that said, future work should still explore biologically plausible prescriptive frameworks for learning synaptic weights. potentially building on emerging theories like eligibility traces (282..
feedback alignment [88 284). and e-prop (275).
A natural extension: combining BONG and FOND for continual learning. Beyond biological plausibility, an exciting direction is to combine BONG with FOND. As we discussed in appendix D.I BONG and FOND are complementary specializations of BLR. Using the same framing adopted here. BONG can be viewed prescribes inference dynamics, ú. Since both frameworks minimize the same free energy objective, their combination is mathematically natural.
prescriptive framework for deriving weight dynamics, 0, while FOND
This BONG/FOND fusion would yield a fully Bayesian, principled algorithm that is prescriptive about both learning and inference: a continually learning, online Bayesian system that adapts both its beliefs and its generative model simultaneously- catastrophic forgetting. We consider this unified framework promising direction for future work.
enabling lifelong learning without
70

---

D.5.3 Predictive dynamics, with application to non-stationary sequences
Overwhelming evidence, ranging from retina [287] to cortex [288] J. suggests that biological circuits are engaged in predictive computations, anticipating the future [289]. rather than merely adapting to the past. However, our current framework lacks a concrete mechanism for such anticipatory computation. While we emphasized the importance of online inference for modeling biological perception, we left a crucial element unaddressed: online inference is most meaningful in the context of non-stationary inputs. Our work, however, explored only the simplest type of sequence: stationary imputs with the same sample repeated multiple times.
Brains are prediction machines [146] 285 286
Future extensions should incorporate genuinely predictive updates that evolve the current posterior into a predictive prior distribution, rather than relying on the identity mapping we used in this work.
Several promising approaches could guide this development. In machine learning, Marain et al. 124 introduced Bayesian online learning in sta-siaaionary gnvironments (BONE), a unifying framework for probabilistic online learning in changing environments. Within neuroscience, Jiang and Rao [290) introduced dynamic predictive coding, which higher-level neurons modulate the transition dynamics of lower-level ones, which is conceptually similar to HyperNetworks (29T). Millidge al. [292] introduced temporal predictive coding networks, where recurrent connections are used to predict the network's 's own future activity, allowing it to approximate Kalman filter with biologically plausible, local learning rules. Another notable contribution is the polar prediction model proposed by Fiquet and Simoncelli (293). Inspired by the temporal "straightening" hypothesis (294). the model performs linear extrapolation of phase in learned complex- copllex-valued latent space, enabling next-frame prediction with strong performance, interpretability, and computational simplicity.
We believe predicting the future and adapting the past represent two distinct yet complementary computations employed by biological systems. When modeling these processes, both should be clearly motivated and, ideally, derived from first principles. In this work, we addressed adaptation but left prediction largely unexplored. We aim to develop the missing predictive component in future work, while evaluating the resulting architectures on non-stationary datasets such as natural videos.
D.5.4 Hierarchical extensions with realistic synaptic transmission delays
Another dimension of being preseriptive lies in architectural design. In this paper, focused on single-layer models and did not provide prescriptions for multi-layer, hierarchical architectures (5)
295]. Yet this is an important direction, given growing evidence that hierarchical VAEs exhibit
cortex-like organization [125] 126). Future extensions should address two key components [296]: (1)
the structure of the generative model, and (2) information flow during inference.
For the generative model, a fundamental decision concerns the probabilistic dependencies between latent variables across different layers. Should the joint distribution be structured in Markovian chain, where each layer interacts only with its immediate neighbors? Or might a more flexible heterarchical structure (297-299 with arbitrary graph topology [300] better capture the complex causal structure of natural signals?
For information flow during inference, several questions arise: what information propagates between layers, and in which directions [30T]? Each choice leads different computational trade-offs and makes distinct predictions about neural dynamics. For example, predictive coding typically propagates prediction errors upward the hierarchy. Crucially, biological systems do not transmit raw signals between hierarchical levels, but instead apply various forms of pooling (302) or coarse -aariiiggiinng [255 operations that transform lover-level representations into increasingly abstract features at higher layers. This pooling serves both computational efficiency and representational purposes, enabling the extraction of invariant features while reducing dimensionality [303]. In the future, our framework could be extended to derive normative pooling operations through free energy minimization under appropriate architectural constraints, potentially explaining why the visual system employs specific receptive field pooling strategies rather than alternatives.
Finally, realistic neural models should account for finite signal propagation delays in biological axons (304]306]. In future work, we plan to incorporate such transmission delays into our framework to enhance biological realism and potentially uncover computational mechanisms uniquely enabled by temporally structured information flow (377.
71

---

D.5.5 Applications to explain neural data variance
The ultimate test of model is its ability to explain empirical data. iP- demonstrated several emergent behaviors that mimic cortical properties at phenomenologica level ( and [T). but this is only first step toward rigorous validation against neural data. A comprehensive assessment would require testing our model's capacity to explain neural response variance [308] 309 for
example, in frameworks like Brain-Score [127].
Our philosophical approach differs from purely descriptive models that are directly optimized fit neural responses [128] [10]. Instead, we advocate for normative models that are validated against neural data, rather than trained explicitly to reproduce it, line with critiques of purely descriptive approaches [31T]. This approach offers greater interpretability. as the model's 's behavior stems from principled objectives (e.g.. free energy minimization) rather than parameter fitting (312). From this point of view, there are two promising paths forward.
First, we can systematically explore different prescriptive choices within our framework identify which configurations best explain neural data. For instance, comparing how alternative inference dynamics distributional assumptions affect the model's ability predict neural responses novel stimuli. Second, we might consider semi-supervised training approach (313), where free energy minimization is augmented with a neural regularization term that encourages alignment between iP-VAE spike patterns and recorded neural activity in response to identical stimuli.
The spiking nature of fP-VAE makes it particularly appropriate for comparison with neural recordings, offering closer parallels to biological data than non-spiking alternatives. Beyond explaining variance, iP-VAE's E spiking representation also enables in silico experiments that can potentially inform future in vivo studies [128 314). Finally, this compatibility with spike train data creates an opportunity for rigorous comparisons between variational inference approaches like and sampling-based spiking
models [118 119 245 247 249 252 305) (appendix D.4). in terms of their ability to account for
empirical neural responses.
In the future, we are excited to test fully predictive, hierarchical version of iP- against neural data to put our framework to the ultimate test. Such experiments could validate the theory and help identify which aspects of neural computation are most effectively captured by variational principles.
72

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: In section we show that online natural gradient descent on F, under Poisson assumptions, leads to a recurrent spiking neural network that performs variational inference via membrane potential dynamics. In the same section also show its desirable theoretical features. In section 5 and appendix present the empirical results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We briefly overview limitations in section and then substantially expand upon
them in appendix D.5 Specifically, we discuss limitations concerning iterative inference
(appendix DD5. T) learning rules (appendix(D.5. DD5.2), sequences (appendix (.5.3). architecture
(appendix D.5.4 and applications to neural data (appendix D)5.5)
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model vell-specifiiattio, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
73

---

3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: Our theory includes derivations from first principles, with the full set of assumptions and details provided in section and appendix
Guidelines:
The answer NA means that the paper does include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Details about architectures, hyperparameters, training, and experiment provided section and appendix (specifically appendix C.I
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
also be provided via detailed
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility.
74

---

In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification:
Our code, data, and model checkpoints are available at:
http://ggthbb..oommmaaaaaa/a///t
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
Soaeeeess/ooSSSSmmeeiaa SonedeSubmissiooPPollo ssionPolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Most details needed for understanding the train and test details are provided in section S and appendix The code will also be provided for full details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: For the main experiments shown in Fig. 4 report variability with respect to hyperparameters using box plots (showing median, interquartile range, minimum, and maximum). For our only numerical result in Table 3 report 99% confidence intervals estimated from 5 random initializations, assuming the sample means follow a t-distribution by the central limit theorem.
Guidelines:
75

---

The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi-
dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative етror rates).
If error bars are reported in tables or plots. The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We specify the hardware and compute usage in appendix C.I.4 Overall our experiments were very light on compute and are accessible to academic research.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
Our research conforms to the NeurlPS code of ethics every aspect.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [NA]
76

---

Justification: Our work aims to provide a theoretical understanding of how brain-like inference models can be derived from first principles. The experiments are conducted on small-scale, controlled settings without practical deployment. As such, we do not foresee immediate societal impact, either positive or negative, or potential for malicious applications.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deploymentt of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Our work is primarily theoretical and involves small-scale experiments using publicly available datasets. It does not involve the release models or data with high risk of misuse, so this question is not applicable.
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We credit and cite the creators of the code libraries, models, and datasets used in the paper. We specify licenses in appendix C
77

---

Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. com camdsataeets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We provide code for reproducibility, which contains documentation on how to replicate experiments. We will also release code notebooks with more leamiig-orrented documentation.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/cood/mmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve research with human subjects or crowdsourcing.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
78

---

Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not non non- stanstandard components.
involve LLMs as any important, original,
Please refer to our LLM policy for what should or should not be described.
79


Reviews:
[{"review_id": "FL1mI51jBR", "reviewer": "Reviewer_iEYy", "summary": "This paper introduces FOND (Free-energy Online Natural Gradient Dynamics), a framework for deriving inference procedures as brain-inspired dynamics from online natural gradient descent on variational free energy. Implementing FOND with Poisson latent variables yields the iterative Poisson VAE (iP-VAE), a recurrent spiking network whose dynamics implement iterative variational Bayesian inference. Empirical evaluation demonstrates that iP-VAE achieves superior reconstruction–sparsity trade-offs compared to amortized VAEs and standard sparse/predictive coding baselines on small-scale benchmarks, with robust out-of-distribution performance.", "questions": "- How sensitive are the reported performance metrics (reconstruction error and sparsity) to the dimensionality of the latent variable space (K)?\n\n- Would replacing natural gradient updates with standard gradient descent significantly affect the observed performance?", "limitations": "The authors give an adequate explanation of the works limitation in appendix D.5. A concise overview of these points should be brought into the main paper for a final version.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n- The paper is technically sound and will have broad appeal to researchers across machine learning and theoretical neuroscience.\n\n- The work successfully unifies multiple influential frameworks (variational inference, sparse coding, natural gradient descent) within a coherent theoretical framework, which is further used to derive novel architectures.\n\n- While the empirical results presented in the main paper are limited in scope (i.e., very simple datasets), the results regarding sparsity and OOD performance are compelling.\n\n- The conceptual integration of online natural gradient descent, variational inference, and biologically plausible neural dynamics is insightful and (to my knowledge) novel.\n    \n\nWeaknesses:\n\n- While the writing is clear, the structure of the manuscript could be improved for readability. The main text frequently cross-references an extensive appendix, splitting core methodological details across sections and appendices.\n\n- A quantitative comparison of runtime (training and inference) between the iterative and amortized algorithms is lacking.\n\n- A concise algorithmic description or pseudocode detailing the iterative algorithm steps would significantly enhance clarity.", "was_revised": true, "final_justification": "I thank the authors for appropriately addressing my points. I believe this is a good contribution and thus will increase my score and recommend acceptance.", "rebuttal": "We sincerely thank Reviewer iEYy for their constructive feedback and encouraging evaluation. We appreciate that they found our work to be a novel and insightful unification of key frameworks, with broad appeal and compelling empirical results (although \"limited in scope\"). \n\nBelow, we address the scope, other concerns and questions, and the comment regarding limitations.\n\n---\n---\n---\n\n> While the writing is clear, the structure of the manuscript could be improved for readability. The main text frequently cross-references an extensive appendix, splitting core methodological details across sections and appendices.\n\nWe will restructure the manuscript and make the contents of the main paper more coherent and self-contained. Specifically, given the additional 1 page, we plan to bring all the necessary details for deriving the main theoretical result into the main paper. This should further enhance the clarity of the core derivations and methodological details.\n\n---\n---\n--- \n\n> A quantitative comparison of runtime (training and inference) between the iterative and amortized algorithms is lacking.\n\nWe agree with the reviewer. A significant challenge with iterative methods is their slower training and inference time, and it is important to provide a quantitative measure of runtime.\n\n#  Runtime comparison between iterative and amortized VAEs\n\nHere, we will focus only on comparing iP-VAE and P-VAE runtimes. For the final revised manuscript, we will also explore the Gaussian and Gaussian + ReLU models.\n\n## Training time comparison\n\nWe partially address this in **Appendix C.1.4 (Training compute details)**, lines 1894-1898:\n\n*\"The computational requirements for iterative VAEs scale with the number of inference iterations ($T_\\text{train}$). For example, training the models presented in Fig. 2 with $T_\\text{train} = 16$ requires approximately three hours per model on an NVIDIA A6000 GPU. Memory usage and training time both scale approximately linearly with $T_\\text{train}$, as each additional inference iteration requires maintaining the computational graph for backpropagation.\"*\n\nBut as the reviewer pointed out, we do not compare to the amortized counterparts. To address this, we examined the P-VAE counterpart which achieves the best overall performance in Figure 3. This model uses a five-layer ResNet (lines 1864-1865), but is otherwise identical to the aforementioned iP-VAE.\n\nThis amortized P-VAE takes ~1.5 hours to train (versus ~3h for iP-VAE).\n\nTherefore, at least for the van Hateren dataset, the amortized training time is roughly 2x faster. We expect this gap to widen with (1) increased $T_\\text{train}$, and (2) dataset scale. For the final paper, we will explore this question more thoroughly and quantify relative training times as a function of various hyperparameters and datasets.\n\n## Inference time comparison:\n\nTo quantify this, we looked at how many test-time iterations does it take for iP-VAE to outperform its P-VAE counterpart, in terms of both sparsity and reconstruction scores. To this end, we used the distance from the ideal gold star ($R^2$ = 1, Sparsity = 1; introduced in Figure 3).\n\nWe found that at only 13 iterations, iP-VAE surpasses the performance of the amortized P-VAE.\n\nIn conclusion, while iterative inference is slower, it achieves superior performance in a relatively small number of steps. For the final paper, we plan to perform more comprehensive comparisons, calculate actual inference runtimes in seconds, and include them in a dedicated appendix section.\n\n---\n---\n---\n\n> A concise algorithmic description or pseudocode detailing the iterative algorithm steps would significantly enhance clarity.\n\nWe agree. For the revised manuscript, we plan to add both an algorithm box and a graphical model to clearly and unambiguously describe the full iterative algorithm.\n\n---\n---\n---\n\n\n> How sensitive are the reported performance metrics (reconstruction error and sparsity) to the dimensionality of the latent variable space (K)?\n\nThank you for this suggestion. We performed additional experiments, exploring latent dimensionalities $K \\in \\[32, 64, 128, 256, 512, 768, 1024, 2048\\]$. We report $R^2$, sparsity (i.e., portion_zeros), and the overall performance as measured by the Euclidean distance from the gold star (i.e., $R^2$ = 1, Sparsity = 1; introduced in Figure 3). We also report the average total number of active neurons, defined as K * (1 - portion_zeros).\n\nHere are the results:\n\n| Latent dim (K) | $R^2$ ↑ | Portion zeros ↑ | Overall perf. ↓ | Avg. num. active neurons |\n|---------|------|---------------|---------------------|--------------|\n| 32 | 0.24 | 0.28 | 1.05 | 23.2 |\n| 64 | 0.42 | 0.28 | 0.93 | 46.4 |\n| 128 | 0.64 | 0.32 | 0.77 | 87.4 |\n| 256 | 0.77 | 0.55 | 0.51 | 115.6 |\n| 512 | 0.83 | 0.77 | 0.28 | 116.1 |\n| 768 | 0.83 | 0.86 | 0.22 | 110.4 |\n| 1024 | 0.83 | 0.90 | 0.20 | 106.9 |\n| 2048 | 0.84 | 0.95 | 0.17 | 103.1 |\n\nWe observe two key results:\n\n- First, the overall performance consistently improves as the dictionary becomes more overcomplete, corroborating a known desirable feature of sparse coding models.\n- Second, and more surprisingly, the total number of active neurons plateaus around ~110, even as the latent space grows to 2048 dimensions. This suggests the model learns an efficient code that uses a consistent budget of active neurons, regardless of the available capacity.\n\nWe thank the reviewer for this excellent question, which revealed these new results. We will include a full analysis in the appendix of the final paper.\n\n---\n---\n---\n\n\n> Would replacing natural gradient updates with standard gradient descent significantly affect the observed performance?\n\nYes. We explored this in our early experiments and found that standard gradient descent updates, which do not account for the geometry of the parameter space, are unstable. Without the Fisher preconditioning provided by natural gradients, the updates are poorly scaled. This caused the inference dynamics to quickly diverge.\n\nWe will add a sentence stating this in the revised manuscript to further highlight the importance of natural gradients for our framework.\n\n---\n---\n---\n\n\n> The authors give an adequate explanation of the works limitation in appendix D.5. A concise overview of these points should be brought into the main paper for a final version.\n\nWe agree. With the additional 1 page for the final version, we plan to more clearly and explicitly discuss limitations."}, {"review_id": "898XcV2bFR", "reviewer": "Reviewer_UGdd", "summary": "The authors introduce a family of iterative variational autoencoders (VAEs) inspired by Bayesian theories of probabilistic inference in the brain and biological plausibility. Their broader goal is to propose a prescriptive framework called Free energy Online Natural-gradient Dynamics (FOND) for specifying and learning latent variable models based on variational inference that model the brain’s perceptual inference processes.\n\nTheir main model — the iterative Poisson VAE (iP-VAE) — uses a prior and posterior to model the latent variable hierarchically, mimicking neural activity with components such as continuous membrane potential and firing rate that includes temporal dynamics, and a Gaussian likelihood to model how the latents generate observed images. They also include versions with Gaussian priors and posteriors with and without an additional nonlinearity. \n\nAll their models are trained using natural gradient descent to maximize the variational lower bound (ELBO) of images.\n\nThey evaluate performance on the van Heteren and MNIST datasets showing improvements in reconstruction error, learned latent sparsity, out-of-distribution generalization compared to existing amortized VAEs.", "questions": "1. Model details: The authors use $x$ to denote observations and $z$ for latent variables. They introduce \"dynamic variables\" $u$ and $r$ (Lines 167, 168) and use them in subsequent equations (including Eq 3), but they do not explicitly state the mathematical relationships between $x$, $z$, $u$, and $r$. The manuscript would benefit from clearly defining these relationships upfront in the main text, preferably accompanied by a graphical model, before proceeding with their usage. The main text would also benefit from stating the exact forms of the posterior and prior (just the way Gaussian-linear likelihood is stated, but the prior and posterior are pushed to the appendix). Given that the models being proposed are essentially VAEs with specific choices of prior, posterior and likelihood, stating the above quantities upfront makes the necessary equations (ELBO, gradients) follow much more naturally. Also from the appendix, I do not understand the assumed posterior well. It is stated as $Pois(z; r)$, but shouldn’t it be conditioned on $x$? I would like to see the function that relates $x$ to $z$ in the posterior. Including these details upfront would greatly improve the flow of the paper and clarity.\n2. Line 335: what are PCNs? I do not think this is introduced in the main text.\n3. Re Figure 3, panel b: is this a reasonable metric? The gold-star — although I understand at first glance — is not only impossible but also ill-defined (no latent variables are ever active?). \n4. Re evaluation: a more rigorous metric than reconstruction error would be to compare the log-likelihood (ELBO) of images under the given models. It would be informative to see that benchmarked against baselines. Mere reconstruction error only gives us half the picture.", "limitations": "The authors mention their limitations bundled with future work in a single line: \"[L356] Future work should explore acceleration techniques for iterative inference [106], biologically plausible learning rules [115], predictive dynamics for non-stationary sequences [116], hierarchical extensions [117, 118], and neural data applications [119, 120].\" \n\nThis is too brief and doesn't sincerely inform the reader critically of the limitations--of either FOND or the developed VAEs--such as strong assumptions made, scope and computational efficiency. I understand there exists a detailed appendix, but I would strongly encourage the authors to substantiate it in the main text. \n\nI would particularly encourage the authors to substantiate limitations of natural gradient descent and iterative updates as opposed to amortized (these two aspects focusing on FOND), and why these are \"prescriptions\" for more \"brain-like\" architecture; the choices of prior, posterior and likelihood of their proposed models; and perhaps most critically, reflect on their choice of evaluation (please see my point on **Impact** under weaknesses).", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "## Strengths\n- The paper boasts a **strong technical foundation**, building on variational methods in modern probabilistic machine learning and the free energy principle (FEP) of theoretical neuroscience. It also aims to connect several approaches, traditionally siloed, of theoretical neuroscience such as FEP, predictive coding, sparse coding and potentially neural sampling and reviews diverse set of works well.\n- The specification of models (choices of variables, learning algorithms and derivations thereof) are **principled** from a probabilistic machine learning perspective, and their core idea of biologically motivating the structure of the latent space is conceptually appealing.\n- The paper is **well-written**.\n\n## Weaknesses\n\n- **Unclear significance of FOND**: I’m struggling to see substantial novelty or contribution in the proposed prescriptive framework, FOND. As presented, FOND primarily prescribes latent variables that are inspired by the brain’s neural activity—such as time varying firing rates and membrane potentials—and employs natural gradient descent to optimize the ELBO. From a machine learning or statistical standpoint, this appears to be a standard application of variational inference tailored to a specific setting or just latent variable modeling in general. It’s unclear whether this warrants being framed as a new “prescriptive framework.” In my view, the work would be better positioned as introducing a novel, brain-inspired family of VAEs, rather than proposing a new framework. Additionally, it's not clear what exactly is being prescribed and why: the latent variables and their distributions? Online updates? Natural gradient descent? Perhaps most crucially, FOND seems to not prescribe how to evaluate the supposed brain-like models. I would appreciate clarification from the authors regarding their intended contribution and how FOND offers distinctive value. I’m open to correction if I have misunderstood or missed a key aspect.\n- **Unclear novelty of proposed models**: While the models are well-motivated and principled, they come across as incremental extensions with only modestly novel components. Beyond their iterative inference structure, it’s unclear how these models substantially differ from prior work—for example, the Poisson-VAE by Vafaii et al. (2024). The proposed models may offer added expressiveness or biological plausibility (e.g., resembling spike counts and membrane potential to some extent), but the manuscript would benefit from a clearer articulation of what is substantively new and how these aspects advance existing approaches.\n- **Limited impact of proposed models and insufficiently compelling evaluation**: the main text primarily focuses on evaluating the models on image reconstruction error and sparsity (figure 2 and 3) on MNIST and van Hateren and compare the models to amortized baselines. Are these (1) strong enough evaluation criteria and (2) strong enough baselines? Given that the main contribution centers on creating models with brain-like dynamics and biological plausibility, evaluations on real neuroscience data are strongly warranted. In fact, I would have expected their purported framework to include this as a \"prescription\". In that case, the corresponding baselines should also include competitive neural predictive models, at least to note the gaps. Alternatively, if the models are intended as normative, what neural phenomena do they explain that remain difficult for existing models to capture, and how are those insights attributable to the prescriptions of FOND or specific choices of the generative model (beyond commonly observed effects such as the emergence of Gabor-like features from the Gaussian likelihood). Setting aside biological relevance, it's unclear why a rigorous metric such as log-likelihood (ELBO) of images is also left out. Seeing it as an image generative model, it's unclear how it performs compared to state of the art models such as diffusion models. I see this overall as the primary weakness of the work.\n- **Limited choice of datasets**: Given that the proposed models build on modern probabilistic techniques, not having evaluations on more challenging and larger benchmarks such as CIFAR or ImageNet raises concerns over computational efficiency and scalability.\n\n## Overall assessment\n\nEdit: note that this assessment was written prior to the rebuttal phase. Please see below for final justification.\n\nThis paper presents a principled, biologically motivated VAE family, but its impact is limited by the unclear significance and originality of their proposed \"FOND\" framework, modest model novelty, and most crucially, insufficient empirical evaluation to support the paper's claims.\n\n**To raise my scores**, I encourage authors (where feasible) to **(1)** more clearly articulate the originality and distinctive value of the FOND framework and how their developed VAE models substantially advance prior works, **(2)** expand it to include tasks grounded in real neural data to support biological and experimental relevance, **(3)** train the VAEs on more modern datasets, and evaluating using rigorous metrics such as log-likelihood (ELBO), or provide a clear justification for their current choices, **(4)** improve the clarity in terms of defining components of the generative models upfront, and lastly **(5)** discuss limitations.", "was_revised": true, "final_justification": "Among the raised issues, the authors have (1) addressed scaling to modern datasets and shown that their results hold, and (2) substantially clarified the motivation behind FOND, agreeing to reframe it as a promising brain-inspired approach rather than a definitive prescription.\n\nHowever, the central issue remains. While FOND is well-motivated from reasonable first principles, the work does not provide rigorous enough evaluation to substantiate the claim that its derived models are truly “brain-like”. The only metrics reported are image reconstruction error and latent sparsity, compared to simpler versions of the generative models. These evaluations do not convincingly evaluate brain-like properties or biological plausibility. Evaluation on neuroscience datasets, showing that FOND-derived models reproduce or predict neural phenomena concretely is warranted. \n\nWhile the approach has promise, this has yet to be demonstrated conclusively. I therefore maintain my score and encourage resubmission with broader and more targeted neural evaluations, alongside the updated positioning and careful reframing of FOND in light of the rebuttal discussions.", "rebuttal": "We sincerely thank Reviewer UGdd for their constructive, thorough, and well-thought-out feedback, along with their clear directions to improve. We appreciate that they pointed out the paper's strengths, including its \"strong technical foundation,\" principled and \"conceptually appealing\" model design, and that it is \"well written.\"\n\n---\n---\n---\n\n# Scope clarification \n\n>  \"In my view, the work would be better positioned as introducing a novel, brain-inspired family of VAEs, rather than proposing a new framework.\"\n\nWe agree that two distinct contributions are in play:\n\n- **(i)** Introducing a prescriptive framework (FOND) for deriving brain-like inference dynamics from first principles, and\n- **(ii)** Applying FOND to derive a family of concrete, brain-inspired VAEs.\n\nWe also share the reviewer’s enthusiasm for evaluating the brain-inspired models on real neural data to demonstrate biological relevance. However, the primary goal of this manuscript is to establish **FOND**, a rigorous theoretical framework for deriving such biologically interpretable neural inference dynamics from first principles. \n\n---\n---\n---\n\nWe appreciate that the reviewer provided us with a clear roadmap with 5 concrete action items for improvement. We believe we can address four of them quite well, but the point about neural data will have to wait for a future submission.\n\n## Item (1): Originality of FOND\n\nFOND is not a new mathematical rule, but a **recipe** for how to derive and interpret **brain-like dynamical systems** from first principles. To clarify the utility of such a framework, we plan to add a new section to the revised manuscript.\n\nWe provide a sketch of this planned section in our rebuttal to Reviewer ```fucC```. (We apologize for redirecting your attention to the other rebuttal - this should have been a global rebuttal response, but those are not allowed this year.)\n\n> it's not clear what exactly is being prescribed and why\n\nFOND simply makes explicit what choices you have when building a model based on Free Energy ($\\mathcal{F}$) minimization. You must choose your likelihood and approximate posterior, and how to parameterize it. From that, the **dynamics** of the variational parameters are fully determined as natural gradient descent on $\\mathcal{F}$. We show how different models result from different choices, and that new models can be derived. We are actively using this framework to make more capable brain-inspired VAEs that go beyond iP-VAE.\n\n## Item (2): Applications to real neural data\n\n> … support biological and experimental relevance … evaluations on real neuroscience data … what neural phenomena do they explain that remain difficult for existing models to capture\n\nIn the current work, we tested contrast-dependent response latency and found that iP-VAE exhibits cortex-like dynamics (Appendix C.3, Figure 8). But what about other neural phenomena?\n\n[Zhu & Rozell (2013)](https://doi.org/10.1371/journal.pcbi.1003191) demonstrated that LCA sparse coding exhibits a variety of neural phenomena, such as end-stopping, surround suppression, cross-orientation suppression, and so on. Given the strong theoretical and empirical similarities between LCA and iP-VAE, we believe iP-VAE would likely reproduce most of the neural phenomena exhibited by LCA. We will include additional evaluations of phenomenology in the final manuscript. The key advantage is that iP-VAE achieves this with stochastic, spiking neurons, whereas LCA is deterministic and continuous.\n\nMoreover, we are currently pursuing direct comparisons to neural data, such as anchoring model latents with real neural data during training, or finding correlations between real neurons and extensions of iP-VAE neurons (similar to [Vafaii et al., 2023](https://openreview.net/forum?id=1wOkHN9JK8)). We do outline this direction in Appendix D.5.5 (Applications to explain neural data variance). While this is an exciting next step, it is beyond the scope of our current contribution.\n\nOverall, the present work establishes FOND, which offers a clear recipe to derive brain-like inference algorithms that go beyond this, including architectures with forward-predictive and hierarchical components, that we will evaluate against real neural data.\n\n## Item (3a): Training the VAEs on more modern datasets\n\n> … not having evaluations on more challenging and larger benchmarks such as CIFAR or ImageNet raises concerns over computational efficiency and scalability\n\nThis concern was shared by several reviewers. We believe that addressing this limitation will substantially improve the final paper. To this end, we performed experiments on three color-image datasets: ```CIFAR-10``` (3 x 32 x 32), ```tiny ImageNet``` (3 x 32 x 32), and cropped ```CelebA``` (3 x 128 x 128).\n\nThe results, presented with tables in our response to Reviewer ```fucC```, confirm that iP-VAE readily scales up to more complex datasets, and consistently outperforms its amortized counterpart on ```CelebA```. (Once again, apologies for redirecting your attention to the other rebuttal - this should have been a global rebuttal response, but those are not allowed this year.)\n\n## Item (3b): Why not report log-likelihood\n\n> … it's unclear why a rigorous metric such as log-likelihood (ELBO) of images is also left out … Mere reconstruction error only gives us half the picture\n\nWe thank the reviewer for this important question. Our decision to focus on the reconstruction-sparsity trade-off (Figure 3) instead of a single log-likelihood value was a principled one, motivated by two main factors.\n\n**1. A more informative, disaggregated metric:**\n\nFirst, we follow the recommendation of [Alemi et al. (2018)](https://arxiv.org/pdf/1711.00464) in *Fixing a Broken ELBO*, who \"strongly encourage future work to report rate and distortion values independently, rather than just reporting the log likelihood.\"\n\nOur $R^2$-sparsity plot is designed in this spirit. Reconstruction $R^2$ serves as a proxy for inverse distortion, while sparsity serves as a proxy for the inverse coding rate. This connection is grounded in the theory of Poisson models, where the KL-divergence (rate) term acts as a penalty that directly promotes sparsity (see Figure 10 for an empirical demonstration). This disaggregated view provides deeper insight into model behavior than a single ELBO score and also connects to practical benefits like hardware efficiency.\n\n**2. The challenge of comparing online vs. static ELBO:**\n\nSecond, reporting a single, comparable log-likelihood for our online model is non-trivial. Unlike static models with one ELBO, our iterative approach generates a sequence of ELBOs, one for each time step (eq. 54; Figure 7). Averaging these is not directly comparable to the ELBO of a static model, making a fair benchmark against standard VAEs challenging.\n\n**Action plan:**\n\nThat said, we agree that providing this data is valuable for completeness. In the revised paper, we will add an appendix section containing (1) time-averaged log-likelihood scores and (2) plots of the full time-dependent log-likelihood curves for our main models. This will provide a more complete picture, as requested.\n\n\n## Item (4): Improving clarity\n\n> improve the clarity in terms of defining components of the generative models upfront\n\nWe will clearly define the full set of generative model components upfront.\n\n## Item (5): Discussing limitations\n\n> … too brief and doesn't sincerely inform the reader critically of the limitations … I would strongly encourage the authors to substantiate it in the main text … limitations of natural gradient descent and iterative updates as opposed to amortized … why these are \"prescriptions\" for more \"brain-like\" architecture …  most critically, reflect on their choice of evaluation\n\nWe thank the reviewer for this detailed feedback. We agree that the limitations section should be more substantial, and will make the following changes in the final paper:\n\n\n- We will use the additional page to bring more of the critical discussion of limitations from Appendix D.5 into the main text.\n- Specifically, we will expand on the primary limitation of iterative inference (its computational cost versus its generalization benefits) as we discuss in Appendix D.5.1.\n- We will also add discussion clarifying in what sense our prescriptions are \"brain-like,\" focusing on their emergent dynamics and representations.\n- Finally, regarding our choice of evaluation, we will create a new appendix section detailing our principled reasoning (as outlined in our response to item (3b) above), and will refer to it from the main text.\n\n\n---\n---\n---\n\n\n> Seeing it as an image generative model, it's unclear how it performs compared to state of the art models such as diffusion models\n\nThank you for pointing this out, which allows us to clarify our main focus. We are interested in brain-like inference dynamics, and view the generative capacity of the model only in service of inference (i.e., *\"analysis-by-synthesis\"*).\n\nThat said, we agree that assessing the model's generative capacity is informative. We performed a qualitative generation experiment on MNIST and we detail the results in our response to Reviewer ```yjVk```.\n> Line 335: what are PCNs?\n\nPredictive Coding Networks. We will revise the main text to introduce them before using the acronym.\n\n> Re Figure 3, panel b: is this a reasonable metric? The gold-star — although I understand at first glance — is not only impossible but also ill-defined (no latent variables are ever active?).\n\nWe believe that the metric is reasonable in this application, as its goal is to highlight the tradeoff between sparsity (proxy for energy use on neuromorphic hardware), and reconstruction loss $R^2$ (proxy for inverse distortion).\n\nAlthough the gold star is not practically achievable in most scenarios, in high artificial constructs where the data is constant (like a single image dataset), it is possible to achieve full sparsity with perfect reconstruction loss."}, {"review_id": "fAKgY8aNV0", "reviewer": "Reviewer_yjVk", "summary": "The authors modify the (recent) Poisson variational autoencoder (P-VAE) to use iterative, rather than amortized, inference (iP-VAE).  That is, they compute the natural parameters of the Poisson recognition model with gradient descent, rather than a forward pass through a deep neural network.  The authors show that, at least in certain settings, it achieves a better trade-off between reconstruction error and sparsity than models using amortized inference.  It also learns receptive (really, projective) fields that resemble the receptive fields of V1 neurons, as in classic sparse-coding models.  Much of manuscript is devoted to situating the procedure in the context of free-energy minimization (or ELBO maximization).", "questions": "- The receptive fields for the P-VAE in Fig. 16 are much less Gabor-like than those in the original paper (ref. [50], Fig. 4).  Can the authors explain this discrepancy?\n\n - The disappearance of the recognition-model entropy from the inference procedure is justified slightly differently in the BONG paper.  Can the authors explain the relationship between the justifications?  Are they two ways of saying the same thing?\n\n - Training vs. inference:\n\t - Why use the \"Poisson reparameterization algorithm\" during training but the STE during inference?\n\t - If I have understood correctly, the free-energy gradient in Eq. 6 is used in training, even though the beta-dependent term is effectively dropped during inference.  Is that right?\n\n - One of the traditional complaints about iterative inference is that it takes too long.  Can the authors say anything about this?  Can they relate it to biologically relevant time scales?\n\n- It appears that, on the basic metric of Fig. 3 (or Fig. 9), the iP-VAE performs about the same as LCA.  Can the authors comment on what they see as the advantage of the iP-VAE?\n\n- I don't understand the authors' explanation for the amortization gap in the appendix (near line 1376).", "limitations": "See \"Strengths and Weaknesses\" above.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 2, "strengths_and_weaknesses": "The core claims appear sound, the manuscript is very clearly written, and this variation of the Poisson VAE is convincingly more biologically plausible than the original.  The contrast-dependent response latency is compelling (perhaps could have been promoted to the main text) and yet intuitive under the model.\n\n\nWeaknesses:\nMy main concern is that the contribution is not substantial enough:\n\n(1) The algorithm is more or less (just) an application of BONG [ref. 60]/the Bayesian learning rule [34] to the existing Poisson VAE.\n\n(2) This is perhaps obscured because much---really, too much---of the MS is given over to preliminaries and didactic material.  (This reviewer shares the authors' enthusiam for the free energy as a unifying principle, but it crowds out the actual contributions of the MS.)  The new material doesn't really begin till page 5, and then there are only two figures of results in the main text.\n\n(3) As the authors know, the datasets are primitive by modern standards.  This is particularly problematic because amortized inference is supposed to be most useful in the setting of large, complex datasets, which makes the comparisons arguably unfair.  Generation is not assayed at all.  The quality of the latent variables is not quite as good as the P-VAE (Table 2), although MNIST is not a very discriminative dataset.\n\n(Along these lines: the models employing amortized inference use deep neural networks and > 10x as many parameters as the iterative-inference networks [Table 2].  Is this really a good choice when the encoder is linear?)\n\n\nMinor:\n\"In machine learning, diffusion models were originally motivated by non-equilibrium statistical mechanics [29], only later understood as a form of ELBO maximization.\"  Altough the work cited [29] emphasizes Jarzynski's inequality, it also (Sec. 1.2) makes explicit the connection to ELBO maximization, including VAEs, and invokes Hinton's original Science paper on free-energy minimization.", "was_revised": true, "final_justification": "In short, I am not persuaded that the difference in emphasis between FOND and BONG (e.g., laid out in the response to Reviewer fucC) is sufficiently significant to justify publication here--even though I agree with the authors on the importance of a prescriptive framework for learning.\n\nThe authors have laid out a good plan to address many of the other shortcomings of the MS, but too much is promissory (this year's proscription on uploading a one-page pdf is unfortunate). The MS would really benefit from these changes and another round of review.", "rebuttal": "We sincerely thank Reviewer yjVk for their constructive comments and encouraging feedback. We appreciate that they found our core claims to be sound, the manuscript to be \"very clearly written,\" and iP-VAE to be \"convincingly more biologically plausible than the original.\"\n\nWe address the comments and questions below.\n\n---\n---\n---\n\n> The contrast-dependent response latency is compelling (perhaps could have been promoted to the main text)\n\nWe share the reviewer's enthusiasm, as this empirical result connects nicely to the emergent normalization in the theory (eq. 9), and the model exhibits this interesting dynamics even though it was trained only on static images. We will consider promoting this figure to the main text for the final version.\n\n---\n---\n---\n\n> (1) The algorithm is more or less (just) an application of BONG [ref. 60]/the Bayesian learning rule [34] to the existing Poisson VAE.\n\nThis is a crucial point, which was shared by others. The reviewer correctly points out that our FOND framework shares its core mathematical machinery with BONG and the broader Bayesian Learning Rule (BLR) framework. Our primary contribution is not a new mathematical rule, but a new **recipe** for how this machinery can be used to derive and interpret **brain-like dynamical systems** from first principles.\n\nTo address this, we plan to add a new section to the revised manuscript, dedicated to clarifying this relationship. We provide a sketch of this planned section in our rebuttal to Reviewer ```fucC```. (We apologize for redirecting your attention to the other rebuttal - this should have been a global rebuttal response, but those are not allowed this year.)\n\n---\n---\n---\n\n> (2) …too much of the MS is given over to preliminaries and didactic material … The new material doesn't really begin till page 5 …\n\nWe agree that the current balance can be improved. Our goal was to provide a synthesis connecting ML and neuroscience concepts, but we see that it delayed the introduction of our novel contributions. In the revised version, we will streamline the background, move more didactic material to the appendix, and present our FOND framework and the empirical results much earlier in the paper.\n\n---\n---\n---\n\n> (3) As the authors know, the datasets are primitive by modern standards. This is particularly problematic because amortized inference is supposed to be most useful in the setting of large, complex datasets, which makes the comparisons arguably unfair. \n\nWe thank the reviewer for raising this concern, which was shared by several other reviewers. We believe that addressing this limitation will substantially improve the final paper. To this end, we performed experiments on three color-image datasets: ```CIFAR-10``` (3 x 32 x 32), ```tiny ImageNet``` (3 x 32 x 32), and cropped ```CelebA``` (3 x 128 x 128).\n\nWe report these results in our rebuttal to Reviewer ```fucC```. (Once again, apologies for redirecting your attention to the other rebuttal - this should have been a global rebuttal response, but those are not allowed this year.)\n\n---\n---\n---\n\n> Generation is not assayed at all.\n\nThis was deliberate because the main focus of the paper and FOND is brain-like **inference** dynamics, where generation serves as a means to this end (i.e., *\"analysis-by-synthesis\"*). However, we take your point that assessing the model's generative capabilities is informative. To this end, we performed a qualitative evaluation of a <grad|conv> iP-VAE trained on MNIST ($T_\\text{train} = 16$). Our generation algorithm works as follows:\n\n- Draw samples from the learned prior\n- Pass them through the trained decoder, generate an image\n- Treat the generated image as an input sample for the next time point: perform inference on it, update the posterior\n- Draw samples from the current posterior, pass through decoder, generated the next image\n- Repeat\n\nSince we cannot include images in the rebuttal this year, we offer a verbal description of the results:\n\nThe model successfully generated coherent and recognizable digits. We observed an interesting dynamic during the iterative generation process: some initial samples appeared ambiguous (e.g., a shape resembling both a '3' and a '6'), but would resolve into a single, well-defined digit after a few more iterations. This suggests the generative process mirrors the kind of refinement and convergence we see during inference. However, we also identified a class imbalance in the generated samples; for example, the digit '8' appeared more frequently than its ~10% representation in the training set. This could be addressed in future work using more expressive hierarchical priors.\n\nOverall, these promising results confirm the model can generate realistic samples through an iterative procedure (at least on MNIST), and that its dynamics are interesting in their own right. We will include this as a supplemental fig, but we respectfully maintain that the core contribution of this paper is the derivation and analysis of the inference algorithm itself.\n\n---\n---\n---\n\n> The quality of the latent variables is not quite as good as the P-VAE (Table 2), although MNIST is not a very discriminative dataset.\n\nThe reviewer is correct that in our MNIST results (Table 2), the amortized P-VAE, with its deep convolutional encoder, achieves slightly higher downstream classification accuracy than the iterative iP-VAE.\n\nWe believe this demonstrates a classic trade-off: the powerful amortized encoder is highly effective at learning the specific features of the MNIST training set, but this comes at the cost of out-of-distribution (OOD) generalization. Our OOD experiments in the appendix support this conclusion. While P-VAE is competitive on the standard MNIST test set, its classification accuracy is lower than iP-VAE's on rotated MNIST digits (Figure 13), and drops substantially when transferring to the EMNIST dataset (Figure 14).\n\nTo provide a definitive answer on more complex datasets, we will perform the same downstream classification analysis on our newly trained ```CIFAR-10``` and ```tiny ImageNet``` models for the final manuscript. This will clarify the true quality of the latent variables on more challenging benchmarks. \n\n---\n---\n---\n\n> (Along these lines ... Is this really a good choice when the encoder is linear?)\n\nThis is an excellent point, which we addressed with new controlled experiments on ```CelebA```. We tested iterative (iP-VAE) and amortized (P-VAE) models with both deep (8 convolutional layers) and shallow (single layer) decoders. **In all matched comparisons, iP-VAE outperformed P-VAE**, confirming its advantage is **robust to decoder complexity** and not an architectural artifact. We will add a full analysis on all new datasets to the final paper.\n\n---\n---\n---\n\n> The receptive fields … are much less Gabor-like than those in the original paper ... Can the authors explain this discrepancy?\n\nOne possible explanation is a **different choice of hyperparameters**. For P-VAE, the reported receptive fields were obtained from a model trained using the standard $\\beta = 1$; however, our iP-VAE results correspond to $T_\\{train} = 16$ and $\\beta = 24$. It’s possible that a different combination of these hyperparameters would make the receptive fields more or less Gabor-like.\n\n> The disappearance of the recognition-model entropy from the inference procedure is justified slightly differently in the BONG paper ... Are they two ways of saying the same thing?\n\nAs we show in Appendix B.10, this is an inevitable consequence of performing a single update per time point. BONG frames it as a prescription, but it follows directly from the online, single-update setup. Ultimately, these are **different descriptions of the same mathematical result.**\n\n> Why use the \"Poisson reparameterization algorithm\" during training but the STE during inference?\n\nThe two methods serve different goals. We use STE to derive the inference update rule itself. We use the Poisson reparameterization algorithm for the separate task of training the decoder's weights via backpropagation through time.\n\n> If I have understood correctly, the free-energy gradient in Eq. 6 is used in training, even though the beta-dependent term is effectively dropped during inference. Is that right?\n\nThat is correct. The beta term is used for training the decoder. We also experimented with multi-step inference where the term would be active, but saw no significant performance difference, so we adopted the simpler single-step BONG prescription.\n\n> One of the traditional complaints about iterative inference is that it takes too long. Can the authors say anything about this? Can they relate it to biologically relevant time scales?\n\nWe agree and address runtime quantitatively in our response to Reviewer ```iEYy```. We also believe **slowness can be a feature, scaling with task difficulty.** For instance, the macaque visual cortex takes ~30 ms longer to recognize “challenge” images ([Kar et al., 2019](https://www.nature.com/articles/s41593-019-0392-5)).\n\nAn iterative framework has the potential to capture this time-dependent difficulty, a connection we plan to explore in future work.\n\n> It appears that, on the basic metric of Fig. 3 (or Fig. 9), the iP-VAE performs about the same as LCA. Can the authors comment on what they see as the advantage of the iP-VAE?\n\nWhile empirically similar, iP-VAE has key advantages:\n\n1. It is a **stochastic** probabilistic model, while LCA is deterministic.\n2. It uses **compressed, integer spike count representations**, unlike LCA's continuous values.\n3. It naturally extends to **deep decoders** (Figure 6, Appendix B.6).\n\nThe fact that iP-VAE performs on par with LCA despite these more constrained, biologically plausible representations is a strength.\n\n> I don't understand the authors' explanation for the amortization gap in the appendix (near line 1376).\n\nThank you for pointing this out. Our current explanation is cryptic. We will revise that section to enhance clarity."}, {"review_id": "v18jVMpmXQ", "reviewer": "Reviewer_fucC", "summary": "The paper proposes a unifying framework, Free energy Online Natural-gradient Dynamics (FOND), for deriving brain-like inference dynamics by minimizing variational free energy (F). It introduces the iterative Poisson variational autoencoder (iP-VAE), a spiking neural network whose membrane potential dynamics perform variational inference via natural gradient descent. The work claims both theoretical and empirical advances, including emergent normalization, hardware efficiency, and improved reconstruction-sparsity trade-offs compared to standard VAEs and predictive coding models.", "questions": "I liked that the authors (obviously) compared against the P-VAE, and showed superior performance. Do you believe the good results are a consequence of the iterative process, or is there something specific about the Poisson distribution? More precisely, do you believe that in general iterative methods should perform better than amortized ones, and that the gap in performance you observed in simply because of that?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Pros:\n\n1) he paper provides a clear and compelling synthesis of variational inference in machine learning (ELBO maximization) and theoretical neuroscience (free energy minimization), highlighting their mathematical equivalence and bridging the two fields. It was also a very interesting read as it connected this work well with the recent literature on BLR and Poisson VAEs. \n2) The derivation from first principles is interesting.\n\n\nCons:\n\n1) While the theoretical contributions are strong, the empirical results are somewhat limited in scope. The main experiments focus on small-scale datasets (natural image patches, MNIST). What is the limit to which this algorithm can be pushed? FashionMNIST? Colored images like CIFAR10? It would be nice to add a small discussion on the problem of scale.\n2) While the unification is well-argued, the core idea of iterative inference via natural gradients is closely related to recent work (e.g., Bayesian Learning Rule, BONG), and the paper could more clearly delineate what is fundamentally novel in FOND and iP-VAE versus extensions of these frameworks.", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank Reviewer fucC for their encouraging evaluation and constructive feedback. We are glad they found our work to be a \"clear and compelling synthesis\" that connects well with recent literature, as this was a primary goal of our manuscript.\n\nWhile the reviewer acknowledges our theoretical contributions, they rightly point to the limited scope of our initial experiments and ask for clarification on novelty. Below, we address these important points with new experimental results and a clearer framing of our contributions.\n\n---\n---\n---\n\n> While the theoretical contributions are strong, the empirical results are somewhat limited in scope. The main experiments focus on small-scale datasets (natural image patches, MNIST). What is the limit to which this algorithm can be pushed?\n\nWe thank the reviewer for raising this concern, which was shared by several other reviewers. We agree that addressing this limitation will substantially improve the final paper. To this end, we performed experiments on three color-image datasets: ```CIFAR-10``` (3 x 32 x 32), ```tiny ImageNet``` (3 x 32 x 32), and cropped ```CelebA``` (3 x 128 x 128).\n\nDue to time constraints during the rebuttal period, we performed only limited hyperparameter search, and evaluated the models in their sparsity-reconstruction performance. We plan to conduct more extensive experiments, such as classification performance using a linear probe,  and include these as supplemental results in the final paper. Below, we report results from this preliminary exploration.\n\n# Scaling up iP-VAE: experiments on more complex, colored image datasets\n\nHere, we demonstrate how iP-VAE can be readily scaled up to more complex datasets, such as faces and natural images.\n\n## ```CelebA``` (cropped to 128 x 128)\n\nWe trained the following models:\n\n1. iP-VAE with an iterative encoding algorithm and a deep decoder with 8 convolutional layers (interpreted as \"deep sparse coding,\" see Figure 6 and Appendix B.6).\n2. iP-VAE with an iterative encoding algorithm and a single layer convolutional decoder.\n3. P-VAE with a deep convolutional encoder and decoder. This is the amortized counterpart of model # 1.\n4. P-VAE with a deep convolutional encoder and single layer convolutional decoder. This is the amortized counterpart of model # 2\n\nAll models shared a latent dimensionality of 256 * 16 * 16, and the linear decoders (models # 2 and 4) had a kernel size of 38. We trained all models using $\\beta = 1$, and evaluated them in their sparsity-reconstruction performance. We also report overall performance, defined as the Euclidean distance from the ideal point (golden star; $R^2$ = 1, Sparsity = 1), where a lower score is better. Here are the results:\n\n|   | model | encoder | decoder | $R^2$ ↑ | Portion zeros ↑ | Overall performance ↓ | # params ↓ |\n|---|-------|---------|---------|-----|---------------|----------|----------|\n| 1 | iP-VAE | iterative | conv (deep) | 0.95 | 0.93 |  0.09 | 0.7 M |\n| 2 | iP-VAE | iterative | conv (lin) | 0.90 | 0.91 |  0.13 | 1.1 M |\n| 3 | P-VAE | conv | conv (deep) | 0.86 | 0.96 | 0.15 | 1.9 M |\n| 4 | P-VAE | conv | conv (lin) | 0.74 | 0.99 |  0.26 | 2.3 M |\n\n\n## ```CIFAR-10``` and ```tiny ImageNet``` (32 x 32)\n\nFor these datasets, we trained model #1:  iP-VAE with a deep decoder but now with 6 layers. The latent dimensionality was 256 * 8 * 8. We trained all models using $\\beta = 1$. Results:\n\n| Dataset | model | $R^2$ ↑ | Portion zeros ↑ | Overall performance ↓ |\n|---------|-------|-----|---------------|---------------------|\n| ```CIFAR-10``` | iP-VAE (deep conv decoder) | 0.93 | 0.97 | 0.08 |\n| ```tiny ImageNet``` | iP-VAE (deep conv decoder) | 0.95 | 0.94 | 0.08 |\n\nIn conclusion, these new results demonstrate that iP-VAE scales effectively to complex, high-dimensional color image datasets, achieving a superior reconstruction-sparsity trade-off compared to its amortized counterpart, all while using fewer parameters. We believe this substantially strengthens the empirical support for our claims and addresses the concerns about the limited scope of our initial experiments.\n\n---\n---\n---\n\n> While the unification is well-argued, the core idea of iterative inference via natural gradients is closely related to recent work (e.g., Bayesian Learning Rule, BONG), and the paper could more clearly delineate what is fundamentally novel in FOND and iP-VAE versus extensions of these frameworks.\n\nWe thank the reviewer for this crucial point, which was shared by others. The reviewer correctly points out that our FOND framework shares its core mathematical machinery with BONG and the broader Bayesian Learning Rule (BLR) framework. Our primary contribution is not a new mathematical rule, but a new **recipe** for how this machinery can be used to derive and interpret **brain-like dynamical systems** from first principles.\n\nTo address this, we will add a new section to the revised manuscript dedicated to clarifying this relationship. Below is a sketch of our planned explanation.\n\n# The relationship between BLR, BONG, and FOND\n\nWe see a clear lineage connecting these frameworks:\n\n- BLR provides the general principle for deriving learning rules from variational updates, \n- BONG applies BLR to sequential online inference with a single update per observation.\n    - The authors of BONG specifically state that: \"BONG can be seen as a special case of BLR with one update step per observation and learning rate 1\".\n- FOND applies BLR to derive the *governing dynamics* of a neural system.\n\nThe novelty of FOND lies in its explicit emphasis on **dynamics** (the \"D\"). While BONG and BLR provide a recipe for optimizing variational parameters, FOND provides a recipe for deriving the governing dynamics equations of a neural system.\n\nAs we outline in Appendix B.2, this is inspired by prescriptive theories in physics, where one derives dynamics by (1) first identifying the **dynamical variables** (e.g., membrane potentials $\\mathbf{u}$) and then (2) defining the principles that govern their evolution (i.e., **natural gradient descent on free energy**).\n\nThis prescriptive lens is what allows us to start from an abstract objective ($\\mathcal{F}$) and arrive at a concrete, interpretable neural circuit model, such as a spiking network with lateral competition (iP-VAE), or other familiar models. This focus on deriving circuit dynamics is the key differentiator. The goal of BLR/BONG is primarily to perform efficient Bayesian inference. The goal of FOND is to ask:\n\n*\"If a neural system were performing inference by minimizing free energy, what would its dynamics look like?\"*\n\nThe result is a canonical model that is both biologically plausible and empirically effective, a connection not emphasized by prior work.\n\nWe will clarify this distinction in the revised manuscript. We also plan to add a Venn diagram, illustrating that BLR is the general mathematical framework, containing both BONG and FOND as two distinct, specialized applications. BONG focuses on sequential inference, while FOND focuses on deriving biologically plausible neural dynamics.\n\nThank you for pushing us to clarify this relationship. We believe it will significantly strengthen the paper’s framing.\n\n---\n---\n---\n\n> Do you believe the good results are a consequence of the iterative process, or is there something specific about the Poisson distribution? More precisely, do you believe that in general iterative methods should perform better than amortized ones, and that the gap in performance you observed in simply because of that?\n\nWe thank the reviewer for this great question. It gets to a key point of our paper.\n\nYes, we believe the better results come from the iterative process in general, not just from using the Poisson distribution.\n\nThe results in Figure 3b show that all of the iterative models perform better than their amortized counterparts. This is true for all the models we tested.\n\nWe think this shows an important trade-off that is worth more attention. Our work provides clear evidence that iterative inference, while slower, can close the known \"amortization gap\" and produce better results. This leads to an interesting question for both AI and neuroscience: how does the brain handle this? The brain's inference needs to be both *accurate* (like iterative methods), and *fast* (like amortized ones). This suggests that a *hybrid* approach might be at play, which is an exciting direction for future work."}]

Author rebuttal:

Meta-review: This paper introduces FOND, a principled recipe for deriving brain-like inference dynamics as online natural-gradient descent on free energy, and demonstrates it via iP-VAE, a spiking recurrent network that performs iterative variational inference. The model shows appealing theoretical properties—such as emergent normalization and efficient spike-based representations—and strong empirical results, including favorable sparsity–reconstruction trade-offs, robustness to out-of-distribution inputs, and scaling to larger datasets (CelebA, CIFAR-10, tiny ImageNet). The work bridges machine learning and neuroscience by turning abstract free-energy principles into concrete neural circuit dynamics.

The main strengths lie in its conceptual clarity and unification, the empirical effectiveness of iP-VAE, and the broader relevance to both ML and neuroscience communities. Reviewers appreciated the principled derivation and the improved generalization compared to amortized baselines. The weaknesses are that FOND is framed more as a “recipe” than a fundamentally new algorithm, biological plausibility is supported only indirectly (without direct neural data evaluation), and initial experiments were limited in scope (later addressed with new scaling results).

During rebuttal, the authors provided substantial clarifications distinguishing FOND from BONG/BLR, offered first-principles justifications for online natural-gradient inference, and added runtime, sensitivity, and scalability experiments. While some concerns about originality and biological validation remain, the paper is technically solid, intellectually stimulating, and strengthened by rebuttal results. Overall, I recommend acceptance as a poster, as the contributions will be of clear interest to both machine learning and neuroscience audiences.
Decision: Accept (poster)