SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \log \det  \left(\hat {\boldsymbol {R}} _ {\mathbf {\sigma} _ {\mathbf {e}} ^ {(k + 1)}} [ t ] + \epsilon_ {k} \boldsymbol {I}\right) \\ \approx \frac {1}{\epsilon_ {k}} \sum_ {i = 1} ^ {t} \lambda_ {\mathbf {r}} ^ {t - i} \| \mathbf {r} ^ {(k + 1)} [ i ] - \mathbf {W} _ {f f, *} ^ {(k)} [ t ] \mathbf {r} ^ {(k)} [ i ] \| _ {2} ^ {2} + \epsilon_ {k} \| \mathbf {W} _ {f f, *} ^ {(k)} [ t ] \| _ {F} ^ {2} + N _ {k + 1} \log (\epsilon_ {k}) (10) \\ \log \det  \left(\hat {\boldsymbol {R}} _ {\mathbf {e} ^ {\leftarrow (k)}} [ t ] + \epsilon_ {k} \boldsymbol {I}\right) \\ \approx \frac {1}{\epsilon_ {k}} \sum_ {i = 1} ^ {t} \lambda_ {\mathbf {r}} ^ {t - i} \| \mathbf {r} ^ {(k)} [ i ] - \mathbf {W} _ {f b, *} ^ {(k)} [ t ] \mathbf {r} ^ {(k + 1)} [ i ] \| _ {2} ^ {2} + \epsilon_ {k} \| \mathbf {W} _ {f b, *} ^ {(k)} [ t ] \| _ {F} ^ {2} + N _ {k} \log (\epsilon_ {k}), (11) \\ \end{array}
$$
where  $\mathbf{W}_{ff,*}^{(k)}[t]$  is the optimal linear regularized weighted least squares forward predictor coefficients in predicting  $\mathbf{r}^{(k + 1)}[i]$  from  $\mathbf{r}^{(k)}[i]$  for  $i = 1,\dots ,t$ , and  $\mathbf{W}_{fb,*}^{(k)}[t]$  is the optimal linear regularized weighted least squares backward predictor coefficients in predicting  $\mathbf{r}^{(k)}[i]$  from  $\mathbf{r}^{(k + 1)}[i]$  for  $i = 1,\dots ,t$ . Consequently, the optimal choices of forward and backward predictor coefficients are coupled with the optimal choices of layer activations.
In the online optimization process, we initially relax the requirement on the optimality of predictors and start with random predictor coefficient selections. During the learning process, we apply a coordinate ascent-based procedure on activation signals and predictor coefficients. Specifically, at time step  $t$ , we consider two phases:
1. First, we optimize with respect to the activations  $\{\mathbf{r}^{(k)}[t], k = 1, \dots, P\}$ , where we assume predictor coefficients to be fixed. This phase yields network structure and output dynamics,
2. Next, we update the forward and backward predictor coefficients  $\mathbf{W}_{ff}^{(k)}$  and  $\mathbf{W}_{fb}^{(k)}$ , for  $k = 1, \ldots, P$ , to reduce the corresponding forward and backward prediction errors, respectively. This phase provides update expressions to be utilized in learning dynamics.
As the algorithm iterations progress, the predictor coefficients converge to the vicinity of their optimal values.
For the first phase of the online optimization, we employ a projected gradient ascent-based approach for activations: for  $k = 1, \ldots, P - 1$ , the layer activation vector  $\mathbf{r}^{(k)}[t]$  is included in the objective function terms  $\hat{I}^{(\epsilon)}(\mathbf{r}^{(k-1)}, \mathbf{r}^{(k)})[t]$  and  $\hat{I}^{(\epsilon)}(\mathbf{r}^{(k)}, \mathbf{r}^{(k+1)})[t]$ . Therefore, to calculate the gradient with respect to  $\mathbf{r}^{(k)}[t]$ , we can use expressions in (7) and (8). More specifically, we choose  $\hat{J}_k(\mathbf{r}^{(k)})[t] = \hat{I}^{(\epsilon_{k-1})}(\mathbf{r}^{(k-1)}, \mathbf{r}^{(k)})[t] + \hat{I}^{(\epsilon_k)}(\mathbf{r}^{(k)}, \mathbf{r}^{(k+1)})[t]$  for  $k = 1, \ldots, P - 1$ , to represent the components of the objective function in (9a) involving  $\mathbf{r}^{(k)}[t]$ . As described in Appendix G.1, this choice is instrumental in avoiding weight transport problem. Similarly, we can write the component of
the objective function in (9a) that is dependent on the final layer activations as  $\hat{J}_P(\mathbf{r}^{(P)})[t] = \hat{I}^{(\epsilon_{P - 1})}(\mathbf{r}^{(P - 1)},\mathbf{r}^{(P)})[t] - \frac{\beta}{2}\| \mathbf{r}^{(P)}[t] - \mathbf{y}_T[t]\|_2^2$
Based on the derivations presented in Appendix D, which directly incorporate the approximations from (10) and (11), we can express the gradient of the objective function in (9a) with respect to  $\mathbf{r}^{(k)}$  for  $k = 1,\dots ,P - 1$  as:
$$
\nabla_ {\mathbf {r} ^ {(k)}} \hat {J} \left(\mathbf {r} ^ {(1)}, \dots , \mathbf {r} ^ {(P)}\right) [ t ] = 2 \gamma B _ {\mathbf {r} ^ {(k)}} [ t ] \mathbf {r} ^ {(k)} [ t ] - \frac {1}{\epsilon_ {k - 1}} \vec {\mathbf {e}} ^ {(k)} [ t ] - \frac {1}{\epsilon_ {k}} \stackrel {\leftarrow} {\mathbf {e}} ^ {(k)} [ t ], \tag {12}
$$
where  $\gamma = \frac{1 - \lambda_{\mathbf{r}}}{\lambda_{\mathbf{r}}}$
$$
\vec {\mathbf {e}} ^ {(k)} [ t ] = \mathbf {r} ^ {(k)} [ t ] - \boldsymbol {W} _ {f f} ^ {(k - 1)} [ t ] \mathbf {r} ^ {(k - 1)} [ t ], \quad \leftarrow \vec {\mathbf {e}} ^ {(k)} [ t ] = \mathbf {r} ^ {(k)} [ t ] - \boldsymbol {W} _ {f b} ^ {(k)} [ t ] \mathbf {r} ^ {(k + 1)} [ t ], \tag {13}
$$
and  $B_{\mathbf{r}^{(k)}}[t] = (\hat{\mathbf{R}}_{\mathbf{r}^{(k)}}[t] + \epsilon_{k - 1}\pmb {I})^{-1}\approx (\hat{\mathbf{R}}_{\mathbf{r}^{(k)}}[t] + \epsilon_k\pmb {I})^{-1}$  . Similarly, for  $k = P$  , we have
$$
\nabla_ {\mathbf {r} ^ {(P)}} \hat {J} \left(\mathbf {r} ^ {(1)}, \dots , \mathbf {r} ^ {(P)}\right) [ t ] = \gamma B _ {\mathbf {r} ^ {(P)}} [ t ] \mathbf {r} ^ {(P)} [ t ] - \frac {1}{\epsilon P - 1} \vec {\mathbf {e}} ^ {(P)} [ t ] - \beta \left(\mathbf {r} ^ {(P)} [ t ] - \boldsymbol {y} _ {T} [ t ]\right). \tag {14}
$$
2.3 Neural network formulation based on information maximization
In this section, we develop a biologically plausible neural network grounded on the correlative information maximization-based network propagation model outlined in Section 2.2. To achieve this, we employ projected gradient ascent optimization for determining layer activations  $\mathbf{r}^{(1)}[t],\mathbf{r}^{(2)}[t],\dots,\mathbf{r}^{(P)}[t]$ , which shape the network structure and dynamics, as well as updating the corresponding synapses that govern the learning dynamics.
2.3.1 Network structure and neural dynamics
In this section, we show that the projected gradient ascent solution to the optimization in (9) defines a multilayer recurrent neural network. To this end, we introduce the intermediate variable  $\mathbf{u}^{(k)}$  as the updated layer-  $k$  activations prior to the projection onto the domain set  $\mathcal{P}^{(k)}$ . Utilizing the gradient expressions in (12)-(13), we can express the network dynamics for layers  $k = 1,\dots ,P - 1$  as follows (see Appendix E for details):
$$
\tau_ {\mathbf {u}} \frac {d \mathbf {u} ^ {(k)} [ t ; s ]}{d s} = - g _ {l k} \mathbf {u} ^ {(k)} [ t; s ] + \frac {1}{\epsilon_ {k}} M ^ {(k)} [ t ] \boldsymbol {r} ^ {(k)} [ t; s ] - \frac {1}{\epsilon_ {k - 1}} \vec {\mathbf {e}} _ {u} ^ {(k)} [ t; s ] - \frac {1}{\epsilon_ {k}} \leftarrow \vec {\mathbf {e}} _ {u} ^ {(k)} [ t; s ], \tag {15}
$$
$$
\vec {\mathbf {e}} _ {u} ^ {(k)} [ t; s ] = \mathbf {u} ^ {(k)} [ t; s ] - \boldsymbol {W} _ {f f} ^ {(k - 1)} [ t ] \mathbf {r} ^ {(k - 1)} [ t; s ], \quad \stackrel {\leftarrow} {\vec {\mathbf {e}}} _ {u} ^ {(k)} [ t; s ] = \mathbf {u} ^ {(k)} [ t; s ] - \boldsymbol {W} _ {f b} ^ {(k)} [ t ] \mathbf {r} ^ {(k + 1)} [ t; s ], \tag {16}
$$
$$
\mathbf {r} ^ {(k)} [ t; s ] = \sigma_ {+} \left(\mathbf {u} ^ {(k)} [ t; s ]\right), \tag {17}
$$
where  $t$  is the discrete data index,  $s$  is the continuous time index corresponding to network dynamics,  $\tau_{\mathbf{u}}$  is the update time constant,  $M^{(k)}[t] = \epsilon_k(2\gamma B_{\mathbf{r}^{(k)}}[t] + g_{lk}\pmb {I})$ , and  $\sigma_{+}$  represents the elementwise clipped-ReLU function corresponding to the projection onto the nonnegative unit-hypercube  $\mathcal{B}_{\infty , + }$  defined as  $\sigma_+(u) = \min (1,\max (u,0))$
To reinterpret the dynamics in (15) to (17) as a multi-compartmental neural network, for  $k = 1,\dots ,P - 1$ , we define the signals:
$$
\mathbf {v} _ {A} ^ {(k)} [ t; s ] = \boldsymbol {M} ^ {(k)} [ t ] \boldsymbol {r} ^ {(k)} [ t; s ] + \boldsymbol {W} _ {f b} ^ {(k)} [ t ] \mathbf {r} ^ {(k + 1)} [ t; s ], \quad \mathbf {v} _ {B} ^ {(k)} [ t; s ] = \boldsymbol {W} _ {f f} ^ {(k - 1)} [ t ] \mathbf {r} ^ {(k - 1)} [ t; s ], \tag {18}
$$
which allow us to rewrite the network activation dynamics (15) to (17) as:
$$
\tau_ {\mathbf {u}} \frac {d \mathbf {u} ^ {(k)} [ t ; s ]}{d s} = - g _ {l k} \mathbf {u} ^ {(k)} [ t; s ] + g _ {A, k} \left(\mathbf {v} _ {A} ^ {(k)} [ t; s ] - \mathbf {u} ^ {(k)} [ t; s ]\right) + g _ {B, k} \left(\mathbf {v} _ {B} ^ {(k)} [ t; s ] - \mathbf {u} ^ {(k)} [ t; s ]\right), \tag {19}
$$
$$
\mathbf {r} ^ {(k)} [ t; s ] = \sigma_ {+} (\mathbf {u} ^ {(k)} [ t; s ]), \tag {20}
$$
where  $g_{A,k} = \frac{1}{\epsilon_{k-1}}$  and  $g_{B,k} = \frac{1}{\epsilon_k}$ . Similarly, for the output layer, we employ the same expressions as (19) and (20) with  $k = P$ , except that in this case we have:
$$
\mathbf {v} _ {A} ^ {(P)} [ t; s ] = \boldsymbol {M} ^ {(P)} [ t ] \boldsymbol {r} ^ {(k)} [ t; s ] - (\boldsymbol {r} ^ {(P)} [ t; s ] - \boldsymbol {y} _ {T} [ t ]), \quad \mathbf {v} _ {B} ^ {(P)} [ t; s ] = \boldsymbol {W} _ {f f} ^ {(P - 1)} [ t ] \boldsymbol {r} ^ {(P - 1)} [ t; s ], \tag {21}
$$
where  $g_{B,P} = \frac{1}{\epsilon_{P - 1}}$ $g_{A,P} = \beta$  and  $\pmb {M}^{(P)}[t] = \beta^{-1}(\gamma \pmb{B}_{\mathbf{r}^{(P)}}[t] + g_{lk}\pmb {I})$
Remarkably, the equations (18) to (21) reveal a biologically plausible neural network that incorporates three-compartment pyramidal neuron models, as presented in [9, 10]. This intricate architecture, of which two-layer segment is demonstrated in Figure 1, naturally emerges from the proposed correlative information maximization framework. In this network structure:
-  $\mathbf{u}^{(k)}$  embodies the membrane potentials for neuronal somatic compartments of the neurons at layer- $k$ , where  $\tau_{\mathbf{u}}$  is the membrane leak time constant of soma.
-  $\mathbf{v}_B^{(k)}$  corresponds to membrane potentials for basal dendrite compartments, receiving feedforward input originating from the previous layer.
-  $\mathbf{v}_A^{(k)}$  denotes the membrane potentials for distal apical dendrite compartments, which gather top-down input from the subsequent layer and lateral inputs represented by  $M^{(k)}[t]\mathbf{r}^{(k)}$  in (18) and (21). Decomposing  $M^{(k)}$  into  $D^{(k)} - O^{(k)}$ , we find that  $D^{(k)}$  mirrors autapses[44], and the off-diagonal component  $O^{(k)}$  corresponds to lateral inhibition synapses. We use  $\mathbf{i}^{(k)} = -O^{(k)}\mathbf{r}^{(k)}$  to represent the activations of SST interneurons [14] that generate lateral inhibitions to the apical dendrites.
- Forward (backward) prediction errors manifest in the membrane voltage differences between soma and basal (distal) compartments of the pyramidal neurons.
- Forward (backward) prediction coefficients  $\mathbf{W}_{ff}^{(k)}(\mathbf{W}_{fb}^{(k)})$  are associated with feedforward (top-down) synapses connecting layers  $(k)$  and  $(k + 1)$ .
- The inverse of the regularization coefficient  $\epsilon_{k}$  is related to the conductance between soma and dendritic compartments. This is compliant with the interpretation of the  $\epsilon^{-1}$  in Appendix A.2 as the sensitivity parameter that determines the contribution of the prediction errors to the CMI. Conversely, at the output layer, the augmentation constant  $\beta$  corresponds to the conductance between soma and distal compartments. This relationship can be motivated by modifying the objective in (9a) as
$$
\sum_ {k = 0} ^ {P - 1} \hat {I} ^ {\left(\epsilon_ {k}\right)} \left(\mathbf {r} ^ {(k)}, \mathbf {r} ^ {(k + 1)}\right) [ t ] + \frac {1}{2} \hat {I} ^ {\leftarrow} ^ {\leftarrow} (\mathbf {r} ^ {(P)}, \mathbf {y} _ {T}) [ t ], \tag {22}
$$
where, through the first-order approximation, the  $\mathbf{r}^{(P)}[t]$  dependent portion of  $\hat{I}^{(\beta^{-1})}(\mathbf{r}^{(P)},\mathbf{y}_T)[t]$  can be expressed as  $-\beta \| \mathbf{r}^{(P)}[t] - \boldsymbol{W}_{fb}^{(P)}\mathbf{y}_T[t]\| _2^2$ . For accuracy, we enforce  $\boldsymbol{W}_{fb}^{(P)} = \boldsymbol{I}$ .
2.4 Learning dynamics
Network parameters consist of feedforward  $\mathbf{W}_{ff}^{(k)}$ , feedback  $\mathbf{W}_{fb}^{(k)}$  and lateral  $\mathbf{B}^{(k)}$  coefficients. The learning dynamics of these coefficients are elaborated below:
- Feedforward Coefficients are connected to the forward prediction problem defined by the optimization in (3). We can define the corresponding online optimization objective function as  $C_{ff}(\mathbf{W}_{ff}^{(k)}) = \epsilon_k\| \mathbf{W}_{ff}^{(k)}\| _F^2 +\| \vec{\mathbf{e}}^{(k + 1)}[t]\| _2^2$  for which the partial derivative is given by
$$
\frac {\partial C _ {f f} \left(\boldsymbol {W} _ {f f} ^ {(k)} [ t ]\right)}{\partial \boldsymbol {W} _ {f f} ^ {(k)}} = 2 \epsilon_ {k} \boldsymbol {W} _ {f f} ^ {(k)} [ t ] - 2 \vec {\mathbf {e}} ^ {(k + 1)} [ t ] \mathbf {r} ^ {(k)} [ t ] ^ {T}. \tag {23}
$$
In Appendix H, we provide a discussion on rewriting (23) in terms of the membrane voltage difference between the distal apical and soma compartments of the neuron, based on the equilibrium condition for the neuronal dynamics:
$$
- \vec {\mathbf {e}} ^ {(k + 1)} [ t ] \mathbf {r} ^ {(k)} [ t ] ^ {T} = g _ {B, k} ^ {- 1} \left(g _ {A, k} \mathbf {v} _ {A} ^ {(k)} [ t ] - \left(g _ {l k} + g _ {A _ {k}}\right) \mathbf {u} _ {*} ^ {(k)} [ t ] + \mathbf {h} _ {*} [ t ]\right) \mathbf {r} ^ {(k)} [ t ] ^ {T}, \tag {24}
$$
where  $\mathbf{h}_*[t]$  is nonzero only for neurons that are silent or firing at the maximum rate.
- Similarly,  $\text{Feedback Coefficients}$  are connected to the backward prediction problem defined by the optimization in (5), and the corresponding online optimization objective function as  $C_{fb}(\mathbf{W}_{fb}^{(k)}) = \epsilon_k\| \mathbf{W}_{ff}^{(k)}\| _F^2 +\| \mathbf{\bar{e}}^{(-k)}[t]\| _2^2$  for which the partial derivative is given by
$$
\frac {\partial C _ {f b} \left(\boldsymbol {W} _ {f b} ^ {(k)} [ t ]\right)}{\partial \boldsymbol {W} _ {f b} ^ {(k)}} = 2 \epsilon_ {k} \boldsymbol {W} _ {f b} ^ {(k)} [ t ] - 2 \overleftarrow {\mathbf {e}} ^ {(k)} [ t ] \mathbf {r} ^ {(k + 1)} [ t ] ^ {T}. \tag {25}
$$
To compute the updates of both feedforward and feedback coefficients, we use the EP approach [24], where the update terms are obtained based on the contrastive expressions of partial derivatives in (23) and (25) for the nudge phase, i.e.,  $\beta = \beta' > 0$ , and the free phase, i.e.,  $\beta = 0$ :
$$
\delta \boldsymbol {W} _ {f f} ^ {(k)} [ t ] \propto \frac {1}{\beta^ {\prime}} \left. \left(\left(\vec {\mathbf {e}} ^ {(k + 1)} [ t ] \mathbf {r} ^ {(k)} [ t ] ^ {T}\right) \right| _ {\beta = \beta^ {\prime}} - \left. \left(\vec {\mathbf {e}} ^ {(k + 1)} [ t ] \mathbf {r} ^ {(k)} [ t ] ^ {T}\right) \right| _ {\beta = 0}\right), \tag {26}
$$
$$
\delta \boldsymbol {W} _ {f b} ^ {(k)} [ t ] \propto \frac {1}{\beta^ {\prime}} \left. \left(\left. \left(\overleftarrow {\mathbf {e}} ^ {(k)} [ t ] \mathbf {r} ^ {(k + 1)} [ t ] ^ {T}\right) \right| _ {\beta = \beta^ {\prime}} - \left. \left(\overleftarrow {\mathbf {e}} ^ {(k)} [ t ] \mathbf {r} ^ {(k + 1)} [ t ] ^ {T}\right) \right| _ {\beta = 0}\right). \right. \tag {27}
$$
- Lateral Coefficients,  $\mathbf{B}^{(k)}$  are the inverses of the  $\epsilon \mathbf{I}$  perturbed correlation matrices. We can use the update rule in [11] for their learning dynamics after the nudge phase:
$$
\boldsymbol {B} ^ {(k)} [ t + 1 ] = \lambda_ {\mathbf {r}} ^ {- 1} \left(\boldsymbol {B} ^ {(k)} [ t ] - \gamma \mathbf {z} ^ {(k)} [ t ] \mathbf {z} ^ {(k)} [ t ] ^ {T}\right), \text {w h e r e} \mathbf {z} ^ {(k)} = \boldsymbol {B} ^ {(k)} [ t ] \mathbf {r} ^ {(k)} [ t ] \big | _ {\beta = \beta^ {\prime}}. \tag {28}
$$
As we derived in Appendix F, we can rewrite the update rule of the lateral weights in terms of the updates of autapses and lateral inhibition synapses as follows:
$$
\mathbf {D} _ {i i} ^ {(k)} [ t + 1 ] = \lambda_ {\mathbf {r}} ^ {- 1} \mathbf {D} _ {i i} ^ {(k)} [ t ] - \lambda_ {\mathbf {r}} ^ {- 1} \epsilon_ {k} 2 \gamma^ {2} \left(\mathbf {z} _ {i} ^ {(k)} [ t ]\right) ^ {2} + \epsilon_ {k} g _ {l k} (1 - \lambda_ {\mathbf {r}} ^ {- 1}), \quad \forall i \in \{1, \dots , N _ {k} \} \tag {29}
$$
$$
\mathbf {O} _ {i j} ^ {(k)} [ t + 1 ] = \lambda_ {\mathbf {r}} ^ {- 1} \mathbf {O} ^ {(k)} [ t ] _ {i j} + \lambda_ {\mathbf {r}} ^ {- 1} \epsilon_ {k} 2 \gamma^ {2} \mathbf {z} _ {i} ^ {(k)} [ t ] \mathbf {z} _ {j} ^ {(k)} [ t ], \forall i, j \in \{1, \dots , N _ {k} \}, \text {w h e r e} i \neq j \tag {30}
$$
3 Discussion of results
- In (A.14), we devise an update for layer activation  $\mathbf{r}^{(k)}$  by employing two distinct forms of the CMI associated with  $\mathbf{r}^{(k)}$ :  $\hat{I}^{(\epsilon_{k - 1})}(\mathbf{r}^{(k - 1)},\mathbf{r}^{(k)})[t]$ , the CMI with the preceding layer, encompassing the forward prediction error for estimating  $\mathbf{r}^{(k)}$ , and  $\hat{I}^{(\epsilon_k)}(\mathbf{r}^{(k)},\mathbf{r}^{(k + 1)})[t]$ , the CMI with the subsequent layer, incorporating the backward prediction error for estimating  $\mathbf{r}^{(k)}$ . Employing these alternative expressions is crucial in circumventing the weight transport problem and offering a more biologically plausible framework. For further discussion, please refer to Appendix G.
- In the context of the proposed correlative information maximization framework, forward and backward predictive coding naturally emerges as a crucial mechanism. By incorporating both alternative expressions of CMI, the framework focuses on minimizing both forward and backward prediction errors between adjacent layers via feedforward and feedback connections. These connections foster bidirectional information flow, thereby enhancing the overall learning process.
- Figure 1 depicts the interplay between the CorInfoMax objective and the corresponding network architecture. The emergence of lateral connections and autapses can be attributed to the maximization of the unconditional layer entropy component of the CMI, which allows for efficient utilization of the available representation dimensions and avoids dimensional degeneracy. Simultaneously, the minimization of conditional entropies between adjacent layers gives rise to feedforward and feedback connections, effectively reducing redundancy within representations.
- We employ time-contrastive learning, as in GenRec [45], EP [24] and CSM [27], by implementing separate phases with Hebbian and anti-Hebbian updates, governed by an assumed teaching signal. It has been conjectured that the teaching signal in biological networks can be modeled by the oscillations in the brain [2, 46, 47]. Although the oscillatory rhythms and their synchronization in the brain are elusive, they are believed to play an important role in adaptive processes such as learning and predicting upcoming events [48, 49].
In this section, we evaluate the performance of our CorInfoMax framework with two layer fully connected networks on image classification tasks using three popular datasets: MNIST [50], Fashion-MNIST [51], and CIFAR10 [52]. We used layer sizes of 784, 500, 10 for both MNIST and Fashion-MNIST datasets while we used layer sizes of 3072, 1000, 10 for CIFAR10 dataset, and the final layer size 10 corresponds to one-hot encoded output vectors. Further details including full set of hyperparameters can be found in Appendix J. We compare the effectiveness of our approach against
other contrastive methods, such as EP [24] and CSM [27], as well as explicit methods, including PC [22] and PC-Nudge [53], when training multilayer perceptron (MLP) architectures.
We examine two distinct constraints on the activations of CorInfoMax Networks: (i)  $\mathcal{B}_{\infty, +}$ , representing the nonnegative part of the unit hypercube, and (ii)  $\mathcal{B}_{1, +} = \{\mathbf{r} : \mathbf{r} \succ 0, \| \mathbf{r} \|_1 \leq 1\}$ , denoting the nonnegative part of the unit  $\ell_1$ -norm ball [40]. Table 1 presents the test accuracy results for each algorithm, averaged over 10 realizations along with the corresponding standard deviations. These findings demonstrate that CorInfoMax networks can achieve comparable or superior performance in relation to the state-of-the-art methods for the selected tasks. Additional information regarding these experiments, as well as further experiments, can be found in the Appendix. Our code is available online<sup>1</sup>.
Table 1: Test accuracy results (mean ± standard deviation from  $n = 10$  runs) for CorInfoMax networks are compared with other biologically-plausible algorithms. The performance of CSM on the CIFAR10 dataset is taken from [27], while the remaining results stem from our own simulations.
MNIST	FashionMNIST	CIFAR10
CorInfoMax-B∞,+ (Appendix J.4)	97.62 ± 0.1	88.14 ± 0.3	51.86 ± 0.3
CorInfoMax-B1,+ (Appendix J.6)	97.71 ± 0.1	88.09 ± 0.1	51.19 ± 0.4
EP	97.61 ± 0.1	88.06 ± 0.7	49.28 ± 0.5
CSM	98.08 ± 0.1	88.73 ± 0.2	40.79*
PC	98.17 ± 0.2	89.31 ± 0.4	-
PC-Nudge	97.71 ± 0.1	88.49 ± 0.3	48.58 ± 0.7
Feedback Alignment (with MSE Loss)	97.99 ± 0.03	88.72 ± 0.5	50.75 ± 0.4
Feedback Alignment (with CrossEntropy Loss)	97.95 ± 0.08	88.38 ± 0.9	52.37 ± 0.4
BP (with MSE Loss)	97.58 ± 0.01	88.39 ± 0.1	52.75 ± 0.1
BP (with CrossEntropy Loss)	98.27 ± 0.03	89.41 ± 0.2	53.96 ± 0.3
5 Discussion and Conclusion
In this article, we have presented the correlative information maximization (CorInfoMax) framework as a biologically plausible approach to constructing supervised neural network models. Our proposed method addresses the long-standing weight symmetry issue by providing a principled solution, which results in asymmetric forward and backward prediction networks. The experimental analyses demonstrate that CorInfoMax networks provide better or on-par performance in image classification tasks compared to other biologically plausible networks while alleviating the weight symmetry problem. Furthermore, the CorInfoMax framework offers a normative approach for developing network models that incorporate multi-compartment pyramidal neuron models, aligning more closely with the experimental findings about the biological neural networks. The proposed framework is useful in obtaining potential insights such as the role of lateral connections in embedding space expansion and avoiding degeneracy, feedback and feedforward connections for prediction to reduce redundancy, and activation functions/interneurons to shape feature space and compress. Despite the emphasis on supervised deep neural networks in our work, it's crucial to highlight that our approach—replacing the backpropagation algorithm, which suffers from the weight transportation problem, with a normative method devoid of such issues—is potentially extendable to unsupervised and self-supervised learning contexts.
One potential limitation of our framework, shared by other supervised approaches, is the necessity for model parameter search to improve accuracy. We discuss this issue in detail in Appendix K. Another limitation stems from the intrinsic nature of our approach, which involves the determination of neural activities through recursive dynamics (see Appendix J). While this aspect is fundamental to our methodology, it does result in slower computation times compared to conventional neural networks in digital hardware implementation. However, it is worth noting that our proposed network, characterized by local learning rules, holds the potential for efficient and low-power implementations on future neuromorphic hardware chips. Furthermore, our method employs the time contrastive learning technique known as Equilibrium Propagation, which necessitates two distinct phases for learning.
$^{1}$ https://github.com/BariscanBozkurt/Supervised-CorInfoMax
This research was supported by KUIS AI Center Research Award. B. Bozkurt acknowledges the support by Gatsby PhD programme, which is supported by the Gatsby Charitable Foundation (GAT3850). C. Pehlevan is supported by NSF Award DMS-2134157, NSF CAREER Award IIS-2239780, and a Sloan Research Fellowship. This work has been made possible in part by a gift from the Chan Zuckerberg Initiative Foundation to establish the Kempner Institute for the Study of Natural and Artificial Intelligence.