SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Bariscan Bozkurt $^{1,2,3}$  Cengiz Pehlevan $^{4,5}$  Alper T. Erdogan $^{2,3}$
<sup>1</sup> Gatsby Computational Neuroscience Unit, UCL, United Kingdom
$^{2}$ KUIS AI Center, Koc University, Turkey  $^{3}$ EEE Department, Koc University, Turkey
$^{4}$ John A. Paulson School of Engineering & Applied Sciences and Center for Brain Science, Harvard University, Cambridge, 02138 MA, USA
5 Kempner Institute for the Study of Natural and Artificial Intelligence
{bbozkurt15, alperdogan}@ku.edu.tr cpehlevan@seas.harvard.edu
The backpropagation algorithm has experienced remarkable success in training large-scale artificial neural networks; however, its biological plausibility has been strongly criticized, and it remains an open question whether the brain employs supervised learning mechanisms akin to it. Here, we propose correlative information maximization between layer activations as an alternative normative approach to describe the signal propagation in biological neural networks in both forward and backward directions. This new framework addresses many concerns about the biological-plausibility of conventional artificial neural networks and the backpropagation algorithm. The coordinate descent-based optimization of the corresponding objective, combined with the mean square error loss function for fitting labeled supervision data, gives rise to a neural network structure that emulates a more biologically realistic network of multi-compartment pyramidal neurons with dendritic processing and lateral inhibitory neurons. Furthermore, our approach provides a natural resolution to the weight symmetry problem between forward and backward signal propagation paths, a significant critique against the plausibility of the conventional backpropagation algorithm. This is achieved by leveraging two alternative, yet equivalent forms of the correlative mutual information objective. These alternatives intrinsically lead to forward and backward prediction networks without weight symmetry issues, providing a compelling solution to this long-standing challenge.
How biological neural networks learn in a supervised manner has long been an open problem. The backpropagation algorithm [1], with its remarkable success in training large-scale artificial neural networks and intuitive structure, has inspired proposals for how biologically plausible neural networks can perform the necessary efficient credit-assignment for supervised learning in deep neural architectures [2]. Nonetheless, certain aspects of the backpropagation algorithm, combined with the oversimplified nature of artificial neurons, have been viewed as impediments to proposals rooted in this inspiration [3].
One of the primary critiques regarding the biological plausibility of the backpropagation algorithm is the existence of a parallel backward path for backpropagating error from the output towards the input, which uses the same synaptic weights as the forward path [1, 2, 4]. Although such weight transport, or weight symmetry, is deemed highly unlikely based on experimental evidence [3, 4],
some biologically plausible frameworks still exhibit this feature, which is justified by the symmetric structure of the Hebbian updates employed in these frameworks [2, 5, 6].
The concerns about the simplicity of artificial neurons have been addressed by models which incorporate multi-compartment neuron models into networked architectures and ascribe important functions to dendritic processing in credit assignment [7, 8, 9, 10]. This new perspective has enabled the development of neural networks with improved biological plausibility.
In this article, we propose the use of correlative information maximization (CorInfoMax) among consecutive layers of a neural network as a new supervised objective for biologically plausible models, which offers
- a principled solution to the weight symmetry problem: our proposed information theoretic criterion aims to maximize the linear dependence between the signals in two neighboring layers, naturally leading to the use of linear or affine transformations in between them. A key property of this approach is that employing two alternative expressions for the correlative mutual information (CMI) results in potentially asymmetric forward and backward prediction networks, offering a natural solution to the weight transport problem. Consequently, predictive coding in both directions emerges as the inherent solution to the correlative information maximization principle, fostering signal transmission in both forward and top-down directions through asymmetrical connections. While the CorInfoMax principle enhances information flow in both directions, the introduction of set membership constraints on the layer activations, such as non-negativity, through activation nonlinearities and lateral inhibitions, encourages compression of information and sparse representations [11].
- a normative approach for deriving networks with multi-compartment neurons: the gradient-based optimization of the CorInfoMax objective naturally leads to network models that employ multi-compartment pyramidal neuron models accompanied by interneurons as illustrated in Figure 1.
As derived and explained in detail in Section 2, the resulting networks incorporate lateral connections and auto-synapses (autapses) to increase the entropy of a layer, promoting utilization of all dimensions within the representation space of that layer. Meanwhile, asymmetric feedforward and feedback connections act as forward and backward predictors of layer activation signals, respectively, to reduce the conditional entropies between layers, targeting the elimination of redundancy.
Figure 1: A segment of a correlative information maximization based neural network. Each layer consists of three-compartment pyramidal neurons with outputs  $\mathbf{r}^{(k)}$  and membrane voltages  $(\mathbf{u}^{(k)})$ -som,  $\mathbf{v}_B^{(k)}$ -basal dendrites,  $\mathbf{v}_A^{(k)}$ -distal apical dendrites) and interneurons with outputs  $\mathbf{i}^{(k)}$ . The CMI expression  $I^{(\epsilon_k)}(\mathbf{r}^{(k)},\mathbf{r}^{(k + 1)})$  ( $I^{(\epsilon_k)}(\mathbf{r}^{(k)},\mathbf{r}^{(k + 1)})$ ) defines forward (backward) prediction synapses  $\mathbf{W}_{ff}^{(k)}(\mathbf{W}_{fb}^{(k)})$ , for minimizing  $H^{(\epsilon_k)}(\mathbf{r}^{(k + 1)}|_L\mathbf{r}^{(k)})$  ( $H^{(\epsilon_k)}(\mathbf{r}^{(k)}|_L\mathbf{r}^{(k + 1)})$ ) and the lateral connections  $\mathbf{O}^{(k + 1)}(\mathbf{O}^{(k)})$  and autapses  $\mathbf{D}^{(k + 1)}(\mathbf{D}^{(k)})$  connected to distal apical dendrites at the  $k + 1$ -th ( $k$ -th) layer, for maximizing  $H^{(\epsilon_k)}(\mathbf{r}^{(k + 1)})$  ( $H^{(\epsilon_k)}(\mathbf{r}^{(k)})$ ).
1.1.1 Multi-compartmental neuron model based biologically plausible approaches
Experimentally grounded studies, such as [7, 12], have been influential for considering a role for dendritic-processing in multi-compartmental neurons for learning and credit assignment [13]. Subsequent research has explored biologically plausible models with supervised learning functionality, such as the two-compartment neuron model by Urbanczik and Senn [8] and the three-compartment pyramidal neuron model by Sacramento et al. [9]. Both models integrate non-Hebbian learning and spike-time dependent plasticity, while the latter includes SST interneurons [14]. Similar frameworks have been proposed by [15] and [10], with the latter introducing a normative framework based on multi-compartmental neuron structure, top-down feedback, lateral and feedforward connections, and Hebbian and non-Hebbian learning rules, emerging from the optimization of a prediction error objective with a whitening constraint on co-layer neurons.
In a similar vein to [10], we propose an alternative normative framework based on information maximization principle. In this framework, the three-compartment structure and associated forward, top-down and lateral synaptic connections stem from the maximization of CMI between adjacent layers, without the imposition of any whitening constraint.
1.1.2 Weight symmetry problem
A central concern regarding the biological plausibility of the backpropagation algorithm pertains to the weight symmetry issue: synaptic weights in the feedback path for error backpropagation are transposes of those used in the forward inference path [2, 3, 16]. The requirement of tied weights in backpropagation is questionable for physically distinct feedforward and feedback paths in biological systems, leading many researchers to focus on addressing the weight symmetry issue.
Various strategies have been devised to address the weight symmetry issue. For example, the feedback alignment approach, which fixes randomly initialized feedback weights and adapts feedforward weights, was offered as a plausible solution [17]. Later Akrout et.al. [18] proposed its extension by updating feedback weights towards to the transpose of the feedforward weights. Along the similar lines, Amit introduced antisymmetry through separate random initializations [19]. Liao et.al. [20] showed that the sign of the feedback weights (rather than their magnitude) affects the learning performance, and proposed the sign-symmetry algorithm.
Intriguingly, this symmetric weight structure is also observed in biologically plausible frameworks such as predictive coding (PC) [21, 22, 23], equilibrium propagation (EP) [24, 25, 26], and similarity matching [27]. This phenomenon can be rationalized by the transpose symmetry of the Hebbian update with respect to inputs and outputs. The EP framework in [25] unties forward and backward connections inspired by [28, 29], and only yields small performance degradation. A more recent approach by Golkar et al. [10] addresses this challenge by integrating two alternative forward prediction error loss function terms associated with the same network layer and leveraging presumed whitening constraints to eliminate shared feedback coefficients.
In existing predictive coding-based schemes such as [21, 22, 23], the loss function contains only forward prediction error terms. The feedback connection with symmetric weights, which backpropagates forward prediction error, emerges due to the gradient-based optimization of the PC loss. In contrast, our framework's crucial contribution is the adoption of two alternative expressions for the correlative mutual information between consecutive network layers as the central normative approach. Utilizing these two alternatives naturally leads to both forward and backward prediction paths with asymmetric weights, promoting information flow in both feedforward and top-down directions. Unlike the work of [10], our method circumvents the need for layer whitening constraints and additional forward prediction terms to achieve asymmetric weights.
1.1.3 Correlative information maximization
Information maximization has been proposed as a governing or guiding principle in several machine learning and neuroscience frameworks for different tasks: (i) The propagation of information within a self-organized network as pioneered by Linsker [30]. (ii) Extracting hidden features or factors associated with observations by maximizing information between the input and its internal representation such as independent component analysis (ICA-InfoMax) approach by [31]. In the neuroscience
domain, the motivation has been to provide normative explanations to the behaviour of cortical activities evidenced by experimental work, such as orientation and visual stimuli length selectivity of primary visual cortex neurons [32, 33]. The same idea has been recently extended in the machine learning field by the Deep Infomax approach where the goal is to transfer maximum information from the input of a deep network to its final layer, while satisfying prior distribution constraints on the output representations [34]. (iii) Matching representations corresponding to two alternative augmentations or modalities of the same input in the context of self-supervised learning [35].
Correlative mutual information maximization has been recently proposed as an alternative for Shannon Mutual Information (SMI), due to its desirable properties [36]: (i) maximization of CMI is equivalent to maximizing linear dependence, which may be more relevant than establishing arbitrary nonlinear dependence in certain applications [37], (ii) it is based only on the second order statistics, making it relatively easier to optimize. We additionally note that criteria based on correlation are intrinsically linked to local learning rules, leading to biologically plausible implementations, [38, 39]. Erdogan [36] proposed the use of CorInfoMax for solving blind source separation (BSS) problem to retrieve potentially correlated components from their mixtures. Ozsoy et al. [37] proposed maximizing the CMI between the representations of two different augmentations of the same input as a self-supervised learning approach. More recently, Bozkurt et al. [11] introduced an unsupervised framework to generate biologically plausible neural networks for the BSS problem with infinitely many domain selections using the CMI objective.
In this article, we suggest employing the CorInfoMax principle for biologically plausible supervised learning. The key difference compared to the unsupervised framework presented in [11] is the utilization of two alternative forms of mutual information. This leads to a bidirectional information flow that enables error backpropagation without encountering the weight symmetry issue.
2 Deep correlative information maximization
2.1 Network data model
We assume a dataset with  $L$  input data points  $\mathbf{x}[t] \in \mathbb{R}^m$ ,  $t = 1, \dots, L$ , and let  $\mathbf{y}_T[t] \in \mathbb{R}^n$  be the corresponding labels. We consider a network with  $P - 1$  hidden layers whose activities are denoted by  $\mathbf{r}^{(k)} \in \mathbb{R}^{N_k}$ ,  $k = 1, \dots, P - 1$ . For notational simplicity, we also denote input and output of the network by  $\mathbf{r}^{(0)}$  and  $\mathbf{r}^{(P)}$ , i.e.,  $\mathbf{r}^{(0)}[t] = \mathbf{x}[t]$  and  $\mathbf{r}^{(P)}[t] = \hat{\mathbf{y}}[t]$ . We consider polytopic constraints for the hidden and output layer activities, i.e.,  $\mathbf{r}^{(k)} \in \mathcal{P}^{(k)}$ , where  $\mathcal{P}^{(k)}$  is the presumed polytopic domain for the  $k$ -th layer [11, 40]. We note that the polytopic assumptions are plausible as the activations of neurons in practice are bounded. In particular, we will make the specific assumption that  $\mathcal{P}^{(k)} = \mathcal{B}_{\infty, +} = \{\mathbf{r} : \mathbf{0} \preccurlyeq \mathbf{r} \preccurlyeq \mathbf{1}\}$ , i.e., (normalized) activations lie in a nonnegative unit-hypercube. Such nonnegativity constraints have been connected to disentangling behavior [41, 42, 43], however, we consider extensions in the form of alternative polytopic sets corresponding to different feature priors [11] (see Appendix C). More broadly, the corresponding label  $\mathbf{y}_T$  can be, one-hot encoded label vectors for a classification problem, or discrete or continuous valued vectors for a regression problem.
2.2 Correlative information maximization based signal propagation
Our proposed CorInfoMax framework represents a principled approach where both the structure of the network and its internal dynamics as well as the learning rules governing adaptation of its parameters are not predetermined. Instead, these elements emerge naturally from an explicit optimization process. As the optimization objective, we propose the maximization of correlative mutual information (see Appendix A between two consecutive network layers. As derived in future sections, the proposed objective facilitates information flow—input-to-output and vice versa, while the presumed domains for the hidden and output layers inherently induce information compression and feature shaping.
In Sections 2.2.1 and 2.2.2, we outline the correlative mutual information-based objective and its implementation based on samples, respectively. Section 2.3 demonstrates that the optimization of this objective through gradient ascent naturally results in recurrent neural networks with multicompartment neurons. Finally, Section 2.4 explains how the optimization of the same criterion leads to biologically plausible learning dynamics for the resulting network structure.
2.2.1 Stochastic CorInfoMax based supervised criterion
We propose the total correlative mutual information among consecutive layers, augmented with the mean-square-error (MSE) training loss, as the stochastic objective to be maximized:
$$
J \left(\mathbf {r} ^ {(1)}, \dots , \mathbf {r} ^ {(P)}\right) = \sum_ {k = 0} ^ {P - 1} I ^ {\left(\epsilon_ {k}\right)} \left(\mathbf {r} ^ {(k)}, \mathbf {r} ^ {(k + 1)}\right) - \frac {\beta}{2} E \left(\left\| \mathbf {y} _ {T} - \mathbf {r} ^ {(P)} \right\| _ {2} ^ {2}\right), \tag {1}
$$
where, as defined in [36, 37] and in Appendix A,
$$
\overrightarrow {I ^ {(\epsilon_ {k})}} \left(\mathbf {r} ^ {(k)}, \mathbf {r} ^ {(k + 1)}\right) = \frac {1}{2} \log \det  \left(\mathbf {R} _ {\mathbf {r} ^ {(k + 1)}} + \epsilon_ {k} \mathbf {I}\right) - \frac {1}{2} \log \det  \left(\mathbf {R} _ {\mathbf {e} _ {*}} ^ {\rightarrow (k + 1)} + \epsilon_ {k} \mathbf {I}\right), \tag {2}
$$
is the correlative mutual information between layers  $\mathbf{r}^{(k)}$  an  $\mathbf{r}^{(k + 1)}$ ,  $\mathbf{R}_{\mathbf{r}^{(k + 1)}} = E(\mathbf{r}^{(k + 1)}\mathbf{r}^{(k + 1)^T})$  is the autocorrelation matrix corresponding to the layer  $\mathbf{r}^{(k + 1)}$  activations, and  $\mathbf{R}_{\vec{\mathbf{e}}_{*}^{(k + 1)}}$  corresponds to the error autocorrelation matrix for the best linear regularized minimum MSE predictor of  $\mathbf{r}^{(k + 1)}$  from  $\mathbf{r}^{(k)}$ . Therefore, the mutual information objective in (2) makes a referral to the regularized forward prediction problem represented by the optimization
$$
\underset {\boldsymbol {W} _ {f f} ^ {(k)}} {\text {m i n i m i z e}} E \left(\left\| \vec {\mathbf {e}} ^ {(k + 1)} \right\| _ {2} ^ {2}\right) + \epsilon_ {k} \left\| \boldsymbol {W} _ {f f} ^ {(k)} \right\| _ {F} ^ {2} \quad \text {s . t .} \quad \vec {\mathbf {e}} ^ {(k + 1)} = \mathbf {r} ^ {(k + 1)} - \boldsymbol {W} _ {f f} ^ {(k)} \mathbf {r} ^ {(k)}, \tag {3}
$$
and  $\mathbf{e}_{*}^{(k + 1)}$  is the forward prediction error corresponding to the optimal forward predictor  $W_{ff,*}^{(k)}$ .
If we interpret the maximization of CMI in (2): the first term on the right side of (2) encourages the spread of  $\mathbf{r}^{(k + 1)}$  in its presumed domain  $\mathcal{P}^{(k + 1)}$ , while the second term incites the minimization of redundancy in  $\mathbf{r}^{(k + 1)}$  beyond its component predictable from  $\mathbf{r}^{(k)}$ .
An equal and alternative expression for the CMI can be written as (Appendix A)
$$
I ^ {(\epsilon_ {k})} \left(\mathbf {r} ^ {(k)}, \mathbf {r} ^ {(k + 1)}\right) = \frac {1}{2} \log \det  \left(\mathbf {R} _ {\mathbf {r} ^ {(k)}} + \epsilon_ {k} \mathbf {I}\right) - \frac {1}{2} \log \det  \left(\mathbf {R} _ {\mathbf {e} _ {*} ^ {- (k)}} + \epsilon_ {k} \mathbf {I}\right), \tag {4}
$$
where  $R_{\mathbf{e}_{*}^{(k)}}$  corresponds to the error autocorrelation matrix for the best linear regularized minimum MSE predictor of  $\mathbf{r}^{(k)}$  from  $\mathbf{r}^{(k + 1)}$ . The corresponding regularized backward prediction problem is defined by the optimization
$$
\underset {\boldsymbol {W} _ {f b} ^ {(k)}} {\text {m i n i m i z e}} E \left(\left\| \overline {{\mathbf {e}}} ^ {(k)} \right\| _ {2} ^ {2}\right) + \epsilon_ {k} \left\| \boldsymbol {W} _ {f b} ^ {(k)} \right\| _ {F} ^ {2} \quad \text {s . t .} \quad \overline {{\mathbf {e}}} ^ {(k)} = \mathbf {r} ^ {(k)} - \boldsymbol {W} _ {f b} ^ {(k)} \mathbf {r} ^ {(k + 1)}. \tag {5}
$$
We observe that the two alternative yet equivalent representations of the correlative mutual information between layers  $\mathbf{r}^{(k)}$  and  $\mathbf{r}^{(k + 1)}$  in (2) and (4) are intrinsically linked to the forward and backward prediction problems between these layers, which are represented by the optimizations in (3) and (5), respectively. As we will demonstrate later, the existence of these two alternative forms for the CMI plays a crucial role in deriving a neural network architecture that overcomes the weight symmetry issue.
2.2.2 Sample-based supervised CorInfoMax criterion
Our aim is to construct a biologically plausible neural network that optimizes the total CMI, equation (1), in an adaptive manner. Here, we obtain a sample-based version of (1) as a step towards that goal.
We first define the exponentially-weighted sample auto and cross-correlation matrices as follows:
$$
\hat {\mathbf {R}} _ {\mathbf {r} ^ {(k)}} [ t ] = \frac {1 - \lambda_ {\mathbf {r}}}{1 - \lambda_ {\mathbf {r}} ^ {t}} \sum_ {i = 1} ^ {t} \lambda_ {\mathbf {r}} ^ {t - i} \mathbf {r} ^ {(k)} [ i ] \mathbf {r} ^ {(k)} [ i ] ^ {T}, \hat {\mathbf {R}} _ {\mathbf {r} ^ {(k)} \mathbf {r} ^ {(k + 1)}} [ t ] = \frac {1 - \lambda_ {\mathbf {r}}}{1 - \lambda_ {\mathbf {r}} ^ {t}} \sum_ {i = 1} ^ {t} \lambda_ {\mathbf {r}} ^ {t - i} \mathbf {r} ^ {(k)} [ i ] \mathbf {r} ^ {(k + 1)} [ i ] ^ {T}, \tag {6}
$$
for  $k = 0, \dots, P$ , respectively, where  $0 \ll \lambda_{\mathbf{r}} < 1$  is the forgetting factor. Next, we define two equivalent forms of the sample-based CMI,  $\hat{I}^{(\epsilon)}(\mathbf{r}^{(k)}, \mathbf{r}^{(k + 1)})[t]$ :
$$
\hat {I} ^ {(\epsilon_ {k})} \left(\mathbf {r} ^ {(k)}, \mathbf {r} ^ {(k + 1)}\right) [ t ] = \frac {1}{2} \log \det  \left(\hat {\mathbf {R}} _ {\mathbf {r} ^ {(k + 1)}} [ t ] + \epsilon_ {k} \mathbf {I}\right) - \frac {1}{2} \log \det  \left(\hat {\mathbf {R}} _ {\vec {\mathbf {e}} _ {*} ^ {(k + 1)}} [ t ] + \epsilon_ {k} \mathbf {I}\right), \tag {7}
$$
$$
\hat {I} ^ {\left(\epsilon_ {k}\right)} \left(\mathbf {r} ^ {(k)}, \mathbf {r} ^ {(k + 1)}\right) [ t ] = \frac {1}{2} \log \det  \left(\hat {\mathbf {R}} _ {\mathbf {r} ^ {(k)}} [ t ] + \epsilon_ {k} \boldsymbol {I}\right) - \frac {1}{2} \log \det  \left(\hat {\boldsymbol {R}} _ {\mathbf {e} _ {*}} ^ {\leftarrow (k)} [ t ] + \epsilon_ {k} \boldsymbol {I}\right), \tag {8}
$$
where  $\hat{R}_{\vec{\mathbf{e}}_{*}^{(k + 1)}}[t]$  is the exponentially-weighted sample autocorrelation matrix for the forward prediction error at level-  $(k + 1)$ ,  $\vec{\mathbf{e}}^{(k + 1)_*}[t]$ , corresponding to the best linear exponentially-weighted regularized least squares predictor of  $\mathbf{r}^{(k + 1)}[t]$  from the lower level activations  $\mathbf{r}^{(k)}[t]$ . Similarly,  $\hat{R}_{\vec{\mathbf{e}}^{(k)}}[t]$  is the exponentially-weighted autocorrelation matrix for the backward prediction error at level- $(k)$ ,  $\vec{\mathbf{e}}^{(k)}[t]$ , corresponding to the best linear exponentially-weighted regularized least squares predictor of  $\mathbf{r}^{(k)}[t]$  from the higher level activations  $\mathbf{r}^{(k + 1)}[t]$ .
The sample-based CorInfoMax optimization can be written as:
$$
\underset {\mathbf {r} ^ {(k)} [ t ], k = 0, \dots , P} {\text {m a x i m i z e}} \quad \sum_ {k = 0} ^ {P - 1} \hat {I} ^ {(\epsilon_ {k})} \left(\mathbf {r} ^ {(k)}, \mathbf {r} ^ {(k + 1)}\right) [ t ] - \frac {\beta}{2} \| \mathbf {y} _ {T} [ t ] - \mathbf {r} ^ {(P)} [ t ] \| _ {2} ^ {2} = \hat {J} \left(\mathbf {r} ^ {(1)}, \dots , \mathbf {r} ^ {(P)}\right) [ t ] \tag {9a}
$$
subject to  $\mathbf{r}^{(k)}[t]\in \mathcal{P}^{(k)},k = 1,\ldots ,P,$  (9b)
$\mathbf{r}^{(0)}[t] = \mathbf{x}[t],$  (9c)
As outlined in Appendix B, we can employ Taylor series linearization to approximate the logdet terms associated with forward and backward prediction errors in (2) and (4) in the form