Brain-like approaches to unsupervised learning of hidden representations - a comparative study 
Authors: Naresh Balaji
Anders Lansner
Pawel Herman
Venue:  (2021)
Keywords: neural networks
bio-inspired
brain-like
unsupervised learning
structural plasticity

Abstract: Unsupervised learning of hidden representations has been one of the most vibrant research directions in machine learning in recent years. In this work we study the brain-like Bayesian Confidence Propagating Neural Network (BCPNN) model, recently extended to extract sparse distributed high-dimensional representations. The saliency and separability of the hidden representations when trained on MNIST dataset is studied using an external linear classifier and compared with other unsupervised learning methods that include restricted Boltzmann machines and autoencoders. 

Under review as a conference paper at ICLR 2021
BRAIN-LIKE APPROACHES TO UNSUPERVISED LEARNING OF HIDDEN REPRESENTATIONS- A COMPARATIVE STUDY
Anonymous authors Paper under double-blind review
ABSTRACT
Unsupervised learning of hidden representations has been one of the most vi- brant research directions in machine learning in recent years. In this work we study the brain-like Bayesian Confidence Propagating Neural Network (BCPNN) model, recently extended to extract sparse distributed high-dimensional represen- tations. The saliency and separability of the hidden representations when trained MNIST dataset is studied using external linear classifier and compared with other unsupervised learning methods that include restricted Boltzmann machines and autoencoders.
INTRODUCTION
Artificial neural networks have made remarkable progress in supervised pattern recognition in recent years. In particular, deep neural networks have dominated the field largely due to their capability to discover hierarchies of salient data representations. However, most recent deep learning methods rely extensively supervised learning from labelled samples for extracting and tuning data repre- sentations. Given the abundance of unlabeled data there is an urgent demand for unsupervised or semi-supervised approaches to learning of hidden representations (Bengio al. (003)). Although early concepts of greedy layer-wise pretraining allow for exploiting unlabeled data, ultimately the application of deep pre-trained networks to pattern recognition problems rests on label dependent end-to-end weight fine tuning (Erhan et al. 2009). At the same time, we observe a surge of in- terest in more brain plausible networks for unsupervised and semi-supervised learning problems that build on some fundamental principles of neural information processing in the brain (Pehlevan & Chklovskii 2019 ming et al. 2019). Most importantly, these brain-like computing approaches rely on local learning rules and label independent biologically compatible mechanisms to build data representations whereas deep learning methods predominantly make use error back proepprrppattion (backprop) for learning the weights. Although efficient, backprop has several issues that make it an unlikely candidate model for synaptic plasticity in the brain. The most apparent issue is that the synaptic connection strength between postulate, i.c. to depend only on the available local information provided by the activities of pre- and postsynaptic neurons. This is violated in backprop since synaptic weight updates need gradi- ent signals to be communicated from distant output layers. Please refer to (Whitington & Bogacz, 2019 Lillicrap al., 2020) for a detailed review of possible biologically plausible implementations of and alternatives to backprop.
biological neurons is expected to comply with Hebb's
In this work we utilize the MNIST dataset to compare two classical learning systems, the autoen- coder (AE) and the restricted Boltzmann machine (RBM), with two brain-like approaches to unsu- pervised learning hidden representations, i.e. the recently proposed model by Krotov and Hopfield (referred to the KH model) (Krotov & Hopfield 2019), and the BCPNN model (Ravichandran ctt al. 2020), which both rely on biologically plausible learning strategies. In particular, we qualita- tively examine the extracted hidden representations and quantify their label dependent separability using a simple linear classifier on top of all the networks under investigation. This classification step is not part of the learning strategy, and use it merely to evaluate the resulting representations.
Special emphasis is on the feedforward BCPNN model with a single hidden layer, which frames the update and learning steps of the neural network as probabilistic computations. Probabilistic ap-

---

Under review as a conference paper at ICLR 2021
proaches widely used in both deep learning models (Goodfellow et al. 2016) and computational models of brain function (Doya et al. 2007). One disadvantage of probabilistic models is that exact inference and learning distributed representations is often intractable and forces approximate ap- proaches like sampling- haaeiinggg-dddddooo or variational methods (Rezende et al. 2014). In this work, we adopt a modular BCPNN architecture, previously used in abstract models of associative memory (Sandberg) et al. 2002] Lansner et al., 2209)) action selection (Berthet et al.) 2012), and in application to brain imaging [Benjaminsson et al. 2010 Schain et al. 2013) and data mining (Orre al.) 2000). Spiking versions of BCPNN have also been used in biologically detailed models of different forms of cor- tical associative memory (Lundqvist et al., 2011] Fiebig Lansner] 2017] Tully et al.] (014). The modules in BCPNN, referred to as hypercolumns (HCs), comprise a set of functional minicolumns (MCs) that compete in a soft--winer-take-all manner. The abstract view of a HC in this abstract cortical-like network is that it represents some attribute, e.g. edge orientation, in a discrete coded manner. A minicolumn comprises unit that conceptually represents one discrete value (a realiza- tion of the given attribute) and, biological parallel, it accounts for local subnetwork of around a hundred recurrently connected neurons with similar receptive field properties (Mounteastle) 1997). Such architecture was initially generalized from the primary visual cortex, but today has more support also from later experimental work and has been featured in spiking computational models of cortex (Rockland 2010
Lansner 2009).
Finally, in this work we highlight additional mechanisms of bias regulation and structural plasticity, introduced recently to the BCPNN framework (Ravichandran al.] (020), which enable unsuper- vised learning of hidden representations. The bias regulation mechanism ensures that the activities of all units in the hidden layer are maintained near their target activity by regulating their bias pa- rameter. Structural plasticity learns set of sparse connections from the input layer to hidden layer by maximizing local greedy information theoretic score.
2
RELATED WORKS
A popular unsupervised learning approach is to train a hidden layer to reproduce the input data as, for example, in AE and RBM. The AE and RBM networks trained with single hidden layer are relevant here since learning weights of the hnput-to-hidden-lap the representations can be stacked on top of each other to extract hierarchical features. However, stacked autoencoders and deep belief nets (stacked RBMs) have typically been used for pre-training procedures followed by end-to-end supervised min-ttning (using backprop) (Erhan et al. 2009). The recently proposed KH model (Krotov & Hopfield 2019) addresses the problem of learning solely with local gradients by learning hidden representations only using an unsupervised method. In this network the input-to-hidden connections provides competition within the hidden layer. For evaluating the representation, the weights are frozen, and a linear classifier trained with labels is used for the final classification. Our approach shares some common features with the KH model, c.g. learning hidden representations solely by unsupervised methods, and evaluating the representations by separate classifier (Illing et al. (0019) provides an extensive review of methods with similar goals).
connections relies on local gradients, and
trained and additional (non-plastier lateral inhibition
All the aforementioned models employ either competition within the hidden layer (KH), or feedback connections from hidden to input (RBM and AE). The BCPNN uses only the feedforward connec- tions, along with an implicit competition via a local softmax operation, the neural implementation of which would be lateral inhibition.
also observed that, for unsupervised learning, having sparse connectivity in the feedforward connections performs better than full connectivity (Iling et al. (019). In addition to the unsuper- vised methods, networks employing supervised learning like convolutional neural networks (CNNs) force a fixed spatial filter to obtain this sparse connectivity (Lindsay) 2020). The BCPNN model takes an alternate approach where, along with learning the weights of the feedforward connections, which is regarded biological synaptic plasticity, sparse connectivity between the input and hid- den layer is learnt simultancously, in analogy with the structural plasticity in the brain (Butz et al. 2009).
2

---

Under review as a conference paper at ICLR 2021
3 BAYESIAN CONFIDENCE PROPAGATION NEURAL NETWORK
Here describe the BCPNN network architecture and update rules (Sandberg al. 2002 Lansner et al., (200).. The feedforward BCPNN architecture contains two layers, referred to as the input layer and hidden layer. A layer consists variable X; (upper case). Each HC, in urr, is composed of a set of MCs representing particular value 2; (lower case) of X,, The probability of X; then a multinomial distribution, defined as p(X; z;), such that Ex, P(X; 24) 1. In the neural network, the activity of the MC is interpreted as X; 2;), and the activities of all the MCs inside HC sum to one.
HCs, each of which represents discrete random
Since the network is probabilistic graphical model (see Fig. T we can compute the posterior of a target HC in the hidden layer conditioned on all the source HCs in the input layer. We will use a's and y's for referring the HCs in the input and hidden layer respectively. Computing the exact posterior p(Y;|X1,,... Xw) over the target HC is intractable, since it scales exponentially with the number of units. The naive Bayes assumption p(X1,... XN(Y,) write the posterior as follows:
I P(),)Y)) allows us to
pI XN) P(Y IL ((X())))
& p(y)) (X.))))
p(XI, XN)
i=1
When the network is driven by input data {X1, XNJ = {of .. x we can write the posterior probabilities of target MC in terms of the source MCS
N pluslep...., xR) x p(u,) II P< wy) = p(u)) IIIII p(r; )( (e=ep)
N
(2)
where I(() is the indicator function that equals I if its argument is true, and zero otherwise. We have written the posterior the target MC as a function of the source MCs (all x; S). The log posterior can be written as:
N
log logply; P IR. x logp(1;)
I( ( P) logg(sssuu
(3)
Since the posterior is linear in the indicator function data sample, T(x;= P can approximated by its expected value defined <<x;)= = p(z, =x:"), Except for (xx)). all the terms in the posterior are functions of the marginals p(y;) and p(z;, y₁ We define the terms bias B(y,) = log p(y;) and
weight (I,, 8;) log (z in analogy with artificial neural networks.
The inference step to calculate the posterior probabilities of the target MCs conditioned on the input sample given by the activity update equations:
Myy) B(Us) +
r. (x.))o(E11)))
(4)
1=1
exp(h(y;))
T(,)) Ee exp(((((xx))
(5)
where h(y;) is the total input received by each target MC from which the posterior probability ((;)) = p(u, ..., xQ) is recovered by softmax normalization of all MCs within the HC.
)
As each data sample is presented, the learning step updates the marginal probabilities, weights, and biases follows:
3

---

Under review as a conference paper at ICLR 2021
A
T(2))
Y
p(Y)
II (X, 1=1
T (x;))
Figure 1: Schematic of the BCPNN architecture with three input HCs and hidden HC. A. The probabilistic graphical model illustrating the generative process with each random variable (circle) representing a HC. The input HCs observable (shaded circle) and the hidden HC is latent (open circle). The naive Bayes assumption renders the likelihood of generating inputs factorial. B. The equivalent neural network where the input HCs are binary (gray circle), hidden HC is multinomial (gray box), and hidden MCs within the HC are the discrete values of the hidden variable (gray circle inside the box).
dp(u;) T(y)) - p(yj)
(6)
dt
dp(xi, Jri, Ni) = T ((Z, ( (11) - p( U5)
Tp
(7)
dt
B(U;) = Kg log p(U;)
(8)
w(xs, My) log p(z,, U4)
p(u;)
The terms Ty is learning time constant and ky is the bias gain. The set of Equations 4-9 define the update and learning equations of the BCPNN architecture. In this work, we use the abstract non- spiking model of BCPNN for the purpose of representation learning. The network for unsupervised representation learning requires, in addition to the update and learning equations, the following two mechanisms to enable learning representations (Ravichandran et al. 2020).
3.1 BIAS REGULATION
The BCPNN update rule implements Bayesian inference if the parameters are learnt with the source and target layer probabilities available as observations. When the target layer is hidden, we are learning the representations, and cannot expect the update rule to follow Bayesian inference. In fact, we can see that performing learning and inference simultaneously this case. Consider hidden representation with random initialization that assigns some MCs with slightly higher marginal probability p(y;) than others. Learning would then amplify this difference and find parameters that would associate more input samples with the MCs with high p(y,). ), causing the marginals to increase further. One way to cireumvent this effect is to promote MC's with low p(y,) more active the future, like an activity dependent homeostasis process in biological terms (Turrigiano & Nelson 2004).
counter-production in
We use a bias regulation mechanism, where the bias gain ky for each MC (equal if only Bayesian inference is performed) depends p(y;). One motivation for choosing the bias gain is to influence the marginal p(yj) alone without affecting the weight parameters that are responsible for learning the input to hidden mapping. The value of p(y;) is compared with respect to the maximum entropy probability, PMaxEnt = 1/NMc, where NMC is the number of MCs per HC. is worth noting that the maximum entropy is the ideal representation without the input layer since all the MCs have equal marginal probability, and hence acts as the homeostatic reference for bias regulation. The dynamic update of k, with the time constant Ty follows Eq. T0

---

Under review as a conference paper at ICLR 2021
B
Input layer
Knat
7 DMaxent
PMaxEnt
PMaxEnt
10
10
10
10
ply))
Hidden layer
Figure 2: A: Bias regulation mechanism. For generating the figure, k and Em was used. The schematic the network used for unsupervised learning. In this network, the input layer contains nine binary HCs (grey circles the left), and the hidden layer contains three HCs (grey boxes), each of which contains four MCs (grey circles inside the boxes). of connection between input HC and hidden HC is shown blue strip. i.e., M,==1. The input-hidden weights are shown as yellow dots and are present only when a connection already exists.
existence
dkg TB
Piaa Ens
+ (kihalf 1)
4
ki
(10)
dt
p(1))
PaaanEn
PararRa and drops sharply negative values when p(y,) is below Pat (see Fig. 2A). The rate of this drop is controlled using the hyperparameter Khaif, defined as the value of gain kg = khalf at p(y;) = PMaee Had
The mechanism maintains the value of gain kg at around when p(y,) >
3.2 STRUCTUAL PLASTICITY
Structural plasticity builds a set receptive fields for the hidden layer from the input. We define Boolean variable M, for the connection from the ith input HC jth hidden HC as active, Mj 1, or silent, Mj = 0 Each Mij initialized randomly with probability Pas. where setting PM to low value ensures patchy and sparse connectivity (Fig. 3). Once initialized, the total number of active incoming connections to each hidden HC is fixed whereas the outgoing connections from a source HC can be changed. The mutual information (MI) between the ith input HC and jth hidden HC is estimated from the BCPNN weights: Lij Σx. Σn, P(x,, 35) w Uj). Each input HC normalizes the MI by the total number of active outgoing connections:
I I+ Ex Ma
i,
(1I)
Since the total number active incoming connections is fixed, each hidden HC greedily maximizes the Jij it receives by removing the active connection with the lowest I.y (set M,y from 1 to and adds an inactive connection with the highest I, (set My from 0 to 1). We call this operation flip and use parameter Nytis set the number of flips made per training epoch.

---

Under review as a conference paper at ICLR 2021
Table 1: BCPNN model parameters
Symbol
Value
Descrption
Nhc
30
Number of HCs in hidden layer
NMC
100
Number of MCs per HC in hidden layer
At
0.01
Time-step
p
10
Mean of poisson distribution for initializing MCs Number of time-steps per sample
N Nsample
5
N Nepock
5
Number of epochs of unsupervised learning
Khalf
-100
Bias gain when marginal is PMaxEnt/2
O
0.1
Multiplier for learning time-constant
T
0.1
Multiplier for bias gain time-constant
PM
0.1
Probability of connections from input to hidden layer
Slipss
16
Number of flips per epoch for structural plasticity
4 EXPERIMENTS
Here we describe the experimental setup for the BCPNN and three other related models for unsuper- vised learning, as discussed in section 2. Next, we introduce a supervised classification layer trained on the representations learnt by the four models. Finally, we qualitatively study these representations and provide quantitative performance results of the models in supervised classification.
We ran the experiments the MNIST handwritten digits dataset (LeCun) 1998). MNIST contains Ntrain = 60000 training and Ntest = 10000 test images of 28x28 handwritten digits. The images were flattened to 784 dimensions and the grey-scale intensities were normalized to the range [0.1]. The images act as the input layer for the models.
4.1 MODELS
We considered four network architectures: BCPNN (c.f. section 3), AE, RBM and, KH. All the models had one hidden layer and 3000 hidden units.
BCPNN The BCPNN network had hidden layer with 30 HCs and 100 MCs per HC. Each sam- ple was clamped to the input layer for Nsampte iterations of time-step At, and the training was performed for Nepoch epochs of the training set. The time constants 7 and T were scaled by the total training time per epoch, that is, = Th NNtrainnsaaaan Al and Tp = - For tuning the parameters, T To khalf and, Nytips: we used held-out validation set of 10000 samples from the training set, and chose values that maximize the validation accuracy (for details, see Ravichandran et al. (2020)). The entire list of parameters and their values are listed in Table 1. The simulations were performed on code parallelized using MPI on 2.3 GHz Xeon ES processors and the training process took approximately two hours per run.
Tp Nerain Sarmpteeammmeeeee Al.
KH The KH network was reproduced from the original work using the code provided by (Krotov Hopfield 2019). We kept all the parameters as originally described, except for having 3000 hidden units instead of 2000, to be consistent in the comparison with other models.
RBM For the RBM network, used sigmoidal units for both input and hidden layer. The weights were trained using the Contrastive Divergence algorithm with one iteration of Gibbs sampling (CD- 1) (Hinton 2012). The learning rate was set as 0.01 and the training done in minibatches of 256 samples for 300 epochs.
AE For the AE network, we used sigmoidal units for both hidden layer and reconstruction layer and two sets of weights, for encoding from input to hidden layer and another for decoding from hidden to reconstruction layer. The weights were trained using the Adam optimizer and L2 reconstruction loss with additional L. sparsity loss the hidden layer. The sparsity loss coeffi-

---

Under review as a conference paper at ICLR 2021
A
B
BCPNN
10
Linear classifier
10 KH
10
OOOOOO
10'
RBM
000
OOO
O0O
000 000
-1
10
BCPNN
KH
RBM
AE
100
-15
0
weight
Histogram of weights from the input layer to hidden layer. The horizontal axis has the minimum to maximum value of the weights as the range, and the vertical axis is in log scale. B. Schematic of the four unsupervised learning models under comparison and the supervised classifier. The dotted lines imply we use the representations of the hidden layer input for the classifier.
Figure 3:
cient was determined as X=le-7 samples. The training was in minibatches of 256 samples for 300 epochs.
7 by maximizing the accuracy of a held-out validation set of 10000
4.2 RECEPTIVE FIELD COMPARISON
As can be observed in Fig. A the distribution of weight values considerably differs across the networks examined in this work. It appears that the range of values for BCPNN corresponds to that reported for AE, whereas for KH and RBM, weights lie a far narrower interval centered around O. Importantly, BCPNN has by far the highest proportion of zero weights (90%), which renders the connectivity truly sparse.
In Fig. 4 visualize the receptive fields of the four unsupervised learning networks. Firstly, is straightforward to see that the receptive fields of all the networks differ significantly. The RBM (Fig. (C) and AE (Fig. 4D) have receptive fields that highly localized and span the input space, char- acteristic of distributed representations. The KH model (Fig. 4B) has receptive fields that resemble the entire image, showing both positive and negative values over the image, as a result of Hebbian and anti-Hebbiaa learning Krotov & Hopfield (019). Generally, local representations like mixture models and competitive learning, as opposed distributed representations, tend have receptive fields that resemble prototypical samples (Rumelhart & Zipser 1985). With this distinction in mind, the receptive fields the BCPNN should be closely examined (Fig. 4A). The receptive fields of HCs (first column) are localized and span the input space, much like distributed representation. Within each HC however, the MCs have receptive fields (each row) resembling prototypical samples, like diverse sets of lines and strokes. This suggests that the BCPNN representations are "hybrid", with the higeer-level HCs coding distributed representation, and the lower level MCs coding local representation.
4.3 CLASSIFICATION PERFORMANCE
For all the four models of unsupervised learning, employed the same linear classifier for predict- ing the labels (see Fig. 33. This allowed us to consistently evaluate the representations learned by the different models. The linear classifier considers the hidden layer the input and the MNIST labels as the output. The output layer consists of softmax units for the 10 labels. The classifier's weights trained by stochastic gradient descent with the Adam optimizer (Kingma & Ba 2014) using cross-entropy loss function. The training procedure used minibatches of 256 samples and total of 300 training epochs.
The results of the classification are shown Table II. All the results presented here are the mean and standard deviation of the classification accuracy over random runs of the network. We performed

---

Under review as a conference paper at ICLR 2021
B
5
3
2
3
2
8
27
6
7
4
3
7
0
8 3 4 7
6 3
4
3
5
2
8 4 5
3
8
6
4
5
8
6
6
7
$
0
1
3
2
6
4
2
8
7
4
9
3
2
4
9
D
Figure 4: Receptive fields of different unsupervised learning methods. For each model, the positive and negative values are normalized, such that blue, white, and red represent the lowest, zero, and highest value of weights. A. BCPNN: Each row corresponds constituent MCs of BCPNN. First column shows the receptive field HC (black means M,y=1). The remaining columns show the receptive field of nine randomly chosen MCs out of 100 MCs within the HC. B. KH, C. RBM, D. AE: Receptive fields of 60 randomly chosen hidden units out of 3000.
randomly chosen HC and the
Table 2: Accuracy comparison
Model
Tuned parameters
Accuracy (train)
Accuracy (test)
BCPNN KH
Tp =0.1, -0.11 Khol 100
100.00 + 0.00
97.77 + 0.12
See Krotov & Hopfield 2019
98.75 + 0.01
97.39 + 0.06
RBM AE
0.01 X =
08.92 0.04
97.67 = 0.10
100.00 + 0.00
97.78 = 0.09
three independent comparisons BCPNN with KH, RBM, and AE using the Kruskal- Wallls H test. BCPNN outperforms KH (==0.02), while there is statistical difference with RBM (p=0.28) AE (p=0.30).
5 DISCUSSION
We have evaluated four different network models that can perform unsupervised representation learning using correlation based biologically plausible local learning rules. We made our assessment relying the assumption that the saliency of representations is reflected in their class dependent separability, which can be quantified by classification performance, similar to Illing et al. (2019) and Krotov & Hopfield 2019). Learning representations without supervised fine-tuning is harder task
'This is lower than the test accuracy of 98.54% reported by Krotov & Hopfield (2019). This is due to differences in the classifier used: exponentiated ReI.U activation along with an exponentiated loss function. We instead used simpler softmax activation and cross-entropy loss.
8

---

Under review as a conference paper at ICLR 2021
compared similar networks with end-to-end backprop, since the information about the samples' corresponding labels cannot be utilized. Consequently, representations learnt with unsupervised methods cannot be expected to offer better class separability than the classification performance reported by supervised end-to-end approaches. We show that the investigated unsupervised meth- ods score remarkably similar around 97.7%, which is only slightly worse compared to the 98.5% % accuracy of networks with one hidden layer trained with end-to-end backprop (LeCun et al., 1998).
We also showed that the recently proposed BCPNN model performs competitively against other unsupervised learning models. The modular structure of the BCPNN layer led to "hybrid" repre- sentations that differ from the well-known distributed and local representations. In contrast to the minibatch method of other unsupervised learning methods, learning in BCPNN was chosen to re- main incremental using dynamical equations, since such learning is biologically feasible and useful in many autonomous engineering solutions. Despite the slow convergence properties of an incre- mental approach, BCPNN required only 5 epochs of unsupervised training, in comparison to 300 epochs for AE and RBM, and 1000 epochs for KH. The incremental learning, along with modular architecture, sparse connectivity, and scalability of BCPNN is currently also taken advantage of in dedicated VLSI design (Stathis al. 2020
One important difference between current deep learning architectures and the brain concerns the abundance of recurrent connections in the latter. Deep learning architectures rely predominantly on feedforward connectivity. A typical cortical area receives only around 10% of synapses from lower order structures, e.g. thalamus, and the rest from other cortical areas (Douglas & Martin, 2007]. These feedback and recurrent cortical connections are likely involved in associative mem- ory, constraint-satiifaction e.g. for figure-ground segmentation, top-down modulation and selective attention (Douglas & Martin 2007). Incorporating these important aspects of cortical computation can play a key role in improving our machine learning models and approaches.
It is important to note that the unsupervised learning models discussed in this work are proof-of- concept designs and not meant directly model some specific biological system or structure. Yet, they may shed some light on the hierarchical functional organization of e.g. sensory processing streams in the brain. Further work will focus on extending our study to multi-layer architectures.
REFERENCES
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions pattern analysis and machine intelligence, 35(8):1798-188) 1828, 2013.
Simon Benjaminsson, Peter Fransson, and Anders Lansner. novel model-free data analysis tech- nique based on clustering in mutual information space: application to resting-state state fmri. Fron- fiers systems neuroscience, 4:34, 2010.
Pierre Berthet, Jeanette Hellgren Kotaleski, and Anders Lansner. Action selection performance of a reconfigurable basal ganglia inspired model with bebbian-bayesine go-nogo connectivity. Frontiers behavioral neuroscience, 6:65, 2012.
Markus Butz, Florentin Wörgötter, and Arjen van Ooyen, Activity deeeede- structural plasticity. Brain research reviews, 60(2):287-305, 2009.
Rodney J Douglas and Kevan AC Martin. Recurrent neuronal circuits in the neocortex. biology, 17(13):446-R500, 2007.
Current
Kenji Doya Shin Ishii, Alexandre Pouget, and Rajesh PN Rao. Bayesian brain: approaches to neural coding. MIT press, 2007.
Probabilistic
Dumitru Erhan, Yoshua Bengio, Aaron Courville, Pierre-Antoine Manzagol, Pascal Vincent, and
Samy Bengio. Why does unsupervised pre-training help deep discriminant learning? 2009.
Florian Fiebig and Anders Lansner. A spiking working memory model based on hebbian short-term potentiation. Journal of Neuroscience, 37(1):33-9, 2017.
Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. Deep learning, volume 1. MIT press Cambridge, 2016.

---

Under review as a conference paper at ICLR 2021
Geoffrey E Hinton. A practical guide training restricted boltzmann machines. In Neural networks: Tricks of the trade, pp. 599-619, Springer, 2012.
Bernd Illing, Wulfram Gerstner, and Johanni Brea. Biologically plausible deep learning-but how far can we go with shallow networks? Neural Networks, 118:90-101, 2019.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Dmitry Krotov and John Hopfield. Unspervissed by competing hidden units. Proceedings of the National Academy of Sciences, 116(16):7733-7331 2019.
Anders Lansner. Associative memory models: from the cell-assembly theory to biophysically de- tailed cortex simulations. Trends in neurosciences, 32(3):178-186, 2009.
Anders Lansner, Simon Benjaminsson, and Christopher Johansson. From ann to biomimetic infor- mation processing. In Biologically Inspired Signal Processing for Chemical Sensing, pp. 33-43. Springer, 2009.
Yann LeCun. The mnist database of handwritten digits. hllp://yann. lecun. com/exdb/mmisl// 1998.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11)22278-2324 1998.
Timothy P Lillicrap. Adam Santoro, Luke Marris, Colin J Akerman, and Geoffrey Hinton. Back- propagation and the brain. Nature Reviews Neuroscience, pp. 1-12, 2020.
Grace Lindsay. Convolutional neural networks as a model of the visual system: past, present, and future. Journal of Cognitive Neuroscience, pp. 1-15, 2020.
Mikael Lundqvist, Pawel Herman, and Anders Lansner. Theta and gamma power increases and alpha/beta power decreases with memory load in an attractor network model. Journal of cognitive neuroscience, 23100333008-300 2011.
Vernon Mountcastle. The columnar organization of the neocortex. Brain: a journal of neurology, 120(4):701-722, 1997.
Roland Orre, Anders Lansner, Andrew Bate, and Marie Lindquist. Bayesian neural networks with confidence estimations applied to data mining. Computational Statistics & Data Analysis, 34(4): 473-493, 2000.
Cengiz Pehlevan and Dmitri Chklovskii. Neuroscience-inspinn online unsupervised learning algorithms: Artificial neural networks. IEEE Signal Processing Magazine, 36(6):88-96. 2019.
Naresh Balaji Ravichandran, Anders Lansner, and Pawel Herman. Learning representations in
bayesian confidence propagation neural networks. In International Joint Conference on Neural Networks (IJCN),, 2020.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. arXiv preprint arXiv: 2014.
Kathleen S Rockland. Five points on columns. Frontiers Neuroanatomy, 4:22, 2010.
David E Rumelhart and David Zipser. Feature discovery by competitive learning. Cognitive science, 9():75-112, 1985.
Anders Sandberg, Anders Lansner, Karl Magnus Petersson, and Ekeberg. A bayesian attractor network with incremental learning. Network: Computation in neural systems, 13(2):179-194, 2002.
Martin Schain, Simon Benjaminsson, Katarina Varnäs, Anton Forsberg, Christer Halldin, Anders Lansner, Lars Farde, and Andrea Varrone. Arterial imput function derived from pairwise corre- lations between pet-image voxels. Journal of Cerebral Blood Flow Metabolism, 3058):1058- 1065, 2013.
10

---

Under review as a conference paper at ICLR 2021
Dimitrios Stathis, Chirag Sudarshan, Yu Yang, Matthias Jung, Christian Weis, Ahmed Hemani, Anders Lansner, and Norbert Wehn. ebraini:: a kw realtime custom 3d dram integrated asic implementation of biologically plausible model of human scale cortex. Journal of Signal Processing Systems, pp. 1-21, 2020.
Philip Tully, Matthias H Hennig, and Anders Lansner. Synaptic and nonsynaptic plasticity approx- imating probabilistic inference. Frontiers in synaptic neuroscience, 6:8, 2014.
Gina G Turrigiano and Sacha B Nelson. Homeostatic plasticity in the developing nervous system.
Nature reviews neuroscience, 5(2)997-077, 2004.
James CR Whittington and Rafal Bogacz, Theories of enror back-propagation in the brain. Trends in cognitive sciences, 23(3):235-255)) 2019.


Reviews:
[{"review_id": "y3cd0C6bQfK", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper evaluated four unsupervised learning approaches (BCPNN, KH, RBM, AE) by training a supervised classification layer on top of the hidden representation. Specifically, the authors qualitatively compared the receptive fields and quantitatively compared the classification performance across four models. The authors emphasized the advantages of BCPNN since it applies biologically plausible local learning rules and requires fewer epochs for convergence.\n\nOverall, the comparison was fair and solid. The description of the BCPNN model in section 3 was clear and comprehensive. But the authors did not provide sufficient details of key mechanisms in the other three models (especially the KH model, which also used brain-like learning rules). The detailed introduction of the other three models should be an important component since this is a comparative study. The results were clearly stated, but the insignificant difference in the classification accuracy comparison (Table 2) can hardly lead to a reliable conclusion about which unsupervised method is better. And it would be better if the authors could provide more interpretations about the \"hybrid\" receptive fields of HCs and MCs in BCPNN (Fig. 3A).\n\n##### Specific comments and questions:\n1. In the original KH model (Krovtov & Hopfield, 2019), they also tested the classification accuracy on the MNIST dataset, and the result reached an error rate of 1.46% with 2000 hidden states. This is better than the reproduced result shown here (97.39% accuracy) with 3000 hidden states. Is this accuracy drop caused by a different setting of hyperparameters? Then is it fair to say that \"BCPNN outperforms KH\"?\n2. Could you provide more explanation on Eq. 11? Why this dynamic update of $k_{\\beta}$ could be used as a desired bias regulation?\n3. When the number of total hidden units is fixed, what would be the effect of changing the ratio $\\frac{N_{MC}}{N_{HC}}$ in BCPNN?\n4. The \"hybrid\" structure of BCPNN provides interesting receptive field results in Fig. 3A. Is this structure generalizable to a model with multiple hidden layers?\n\n##### Minor:\nPage 4. in 3.1 Bias regulation. Typo: Eq. 6 should be Eq. 11. And \"the value of gain $k_{\\beta}$ at around 1 when $P(y_j) - \\frac{p_{MaxEnt}}{4}$,\" missing $\\gg 0$ here?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "kodbT9JI86", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThe Bayesian Confidence Propagating Neural Network has recently been extended to the case of unsupervised learning (Ravichandran et al., IJCNN, 2020). This paper compares this extension to restricted Boltzmann machines, autoencoders, and a biologically plausible model proposed by (Krotov & Hopfield, PNAS, 2019) on the MNIST dataset. For evaluation the authors consider the learned receptive fields and the classification performance of a linear classifier. The paper is very similar to (Ravichandran et al., IJCNN, 2020) but with an extended experimental section.\n\n\nPositives:\n\n+ Biologically plausible methods for unsupervised learning are an interesting area of research.\n\n+ There has been relatively little research on structural plasticity.\n\n\nConcerns:\n\n- The paper does not introduce anything new but merely compares existing methods.\n\n- The comparison is not an extensive study, but limited to one dataset, MNIST, and few alternative proposals, of which only the KH model is deemed 'brain-like'.\n\n- There seems to be something off with the experimental results. Krotov & Hopfield report a better test accuracy of 98.54% in their original paper in spite of using less hidden units.\n\n- BCPNN's performance is mediocre. It is even outperformed by random shallow networks with fixed, localized, random & random Gabor filters in the hidden layer (Illing et al, Neural Networks, 2019)\n\n- Lacking performance could be excused by greater biological plausibility as a neuroscientific contribution, which is however not the case here. As the authors themselves state, their model is 'abstract' (page 4) and not a neural implementation but merely 'uses implicit competition via a local softmax operation, the neural implementation of which would be lateral inhibition'.\n\n\nMinor comments:\n\n$\\pi(x_i)$ in Eq (6) is never introduced.\n\nThe line above Eq (11) should probably refer to (11) not (6).\n\nThere's a typo in the sentence after Eq (11).\n\nThe hybrid representation might be interesting. Is it any more biological than the well-known distributed and local representations?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "PlpnYIYfN53", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Systematic investigation of biologically inspired algorithms and architectures is a very important research topic. \n\nThe paper investigates Bayesian Confidence Propagating Neural Networks (BCPNN) on learning unsupervised representations on MNIST dataset. It presents a comprehensive comparison of four different commonly used unsupervised methods. \n\nThe strong merit of BCPNN approach is the nice receptive fields of the hypercolumns (HC) and minicolumns (MC) learned by the proposed algorithm. As the authors point out, the advantage of the proposed algorithm is that it is able to produce sparse and highly localized (in the image plane) receptive fields for the MCs. Also, those filters look much cleaner than the counterparts of the classical algorithms considered in figure 3.  Additionally, the authors demonstrate that their representations stand in line with previously published proposals in terms of the classification accuracy. \n\nThe main weakness of this work is that the proposed method has only been investigated on MNIST and in one-layer architectures. At the same time, given the novelty of the approach, I think it deserves attention even in this simplest setting considered in this manuscript. \n\nSmall comments: \n1. Section4 (3rd line) should be “learned” \n2. There are some misprints around equation 11, such as the use of p(y) vs. P(y) is inconsistent. Also, it seem that the word “large” is missing following the formula in the line after equation 11. \n3. I also find panel B in figure 2 confusing. The way it is presented makes it look like the authors have combined the outputs of four networks to feed into the classifier, while in reality those four networks were evaluated one by one. \n4. It would be better to designate a new variable for the left hand side of equation 12, since I_ij is already taken. \n\nPost Rebuttal: \n\nThank you for the response. I have read the discussion with other reviewers. A small comment: while I agree that it is reasonable to keep the classifier the same for all the models (softmax with cross entropy) for a fair comparison, I disagree that the activation function for the first layer should be kept as ReLU in the KH model. In fact KH explain that this is a suboptimal choice in Fig. 4 of their original paper. Using powers of ReLUs should increase the KH accuracy. Overall, I think that this is a nice paper, and I am inclined to keep my initial score. \n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "-BW0o95zIy-", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper proposes a set of biologically plausible update rules that can be used to compute latent representations. The paper presents a number of heuristics to set hyper-parameters and train the proposed model. The model is compared to RBMs, auto-encoders and another recently proposed biologically-plausible model.\n\nPros:\n- The method explores a new kind of model motivated by having biologically plausible update rules.\n- The experimental results include an interesting comparison of the learned features.\n\nCons:\n- The paper places emphasis on interpreting the update rules as Bayesian confidence propagation. Yet, the underlying probabilistic graphical model is not clearly described. It would be useful to have a figure in the paper that describes the model, including whether the model is directed or undirected, which variables are observed or unobserved, if the model is directed what is the generative model, etc.\n- Two assumptions are mentioned about the graphical model : \\\\( P(X_1,..,X_N) = \\prod_i P(X_i) \\\\), and \\\\( P(X_1,..,X_N|Y_j) = \\prod_i P(X_i|Y_j) \\\\). Unless I misunderstood, the first assumption is saying that each dimension in the input data \\\\(X\\\\) is independent of the others. This is a very strong assumption and makes the model weak. The paper does not include an explanation about why these assumptions make sense, or how they influence the model's inductive bias relative to other probabilistic models.\n- The derivation of the update rules is also unclear. The approximation that the indicator function \\\\(I(x_i=x^D_i)\\\\) can be replaced by its expected value \\\\(P(x^D_i)\\\\) is hard to understand. In particular, I could not understand how to parse Eq (4) which has a sum over \\\\(x_i\\\\). Is this still dependent on the input, given that the indicator function has been replaced by its expectation ? It would be good to have a more clear derivation of the update rules starting from the probabilisitic model.\n- The paper makes some very general assertions that do not add to the point being made. For example, \"One disadvantage of probabilistic models is that the known methods do not scale well in practice.\" Models such as VAEs are probabilistic models which scale quite well to high-dimensional inputs and large amounts of data.\n- In the introduced model, each HC seems to have a similar representational capacity as a softmax hidden unit. Therefore, instead of comparing to RBMs and AEs with sigmoid units, comparisons to RBMs and AEs with softmax hidden units will be more relevant.\n \nOverall, the paper can be improved along two directions : making the probabilistic interpretation more clear (specifying the graphical model clearly, deriving update rules), and doing experiments with softmax hidden units (so that the only thing changing is the update rules, and not the model architecture).\n\n---------------\nPost-rebuttal\n\n- Figure 1 is helpful in understanding the model. Thank you for adding that.\n- The additional experiments are also appreciated. This seems to indicate that it's not just the architecture but the learning rules that make the BCPNN model work well.\n- It would also be helpful to visualize the features learned by softmax units in RBMs and AEs and see if this results in a similar pattern of HCs encoding broad regions and MCs encoding variations within those regions.\n- It seems that the RBMs and AEs were not trained using any sparsity penalty. For example, Page 11 of https://www.cs.toronto.edu/~hinton/absps/guideTR.pdf and https://web.stanford.edu/class/cs294a/sparseAutoencoder.pdf. Having a target sparsity can have a significant impact on the learned features. Higher sparsity makes the features look more like stroke like and localized, and less spread out all over the visual field (as they do in Fig 4, C and D).  \n- Based on the additional experiments, I will be increasing my score to 5. However, given that the main contribution of the paper is a comparative study, the paper can add value by doing a more thorough comparison against variants of AEs and RBMs that have otherwise similar properties (such as keeping the HC-MC (softmax) architecture and sparsity levels the same).", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper conducts a comparison between a small set of models (4 in total) for unsupervised learning. Specifically, the authors focus on comparing Bayesian Confidence Propagating Neural Networks (BCPNN), Restricted Boltzmann Machines (RBM), a recent model by Krotov & Hopfield (2019) (KH), and auto-encoders (AE). The authors compare trained weight distributions, receptive field structures, and linear classification on MNIST using the learned representations. The first two comparisons are essentially qualitative comparisons, while on classification accuracy, the authors report similar accuracy levels across the models.

This paper received mixed reviews. Reviewers 4 and 5 felt it did not contribute enough for acceptance, while Reviewers 2 & 3 were more positive. However, as noted by a few of the reviewers, this paper does not appear to achieve much, and provides very limited analysis and experiments on the models. It isn't introducing any new models, nor does it make any clear distinctions between the models examined that would help the field to decide which directions to pursue.  The experiments add little insight into the differences between the models that could be used to inform new work. Thus, the contribution provided here is very limited. 

Moreover, the motivations in this paper are confused. In general, it is important for researchers at the intersection of neuroscience and machine learning to decide what their goal is when building and or comparing models. Specifically, is the goal: (1) finding a model that may potentially explain how the brain works, or (2) finding better machine learning tools?

If the goal is (1), the performance on benchmarks is less important. However, clear links to experimental data, such that experimental predictions may be possible, are very important. That's not to say that a model must be perfectly biologically realistic to be worthwhile, but it must have sufficient grounding in biology to be informative for neuroscience. However, in this manuscript, as was noted by Reviewer 4, the links to biology are tenuous. The principal claim for biological relevance for all the models considered seems to be that the update rules are local. But, this is a loose connection at best. There are many more models of unsupervised learning with far more physiological relevance that are not considered here (see e.g. Olshausen & Field, 1996, Nature; Zylberberg et al. 2011, PLoS Computational Biology; George et al., 2020, bioRxiv: https://doi.org/10.1101/2020.09.09.290601). It is true that some of these models use non-local information, but given the emerging evidence that locality is not actually even a strict property in real synaptic plasticity (see e.g. Gerstner et al., 2018, Frontiers in Neural Circuits; Williams & Holtmaat, 2018, Neuron; Banerjee et al., 2020, Nature), an obsession with rules that only use pre- and post-synaptic activity is not even clearly a desiderata for neuroscience.

If the goal is (2), then performance on benchmarks, and some comparison to the SotA, is absolutely critical. Yet, this paper does none of this. Indeed, the performance achieved with the four models considered here is, as noted by Reviewer 4, very poor. In contrast, there have been numerous advances in unsupervised (or "self-supervised") learning in ML in recent years (e.g. Contrastive Predictive Coding, SimCLR, Bootstrap Your Own Latent, etc.), all of which achieve far better results than the four models considered here. Thus, the models being compared here cannot inform machine learning, as they do not appear to provide any technical advances. Of course, some models may combine goals (1) & (2), e.g. seeking increased physiological relevance while also achieving decent benchmark performance (see e.g. Sacramento et al., 2018, NeurIPS), but that is not really the situation faced here, as the models considered have little biological plausibility (as noted above) and achieve poor performance at the same time.

Altogether, given these considerations, although this paper received mixed reviews, it is clearly not appropriate for acceptance at ICLR in the Area Chair's opinion.
Decision: Reject