Question: Can you address discussion and experiments on learning with larger/more complex datasets?
Evidence: We have described SoftHebb, a highly biologically plausible neural algorithm that is founded on a Bayesian ML-theoretic framework. The model consists of elements fully compatible with conventional ANNs. It was previously not known which plasticity rule should be used to learn a Bayesian generative model of the input distribution in ANN WTA networks. Moreover, we showed that Hard WTA networks and neurons with other activation functions can be described within the same framework as variations of the probabilistic model. This theory could provide a new foundation for normative Hebbian ANN designs with practical significance. For example, SoftHebb's properties are sought-after by efficient neuromorphic learning chips. It is unsupervised, local, and requires no error or other feedback currents from upper layers, thus solving hardware-inefficiencies and bio-implausibilities of backpropagation such as weight-transport and update-locking. Surprisingly, it surpasses backpropagation even in accuracy, when training time and network size are limited. In a demonstration that goes beyond the common greedy-training approach to such networks, we achieved update-unlocked operation in practice, by updating the first layer before the input's full processing by the next layer. It is intriguing that, through its biological plausibility, emerge properties commonly associated with biological intelligence, such as speed of learning, and robustness to noise and adversarial attacks. Significant robustness emerges without specialized defences. Furthermore, SoftHebb tends to not merely be robust to attacks, but actually deflect them as specialized SOTA defences aim to do.
Here, we explored SoftHebb's applicability on several datasets. We measured its accuracy on MNIST, Fashion-MNIST, and CIFAR-10 in preliminary results, and we reported a functional convolutional SoftHebb network that improves accuracy on the significantly harder dataset of CIFAR-10. The convolutional implementation could become the foundation for deeper networks and complex problems. Ultimately, this could provide insights into the role of WTA microcircuits in larger networks in cortex with localized receptive fields, similar to area V1 of cortex. All in all, the algorithm has several properties that are individually interesting and novel, and worth future extension. Combined, however, SoftHebb's properties shown in this work may already enable certain small-scale but previously-impossible applications. For example, fast, on-line, unsupervised learning of simple tasks by edge sensing devices, operating in noisy conditions, with a small battery and only local processing, requires those algorithmic properties that we demonstrated here.
REPRODUCIBILITY STATEMENT Some of our main contributions are theoretical and their proof is fully reproducible by following the rigorous derivations in the main text and in the appendices. Our experimental results are based on standard machine-learning techniques, while we also describe the hyperparameters and experimental protocols. Moreover, we provide Python code with specific instructions to reproduce the main experiments. calculate weight change ∆w
Here, we explored SoftHebb’s applicability on several datasets. We measured its accuracy on MNIST, Fashion-MNIST, and CIFAR-10 in preliminary results, and we reported a functional convolutional SoftHebb network that improves accuracy on the significantly harder dataset of CIFAR-10. The convolutional implementation could become the foundation for deeper networks and complex problems.****Finally, we performed preliminary tests on two more difficult datasets, namely Fashion-MNIST (Xiao et al., 2017), which contains grey-scale images of clothing products, and CIFAR-10 (Krizhevsky et al., 2009), which contains RGB images of animals and vehicles. We did not tune the Hebbian networks’ hyper-parameters extensively, so accuracies on these tasks are not definitive but do give a good indication. On F-MNIST, the SoftHebb model achieved a top accuracy of 87.46% whereas a hard WTA reached a similar accuracy of 87.49%. A supervised MLP of the same size achieved a test accuracy of 90.55%. SoftHebb’s generative interpolations (Fig. 3B) are reconfirmed on the F-MNIST dataset, as is its robustness to attacks, whereas, with very small adversarial perturbations, the MLP drops to an accuracy lower than the SoftHebb model (dashed lines in Fig. 2). On CIFAR-10’s preliminary results, the hard WTA and SoftHebb achieved an accuracy of 49.78% and 50.27% respectively. In every tested dataset, it became clear that SoftHebb learns faster than either backpropagation or a hard WTA, by observing the loss and the learned features as in Fig. 1 C & D. The fully-connected SoftHebb layer is applicable on MNIST because the data classes are well-clustered directly in the feature-space of pixels. That is, SoftHebb’s probabilistic model’s assumptions (Def. 2.1) are quite valid for this feature space, and increasing the number of neurons for discovering more refined sub-clusters does help. However, for more complex datasets, this approach alone has diminishing returns and multilayer networks will be needed. Towards this, we implemented a first version of a convolutional SoftHebb, with an added supervised classifier. In these early results, conv-SoftHebb achieved 98.63% on MNIST, and 60.30% on CIFAR-10. This confirms that conv-SoftHebb is functional, but we did not systematically optimize its accuracy or explore its other strengths. This feasibility is evidence that SoftHebb’s strengths may soon also apply to larger networks and datasets.****Figure 3: Examples generated by the adversarial pair PGD attacker/SoftHebb model. SoftHebb’s inherent tendency to deflect the attack towards truly different classes is visible. This tendency can be repurposed to generate interpolations between different classes of the data distribution, a generative property previously unknown for such simple networks.
Answer: We have now provided results on CIFAR-10 and with convolutional SoftHebb, and added a section dedicated to extensibility