Neural Networks with Fixed Binary Random Projections Improve Accuracy in Classifying Noisy Data
Channel: BVM Conference
Date: 25/06/2021

The trend of Artificial Neural Networks becoming "bigger" and "deeper" persists. Training these networks using back-propagation is considered biologically implausible and a time-consuming task. Hence, we investigate how far we can go with fixed binary random projections (BRPs), an approach which reduces the number of trainable parameters using localized receptive fields and binary weights. Evaluating this approach on the MNIST dataset we discovered that contrary to models with fully-trained dense weights, models using fixed localized sparse BRPs yield equally good performance in terms of accuracy, saving 98% computations when generating the hidden representation for the input. Furthermore, we discovered that using BRPs leads to a more robust performance - up to 56% better compared to dense models - in terms of classifying noisy inputs.

References:
Yang Z., Schilling A., Maier A., Krauss P. (2021) Neural Networks with Fixed Binary Random Projections Improve Accuracy in Classifying Noisy Data. In: Palm C., Deserno T.M., Handels H., Maier A., Maier-Hein K., Tolxdorff T. (eds) Bildverarbeitung für die Medizin 2021. Informatik aktuell. Springer Vieweg, Wiesbaden. https://doi.org/10.1007/978-3-658-33198-6_51