Title: Exploring weightless neural networks: From logic gates to convolutional lookup tables
Primary area: datasets and benchmarks
Keywords: Differentiable Weightless Neural Networks, Logic Gate Networks, Performance Comparison, Edge Computing, Internet of Things

Abstract: Increasing the intelligence of everyday objects is facilitated by miniaturized machine learning (ML) models which operate accurately in resource-constrained environments. Applications abound across the Internet-Of-Things (IOT), the development of personal medical devices, and for consumer electronics such as smart phones and augmented reality glasses. Weightless neural networks (WNN), such as Logic Gate Networks (LGN) and Look Up Table networks (LTN), represent a class of models which accelerate ML inference by orders of magnitude while retaining high predictive accuracy. While small-scale LGNs and LTNs have demonstrated rapid inference capabilities, their training and robustness characteristics are not well reported which hinders the pursuit of large models which are suitable for real-world applications. This paper fills this gap by comparing LGNs, LTNs, Multi-Layer Perceptrons (MLPs), and their convolutional counterparts on the basis of test accuracy, training time, and robustness to noise across key model variations. Presenting challenges with training LGNs and LTNs helps ML engineers design solutions to expedite data-processing workflows. The study's findings reveal that: (1) LGNs and LTNs generally take longer to train than traditional models, though comparable test accuracies can be achieved with relatively larger parameter counts, (2) LGNs are most robust to salt-and-pepper noise, (3) while using learnable interconnects between layers generally increase the maximum accuracy of both LGNs and LTNs, the mean performance of LTNs decreases and the number of model parameters increases for both model types by orders of magnitude. (4) The optimal bit depth depends more on the dataset than on the model type. Next steps are proposed to pursue applications of WNNs to real-world devices across healthcare, robotics, and consumer electronics.

TL;DR: Differentiable logic-gate and LUT networks can outperform traditional DNNs on edge devices, with faster inferences and lower power, but scaling beyond CNNs is hard due to large parameter counts and training times, hindering real-world, large models.