Package: larq-compute-engine
Larq Compute EngineLarq Compute Engine (LCE) is a highly optimized inference engine for deploying
extremely quantized neural networks, such as
Binarized Neural Networks (BNNs). It currently supports various mobile platforms
and has been benchmarked on a Pixel 1 phone and a Raspberry Pi.
LCE provides a collection of hand-optimizedTensorFlow Litecustom operators for supported instruction sets, developed in inline assembly or in C++
using compiler intrinsics. LCE leverages optimization techniques
such astilingto maximize the number of cache hits,vectorizationto maximize
the computational throughput, andmulti-threading parallelizationto take
advantage of multi-core modern desktop and mobile CPUs.Larq Compute Engine is part of a family of libraries for BNN development; you can also check outLarqfor building and training BNNs andLarq Zoofor pre-trained models.Key FeaturesEffortless end-to-end integrationfrom training to deployment:Tight integration of LCE withLarqand
TensorFlow provides a smooth end-to-end training and deployment experience.A collection of Larq pre-trained BNN models for common machine learning tasks
is available inLarq Zooand can be used out-of-the-box with LCE.LCE provides a customMLIR-based model converterwhich
is fully compatible with TensorFlow Lite and performs additional
network level optimizations for Larq models.Lightning fast deploymenton a variety of mobile platforms:LCE enables high performance, on-device machine learning inference by
providing hand-optimized kernels and network level optimizations for BNN models.LCE currently supports 64-bit ARM-based mobile platforms such as Android phones
and Raspberry Pi boards.Thread parallelism support in LCE is essential for modern mobile devices with
multi-core CPUs.PerformanceThe table below presentssingle-threadedperformance of Larq Compute Engine on
different versions of a novel BNN model called QuickNet (trained on ImageNet dataset, released onLarq Zoo)
on a Raspberry Pi 4 Model B at 1.5GHz (BCM2711) board, aPixel 1 Android phone (2016), and aMac Mini with M1 ARM CPU:ModelTop-1 AccuracyRPi 4B 1.5GHz, 1 thread (ms)Pixel 1, 1 thread (ms)Mac Mini M1, 1 thread (ms)QuickNetSmall59.4%27.716.84.0QuickNet63.3%45.025.55.8QuickNetLarge66.9%77.044.29.9For reference,dabnn(the other main BNN library) reports an inference time of 61.3 ms forBi-RealNet(56.4% accuracy) on the Pixel 1 phone,
while LCE achieves an inference time of 41.6 ms for Bi-RealNet on the same device.
They furthermore present a modified version, BiRealNet-Stem, which achieves the same accuracy of 56.4% in 43.2 ms.The following table presentsmulti-threadedperformance of Larq Compute Engine on
a Pixel 1 phone and a Raspberry Pi 4 Model B at 1.5GHz (BCM2711)
board:ModelTop-1 AccuracyRPi 4B 1.5GHz, 4 threads (ms)Pixel 1, 4 threads (ms)Mac Mini M1, 4 threads (ms)QuickNetSmall59.4%12.18.91.8QuickNet63.3%20.812.62.5QuickNetLarge66.9%31.722.83.9Benchmarked on 2021-06-11 (Pixel 1), 2021-06-13 (Mac Mini M1), and 2022-04-20 (RPi 4B) with LCE customTFLite Model Benchmark Tool(seehere)
with XNNPack enabled
and BNN models with randomized inputs.Getting startedFollow these steps to deploy a BNN with LCE:Pick a Larq modelYou can useLarqto build and train your own model or pick a pre-trained model fromLarq Zoo.Convert the Larq modelLCE is built on top of TensorFlow Lite and uses the TensorFlow LiteFlatBuffer formatto convert and serialize Larq models for inference. We provide anLCE Converterwith additional optimization passes to increase the speed of execution of Larq models on supported target platforms.Build LCEThe LCE documentation provides the build instructions forAndroidand64-bit ARM-based boardssuch as Raspberry Pi. Please follow the provided instructions to create a native LCE build or cross-compile for one of the supported targets.Run inferenceLCE uses theTensorFlow Lite Interpreterto perform an inference. In addition to the already available built-in TensorFlow Lite operators, optimized LCE operators are registered to the interpreter to execute the Larq specific subgraphs of the model. An example to create and build an LCE compatible TensorFlow Lite interpreter for your own applications is providedhere.Next stepsExploreLarq pre-trained models.Learn how tobuildandtrainBNNs for your own
application with Larq.If you're a mobile developer, visitAndroid quickstart.See our build instructions for Raspberry Pi and 64-bit ARM-based boardshere.Try ourexample programs.AboutLarq Compute Engine is being developed by a team of deep learning researchers and engineers at Plumerai to help accelerate both our own research and the general adoption of Binarized Neural Networks.