High performance deep learning on resource constrained platforms
Keywords: machine learning systems
winograd
convolutional neural networks
graph neural networks
machine learning
hardware-aware machine learning
deep learning
efficient machine learning
quantization

After a decade of accelerated progress in the different areas of machine learning (ML), it has become virtually impossible to not interact with ML-powered applications and services in our everyday lives. Recommender systems, computational photography, speech translation or, getting directions are only a handful of examples of what can be done from our smartphones or other smart devices, sometimes battery-powered, present in the places where we live, work and, socialize. However, despite the known benefits of running such applications on-device (e.g. privacy, works offline, real-time potential), their reliance on millions (even billions!) of parameters and their increasingly complex execution patterns, have prevented these from leaving the Cloud, where model inference takes place. As a result, only a fraction of ML-powered applications, such as keyword spotting and next-word prediction, remain lightweight enough in terms of memory and compute footprint to run on-device.
Motivated by the urge to deploying more advanced ML-powered applications on commodity devices, the research community has primarily relied on three sets of techniques to lower their runtime costs: pruning, which results in model compression and acceleration during inference by discarding model parameters; quantization, which offers model size savings by representing parameters with fewer bits and, often translates into faster inference when using lower precision arithmetic; and, by making use of lightweight network architecture designs which better map to the target platform's hardware. However, in many cases the aforementioned techniques are insufficient to deliver the required compute, memory and energy savings to run most ML-powered applications on-device. What further optimization avenues would enable the deployment and the efficient execution of complex and better performing models on constrained devices?
This thesis studies this challenging problem from three different perspectives: first, by identifying a new paradigm for compression frameworks where model parameters are generated on-the-fly, enabling inference acceleration by reducing the impact of data movement costs; second, by solving the inherent problem of numerical degradation that has prevented the adoption and deployment of fast Winograd convolutions for models making use of integer arithmetic; and third, by proposing mechanisms that enable the learning of quantized graph neural networks for a variety of applications operating on irregular grids. Each of these contributions materialize into three different frameworks, namely unzipFPGA, wiNAS and Degree-Quant, which are comprehensively evaluated through a wide range of experiments and ablation studies. These contributions push the state of the art on three different fronts and can be used in conjunction with other existing acceleration techniques to collectively enable the deployment of high performance deep learning applications on resource constrained devices.