Performance of Neural Network Image Classification on Mobile CPU and GPU

Aalto University, P.O. Box 11000, FI-00076 Aalto www.aalto.fi Author Sipi Seppälä Title Performance of Neural Network Image Classification on Mobile CPU and GPU School School of Science Master’s programme Computer, Communication and Information Sciences Major Computer Science Code SCI3042 Supervisor Professor Antti Ylä-Jääski Advisors M.Sc. Teemu Kämäräinen, Dr. Matti Siekkinen Level Master’s thesis Date 2018-04-20 Pages 86  In tasks like image classification, which is a well-known problem in computer vision, deep learning convolutional neural networks can even achieve human-level prediction accuracy. Although high-accuracy deep neural networks can be resource-intensive both to train and to deploy for inference, with the advent of lighter mobile-friendly neural network model architectures it is finally possible to achieve real-time on-device inference without the need for cloud offloading. The inference performance can be further improved by utilizing mobile graphics processing units which are already capable of general-purpose parallel computing. This thesis measures and evaluates the performance aspects – execution latency, throughput, memory footprint, and energy usage – of neural network image classification inference on modern smartphone processors, namely CPU and GPU. The results indicate that, if supported by the neural network software framework used, hardware acceleration with GPU provides superior performance in both inference throughput and energy efficiency – whereas CPU-only performance is both slower and more power-hungry. Especially when the inference computation is sustained for a longer time, running CPU cores at full speed quickly reaches the overheat-prevention temperature limits, forcing the system to slow down processing even further. The measurements show that this thermal throttling does not occur when the neural network is accelerated with a GPU. However, currently available deep learning frameworks, such as TensorFlow, not only have limited support for GPU acceleration, but have difficulties dealing with different types of neural network models because the field is still lacking standard representations for them. Nevertheless, both of these are expected to improve in the future when more comprehensive APIs are developed.Artificial neural networks are a powerful machine learning method, with impressive results lately in the field of computer vision. In tasks like image classification, which is a well-known problem in computer vision, deep learning convolutional neural networks can even achieve human-level prediction accuracy. Although high-accuracy deep neural networks can be resource-intensive both to train and to deploy for inference, with the advent of lighter mobile-friendly neural network model architectures it is finally possible to achieve real-time on-device inference without the need for cloud offloading. The inference performance can be further improved by utilizing mobile graphics processing units which are already capable of general-purpose parallel computing. This thesis measures and evaluates the performance aspects – execution latency, throughput, memory footprint, and energy usage – of neural network image classification inference on modern smartphone processors, namely CPU and GPU. The results indicate that, if supported by the neural network software framework used, hardware acceleration with GPU provides superior performance in both inference throughput and energy efficiency – whereas CPU-only performance is both slower and more power-hungry. Especially when the inference computation is sustained for a longer time, running CPU cores at full speed quickly reaches the overheat-prevention temperature limits, forcing the system to slow down processing even further. The measurements show that this thermal throttling does not occur when the neural network is accelerated with a GPU. However, currently available deep learning frameworks, such as TensorFlow, not only have limited support for GPU acceleration, but have difficulties dealing with different types of neural network models because the field is still lacking standard representations for them. Nevertheless, both of these are expected to improve in the future when more comprehensive APIs are developed.