###### Abstract  
Enabling deep learning inferences on resource-constrained devices is important for intelligent Internet of Things. Edge computing makes this feasible by outsourcing resource-consuming operations from IoT devices to edge devices. In such scenarios, sensitive data shall be protected while transmitted to the edge. To address this issue, one major challenge is to efficiently execute inference tasks without hampering the real-time operation of IoT applications. Existing techniques based on complex cryptographic primitives or differential privacy are limited to either efficiency or model accuracy. This paper addresses this challenge with a lightweight interactive protocol by utilizing low-latency IoT-to-edge communication links for computational efficiency. We achieve this with a new privacy-preserving scalar product evaluation technique that caters to the unique requirements of deep learning inference. As compared to the state-of-the-art, our solution offers improved trade-offs among privacy, efficiency, and utility. Experimental results on a Raspberry Pi 4 (Model B) show that our construction can achieve over 14\(\times\) acceleration versus local execution for AlexNet inference over ImageNet. The proposed privacy-preserving scalar-product-evaluation technique can also be used as a general primitive in other applications. privacy; Internet of Things; convolutional neural networks; deep learning; computation outsourcing; edge computing; privacy-preserving scalar product +
Footnote †: journal: Medical Computing  
cloud has become a popular solution. Most of the existing cloud-based methods utilize advanced cryptographic primitives such as Homomorphic Encryption (HE) [9] and Multiparty Computation (MPC) [10; 11] to achieve privacy-preserving computation outsourcing. However, these techniques are still far from efficient even when outsourcing simplified deep networks, despite recent promising progress. For example, CryptoNets [12] needs over 570 s for the inference of a simplified CNN (5 layers) on MNIST. More importantly, existing \"cloud-based\" techniques usually do not consider bandwidth constraints or other latencies between the data source (e.g., IoT devices) and the cloud. For emerging low-latency applications, one trend is to shift from cloud computing to edge computing [13; 14] so that computationally expensive operations can be offloaded to nearby edge devices instead of the remote cloud to minimize the communication delay. Unlike cloud systems that are massively parallel and extremely elastic, local edge servers are usually equipped with relatively limited computing resources, though advanced computing modules such as GPUs are still available. To support secure and efficient offloading of deep networks, one promising approach is to employ customized lightweight encryption (e.g., one-time pad (OTP) or random noises) instead of using heavy cryptographic primitives. To this end, a recent study [15] proposes an edge-assisted interactive scheme that allows IoT devices to outsource only linear operations to edge devices. This is based on the observation that linear operations contribute to most of the computational complexity in popular deep networks. For example, in AlexNet [3], with five convolutional layers and three fully-connected layers, the non-linear operations only count for less than 2 million floating-point operations (FLOPs)--less than 0.1% of the total of 2 billion FLOPs--while over 99.9% are linear operations. Therefore, one practical approach is to let IoT devices only offload linear operations to edge devices while computing non-linear ones locally. To this end, Ref. [15] encrypts input data with one-time random noises and is extremely efficient because of the minimal computational overheads incurred by the OTP-like encryption. The linearity of the outsourced operations ensures that the added noises can then be efficiently eliminated from the scalar products returned by the edge.