Title: HeLutNet: Extremely Fast Privacy-preserving Inference in Milliseconds via LUT-based Machine Learning Models
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: privacy preserving, fully homomorphic encryption

Abstract: Fully Homomorphic Encryption (FHE) holds immense promise for enabling privacy-preserving machine learning (ML) inference, allowing computations on encrypted data without needing to expose sensitive information with untrusted third parties, such as cloud providers. Despite its potential, a major obstacle to FHE's widespread adoption is its prohibitively high computational overhead and slow inference speeds. For instance, inference on a single CIFAR-10 image can take up to 30 seconds on a modern 56-core CPU, making them impractical for real-world applications.
To overcome this, we introduce HeLutNet, a framework that achieves ultra-low-latency FHE-based ML inference. For the first time, we propose accelerating FHE-based ML by using lookup table (LUT) based ML models. LUT-based ML models are well-suited for FHE implementation, as they can represent complex non-linear functions without deep architectures, enabling high accuracy on diverse datasets with shallow networks. Our approach converts a PyTorch-trained, LUT-based ML model into an efficient FHE program by implementing LUTs as Single Instruction, Multiple Data (SIMD) integer arithmetic within the BGV FHE scheme. Our evaluation on vision, speech, and tabular datasets demonstrate consistent speed up over the state-of-the-art FHE-based machine learning models. Notably, we reduce MNIST and Fashion-MNIST inference time to just 13 ms on a single-core CPU, demonstrating the potential for practical, real-time deployment of privacy-preserving ML inference system.