DCT-CryptoNets: Scaling Private Inference in the Frequency Domain

Authors: Arjun Roy, Kaushik Roy
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: homomorphic encryption, deep neural networks, privacy-preserving machine learning, frequency-domain learning

Abstract:
The convergence of fully homomorphic encryption (FHE) and machine learning offers unprecedented opportunities for private inference of sensitive data. FHE enables computation directly on encrypted data, safeguarding the entire machine learning pipeline, including data and model confidentiality. However, existing FHE-based implementations for deep neural networks face significant challenges in computational cost, latency, and scalability, limiting their practical deployment. This paper introduces DCT-CryptoNets, a novel approach that operates directly in the frequency-domain to reduce the burden of computationally expensive non-linear activations and homomorphic bootstrap operations during private inference. It does so by utilizing the discrete cosine transform (DCT), commonly employed in JPEG encoding, which has inherent compatibility with remote computing services where images are generally stored and transmitted in this encoded format. DCT-CryptoNets demonstrates a substantial latency reductions of up to 5.3$\times$ compared to prior work on benchmark image classification tasks. Notably, it demonstrates inference on the ImageNet dataset within 2.5 hours (down from 12.5 hours on equivalent 96-thread compute resources). Furthermore, by *learning* perceptually salient low-frequency information DCT-CryptoNets improves the reliability of encrypted predictions compared to RGB-based networks by reducing error accumulating homomorphic bootstrap operations. DCT-CryptoNets also demonstrates superior scalability to RGB-based networks by further reducing computational cost as image size increases. This study demonstrates a promising avenue for achieving efficient and practical private inference of deep learning models on high resolution images seen in real-world applications.

Decision: accept (poster)
Reviewer scores: 5, 8, 6

Reviews and discussion:
<thread>
creation date:
1730607968815

thread_type:
official_review

<thread_object>
creator: reviewer_eori

<content>
summary:
The paper presents DCT-CryptoNets, a novel framework that enhances privacy-preserving deep learning by utilizing the Discrete Cosine Transform (DCT) to operate in the frequency domain, significantly reducing latency and computational costs associated with fully homomorphic encrypted neural networks (FHENNs). By focusing on low-frequency information, DCT-CryptoNets improve accuracy and reliability in encrypted predictions while demonstrating superior scalability with increasing image resolution. The work addresses key challenges in existing FHE-based neural networks and emphasizes the importance of optimizing cryptographic and quantization parameters for practical applications in secure image processing.

strengths:
1. **Frequency-Domain Optimization**: DCT-CryptoNets leverage the Discrete Cosine Transform (DCT) to focus on low-frequency components of images, which enhances the model's ability to capture perceptually salient information while reducing computational complexity and improving accuracy compared to traditional RGB-based networks.

2. **Reduced Latency and Improved Scalability**: The proposed method achieves significant latency reductions (up to 5.3×) during inference, especially on large datasets like ImageNet, while demonstrating superior scalability as image resolution increases. This makes DCT-CryptoNets more efficient for real-world applications of privacy-preserving deep learning.

3. **Enhanced Reliability through Reduced Error Accumulation**: By minimizing the need for homomorphic bootstrap operations, DCT-CryptoNets reduce the accumulation of approximation errors, leading to more reliable predictions and improved encrypted accuracy. This addresses challenges faced by earlier fully homomorphic encryption (FHE) schemes.

weaknesses:
I have a question about the threat model setting here, why the model is trained locally?

If the client could train the model by themselves, why does they do inference locally.

Or if they want to deploy that encrypted model (key from model trainer) on the cloud for other clients as service. How does the key management should be solved?

questions:
see weakness
</content>
</thread_object>

</thread>
<thread>
creation date:
1730651025552

thread_type:
official_review

<thread_object>
creator: reviewer_sg3z

<content>
summary:
This paper presents a very promising strategy for achieving a practical and computationally efficient FHE-based (CKKS) private inference of deep learning models applied to high resolution images.     In a novel approach, the technique presented relies as a starting point on a Discrete Cosine Transform representation of the image.  It also employs a quantization aware training (QAT) framework.  Thorough comparative benchmarking of accuracy and latency for RGB vs DCT techniques a various image resolutions and at various levels of retained low-frequency components (of the DCT) are presented.  

The paper establishes both a detailed methodology and new "high-water mark" performance capability for ML inference generation for high resolution images.  Though performance is less pronounced on smaller images (32x32) it is the capability on larger images that is most important for future practical applications.

strengths:
Novel DCT based insight for unstructured data yielding excellent computational advantages for high resolution images.

Thorough benchmarking and attention to reproducible science.

Excellent comparative analysis of this CKKS based technique relative to competing TFHE approaches.

Excellent and comprehensive list of references that chronicle the current state of the art and prior advances.

weaknesses:
I find no obvious weaknesses.

I would emphasize to the reader perhaps new to the field ( in Section 3.2 on page 6)  that model training is done in the plaintext domain. (even though this is most evident in Figure 3 presented on Page 14.

questions:
Are there any special scaling or normalization techniques that need to be considered when considering the YCrCb color space components of an image?

Can GPUs be used to any advantage in this approach?

Are there any privacy preservation guarantees or "leakage" guarantees that could be developed around this approach.  (I realize this may be challenging mathematically.)
</content>
</thread_object>

</thread>
<thread>
creation date:
1731103305925

thread_type:
official_review

<thread_object>
creator: reviewer_qw2q

<content>
summary:
This paper introduces DCT-CryptoNets, which performs neural network (CNN) inference entirely using TFHE-based homomorphic encryption in the frequency domain. Unlike hybrid protocols that limit homomorphic encryption to just linear operations, and require MPC for nonlinear operations, this approach leverages TFHE to perform both linear and nonlinear operations under homomorphic encryption. This enables private inference outsourcing to the server without requiring client interaction for intermediate computations. The authors have shown significant speedup over prior TFHE-based implementation (SHE, NeurIPS'19) on CIFAR-10 and ImageNet datasets.

strengths:
$\bullet$ Performing TFHE-based inference in the frequency domain allows the usage of lower-resolution inputs without compromising accuracy. This approach significantly decreases both FLOPs and nonlinear operations (ReLU), while also requiring fewer bootstrapping operations. This results in substantial speedup benefits. More importantly, this makes it feasible to perform inference on larger input images, enhancing the practical applicability (such as semantic segmentation) of homomorphic encryption in neural network inference.


$\bullet$ The authors report ciphertext accuracy, specifically on the ImageNet-1K scale, distinguishing this work from most private inference papers, which typically report plaintext accuracy under the assumption that there is no accuracy loss when operations are conducted in field arithmetic.

$\bullet$ The experimental results are extensive and include a detailed sensitivity analysis of the cryptographic hyper-parameters.

weaknesses:
$\bullet$ The lack of sufficient algorithmic contributions and research insights makes it less suitable for the ML conference. Operating in the frequency domain for private inference benefits is not a novel concept (see [1,2]). Also, the usage of quantization-aware training for lower-frequency components is simply an engineering tweak. 

Thus, a more fitting venue for this work might be a cryptography-focused conference. 

$\bullet$ Moreover, the practicality of HE-only private inference remains questionable, especially when compared to hybrid protocol-based approaches. For example, Cheetah [3] achieves ImageNet-1K inference on ResNet-50 in 80.3 seconds in a LAN setting and 134.7 seconds in a WAN setting. Thus, the primary motivation for pursuing HE-only inference appears to be the benefit of non-interactive private inference.




1. Song et al., ENSEI: Efficient secure inference via frequency-domain homomorphic convolution for privacy-preserving visual recognition, CVPR 2020.

2. Li et al. Falcon: A fourier transform based approach for fast and secure convolutional neural network predictions, CVPR 2020.

3. Huang et al., Cheetah: Lean and fast secure Two-Party deep neural network inference, USENIX Security 2022.

questions:
See the weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731987875895

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Revision

comment:
We sincerely thank all of the reviewers for their insightful comments and the opportunity to strengthen our work. In the following, we present the changes made to this revision which are marked in red.

-   We have included more clarity on the differences between DCT-CryptoNets and other frequency-domain private inference techniques (mainly leveraging homomorphic discrete Fourier transform - HDFT) in Sections 2.2 and 2.5. This is in response to qW2q comment on other frequency-domain private inference techniques.

-   We have added a new appendix section (Section A.1) which discusses different threat models for which DCT-CryptoNets protects against from a server, client and communication standpoint. We also briefly go into mitigation strategies for certain attacks. This is in response to SG3z’s question on privacy preservation guarantees that could be built upon DCT-CryptoNets.
   
-   We have included more detail in a pre-existing appendix section (Section A.2) which elaborates on how key management should be handled in FHE-based private inference applications. This is in response to EoRi’s question on how key’s should be managed in DCT-CryptoNets.

We hope that these new changes as well as the individual review responses enhance the clarity, rigor, and impact of this work. We are more than happy to answer any follow-up questions.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper proposes a new technique DCT-Cryptonets in order to be able to perform cryptographically private (using Fully Homomorphic Encryption) inference on neural networks, particularly for convolutional neural networks. The main contribution is the method to perform operations in frequency space which has benefits both computationally and to a lesser extent statistically due to focus on low-dimensional Fourier transforms. The authors back up their new method by an extensive empirical evaluation and in particular report results on a scale that have not been reported before.