The novelty of our approach is that it is only based on HE, while the existing interactive approach combines HE with 2 PC protocols. Hence, we reduce communication complexity caused by 2 PC protocols. Existing (non-interactive) HE-based SNNI frameworks face three inherent drawbacks, which our approach will address. * Firstly, existing HE schemes only support addition and multiplication, while other operators, such as comparison, are not readily available and are usually replaced by evaluating expensive (high-degree) polynomial approximation [9; 10]. Hence, cur rent HE-based SNNI frameworks **cannot practically evaluate non-linear activation functions**, e.g., ReLU function, which are widely adopted in NNs. The solution for this issue is to replace these activation functions with (low-degree) polynomial approximations, e.g., square function. However, this replacement degrades the accuracy of the NN [11] and requires a costly re-training of the NN. Moreover, training NNs using the square activation function can lead to strange behavior when running the gradient descent algorithm, especially for deep NNs. For example, gradient explosion or overfitting phenomena may occur [12]. This is because the derivative of the square function, unlike the ReLU function, is unbounded. This paper proposes a novel protocol to evaluate the ReLU function, HeReLU, which addresses the challenge of evaluating ReLU in current HE-based SNNI frameworks. * Secondly, in all existing HE schemes, ciphertexts contain noise, which will be accumulated after running homomorphic operations, and at some point, the noise present in a ciphertext may become too large, and the ciphertext is no longer decryptable. A family of HE supporting a predetermined number of homomorphic operators is called leveled HE (LHE). LHE is widely adopted in SNNI frameworks (LHE-based SNNI frameworks). Therefore, an inherent drawback of existing LHE-based SNNI frameworks is that they **limit the depth of NNs**. The bootstrapping technique, an innovative technique proposed by [PERSON] [13], can address this issue. The bootstrapping technique produces a new ciphertext that encrypts the same message with a lower noise level so that more homomorphic operations can be performed on the ciphertext. Roughly speaking, bootstrapping homomorphically decrypts the ciphertext using encryption of the secret key, called the bootstrapping key, then (re-)encrypts the message with the bootstrapping key. This requires an extra security prerequisite, termed the circular security assumption. The assumption implies that disclosing the encryption of the secret key is presumed to be secure. The circular security assumption remains poorly studied and poorly understood. Proving circular security for the HE schemes remains an open problem. Furthermore, bootstrapping is generally considered an expensive operation, so one usually tries to avoid it as much as possible [14]. This paper proposes a novel protocol to refresh ciphertext in HE schemes, HeRefresh, to address the noise growth issue in HE schemes. Our protocol is much faster than bootstrapping (the empirical comparison results are shown in Section 5.2) and does not require circular security. * Thirdly, LHE-based SNNI suffers from **huge computational overhead**. The computational cost of LHE, for example, grows dramatically with the number of levels of multiplication that the scheme needs to support. This means LHE-based SNNI is impractical in the case of deep NNs (equivalent to a very large multiplication level). Interestingly, HeRefresh naturally reduces the size of LHE schemes' parameters, hence significantly reducing computational overhead (details in Section 5.3). The non-interactive LHE-based SNNI approach cannot efficiently address the three above challenges. A natural approach that appears in the literature is interactive LHE-based approaches [11; 15; 16]. There are two branches in interactive LHE-based approaches.