The first branch is _solely_ based on LHE, which we call iLHE, and the second branch combines HE and 2 PC protocols, known as the HE-MPC hybrid approach in the literature. n-Graph-HE2 [11] is the only iLHE-based SNNI framework. n-Graph-HE2 adopted client-aided architecture where the server sends encrypted values (before the non-linear activation function) of intermediate layers to the client, and the client decrypts those encrypted values and then evaluates the activation function on intermediate layers' values; finally, the client encrypts those intermediate layers' values (after non-linear activation function) and sends it to the server. It is worth noting that the process of decryption-then-encryption can be considered as a noise refresher. This solution addresses the three mentioned issues of non-interactive LHE-based SNNI. However, nGraph-HE2 leaks the network's information, namely NNs' architecture and all intermediate layers' values. Recent studies have illuminated the potential risks of this information leakage; a malicious client could feasibly reconstruct the entire neural network parameters by exploiting these intermediate values [17; 18]. nGraph-HE2 is an ad hoc solution to address issues of LHE-based SNNI frameworks and is not considered as a separate approach to SNNI in the literature. This paper proposes a novel iLHE-based SNNI framework that addresses the drawbacks of existing non-interactive LHE-based SNNI frameworks while preserving NNs' architecture and not leaking information about intermediate layers' values. We argue that the iLHE-based approach deserves more attention from the research community. The second branch of interactive LHE-based SNNI is the HE-MPC hybrid approach, which wisely combines HE and MPC. In particular, HE schemes were used to compute linear layers, i.e., fully connected layer and convolution layer, while 2 PC protocols, e.g., garbled circuit (GC) [19] and ABY [20], were used to compute an _exact_ non-linear activation function. The major bottleneck of the approach is the communication complexity caused by 2 PC protocols. On the other hand, our framework is an iLHE-based approach, which is _solely_ based on HE. Hence, we naturally avoid expensive 2 PC protocols. To address the three challenges faced by non-interactive LHE-based SNNI, this paper designs two novel iLHE-based protocols. Firstly, we design the HeReLU protocol to evaluate the ReLU activation function, an essential non-linear function widely adopted in NNs. This protocol addresses the first challenge of non-linear function evaluation in HE-based SNNI. Compared to current LHE-based SNNI frameworks, our protocol can _exactly_ evaluate the ReLU function. Compared to existing HE-MPC hybrid frameworks, i.e., Gazelle [15] and MP2 ML [16], our protocol achieves fewer communication rounds (we detail this in Section 5.1). Secondly, we design HeRefresh to _refresh_ HE ciphertexts, which enable further homomorphic operators. This protocol addresses the second challenge of noise accumulation in HE-based SNNI. By using HeRefresh, rather than selecting HE parameters large enough to support the entire NN, the HE's parameters must now only be large enough to support the computation on linear layers between ReLU activation function layers. For instance, to perform secure inference on a seven-layer NN, which contains convolution layers, fully connected layers followed by ReLU activation function layers, our framework (with HeRefresh) requires \(L=3\ll 7\), while current non-interactive LHE-based SNNI frameworks (without HeRefresh) require \(L=7\) (details in Section 5.3). Therefore, HeRefresh can address the third challenge of computational overhead due to large multiplication depth in LHE-based SNNI. In the HE scheme, HeRefresh plays the role of a bootstrapping procedure, which aims to reduce the noise in ciphertexts. Our experiment showed that HeRefresh outperforms bootstrapping by \(300\times\) in running time (we detail this in Section 5.2). Interestingly, HeRefresh and HeReLU can run in parallel.