Learning to Linearize Deep Neural Networks  for Secure and Efficient Private Inference
Authors: Souvik Kundu
Shunlin Lu
Yuke Zhang
Jacqueline Tiffany Liu
Peter Anthony Beerel
Venue: ICLR 2023 poster (2023)
Keywords: Efficient private inference
cryptographic inference
machine learning as a service
efficient cryptographic inference
automated ReLU reduction

Abstract: The large number of ReLU non-linearity operations in existing deep neural networks makes them ill-suited for latency-efficient private inference (PI). Existing techniques to reduce ReLU operations often involve manual effort and sacrifice significant accuracy. In this paper, we first present a novel measure of non-linearity layers’ ReLU sensitivity, enabling mitigation of the time-consuming manual efforts in identifying the same. Based on this sensitivity, we then present SENet, a three-stage training method that for a given ReLU budget, automatically assigns per-layer ReLU counts, decides the ReLU locations for each layer’s activation map, and trains a model with significantly fewer ReLUs to potentially yield latency and communication efficient PI. Experimental evaluations with multiple models on various datasets show SENet’s superior performance both in terms of reduced ReLUs and improved classification accuracy compared to existing alternatives. In particular, SENet can yield models that require up to ∼2× fewer ReLUs while yielding similar accuracy. For a similar ReLU budget SENet can yield models with ∼2.32% improved classification accuracy, evaluated on CIFAR-100.

Published as a conference paper at ICLR 2023
LEARNING TO LINEARIZE DEEP NEURAL NETWORKS FOR SECURE AND EFFICIENT PRIVATE INFERENCE
Souvik Kundu Intel Labs, San Diego USA souvikk. sundu@intell com
Shunlin Lu, Yuke Zhang, Jacqueline T. Liu, & Peter A. Beerel
Department of Electrical and Computer Engineering University of Southern Califoria, Los Angeles, USA (shunling inlu, yukezhan, jt jtliu, paleuuaabbeeellee @erscee)@uuucc.. Susc. edu
ABSTRACT
The large number of ReLU non-linearity operations in existing deep neural net- works makes them ill-suited for latency-efficient private inference (PI). Exist- ing techniques to reduce ReLU operations often involve manual effort and sac- rifice significant accuracy. In this paper, first present novel measure of non-linearity layers' ReLU sensitivity, enabling mitigation of the time-come consuminn manual efforts in identifying the same. Based on this sensitivity, we then present SENet, three-stage training method that for a given ReLU budget, automatically assigns per-layer ReLU counts, decides the ReLU locations for each layer's s activa- tion map, and trains model with significantly fewer ReLUs potentially yield latency and communication efficient PI. Experimental evaluations with multiple models on various datasets show SENet's superior performance both in terms of reduced ReLUs and improved classification accuracy compared to existing alter- natives. In particular, SENet can yield models that require up to ~2x fewer Re- LUS while yielding similar accuracy. For similar ReLU budget SENet can yield models with 2.32% improved classification accuracy, evaluated on CIFAR-100.
INTRODUCTION
With the recent proliferation of several Al-driven client-server applications including image anal- ysis (Litjens et al. 2017). object detection, speech recognition (Hinton et al. 2012). and voice assistance services, the demand for machine learning inference as service (MLaaS) has grown. Simultaneously, the emergence of privacy concerns from both the users and model developers has made private inference (PI) an important aspect of MLaaS. In PI the service provider retains the proprietary models in the cloud where the inference is performed on the client's encrypted data (ciphertexts), thus preserving both model Kundu et al., 2021b) and data-privacy (Yin et al. 2020).
Dataset: CIFAR-100
78
Sk Néwo St.
70
The ligher the better
The lower better
Existing PI methods rely on various cryptographic proto- cols, including homomorphic encryption (HE) (Braker- ski & Vaikuntanathan 2014; Gentry 2009) and additive secret sharing (ASS) Gooldriic et al. 2019) for the lin- ear operations in the convolutional and fully connected (FC) layers. For example, popular methods like Gazelle (Juvekar et al. 20188) DELPHI MMshrra et al. 2020). and
#RetUs (1)
Figure 1: Comparison of various meth- ods in accuracy vs. #ReLU trade-off plot. SENet outperforms the existing ap- proaches with an accuracy improvement of up to 44.5% for similar ReLU budget. Cheetah (Reagen et al. 2021) HE while MiniONN (Liu et al. (017)) and CryptoNAS (Ghodsi) jet al.) 2020) use ASS. For performing the non-linear ReLU operations, the PI methods generally use Y Garbled Circuits (GC) Yao 1986). However, GCs demand orders of magnitude higher latency and communication than the of linear operations, making latency-efficient PI an exceed- ingly difficult task. In contrast, standard inference latency is dominated by the linear operations (Kunuu et al. 2022bb and is significantly lower than that of PI.
This has motivated the unique problem of reducing the number of ReLU non-linearity operations to reduce the communication and latency overhead of PI. In particular, recent literature has leveraged
Part of the work done when the first author graduate student USC.

---

Published as a conference paper at ICLR 2023
Table 1: Comparison between existing approaches in yielding efficient models perform PI. Note,
SENet++ can yield a model that can be switched to sub-models of reduced channel sizes.
Namse
Method
Redicced
Granularity Keduce modebupports dynamic
used
non-linrarrty
dimentsio channel dropping
Irrequiar
Variuus
K
Scalar weight
Stmectree prusing
Varions
Chansel, litee
Spbynx Cho et (02]
NAS
Laver block
GrymaNAS (Ghodu et di (020)
NAS
Lager-block
DEI PETT (Mitrr
NAS PA
Lager-block
SAPENA (1xe
2021)
NAN
Clammel
Desptebuse 1aa at. 3M4
Manual + IIC
Layer bluck
SNL (Cho
regulatized
Channel, pise
Automated
Channel, pixe
SENeit (ours)
Autommned
Channel pixe)
neural architecture search (NAS) to optimize both the number and placement of ReLUs (Ghodsi et al. 2020 Cho et al. (021). However, these methods often cost significant accuracy drop, par- ticularly when the ReLU budget low. For example, with ReLU budget of 86k, CryptoNAS costs ~9% accuracy compared to the model with all ReLUs (AR) present. DeepReDuce (Jha et al., 2021 used careful multi-stage optimization and provided reduced accuracy drop of ~3% at simi- lar ReLU budgets. However, Decp heavily relies on manual effort for the precise removal of ReLUU layers, making this strategy exceedingly difficult, particularly, for models with many layers. A portion of these accuracy drops can be attributed to the fact that these approaches are constrained to remove ReLUs at a higher granularity of layers and channels rather than at the pixel level. Only very recently, (Cho et al. 2022) proposed I -rrggularrzee pixel level ReLU reduction. However, such approaches are extremely hyperparameter sensitive and often do not guarantee meeting a specific ReI.U budget. Moreover, the large number of training iterations required for improved accuracy may not be suitable for compute- limited servers (Mishra et al. 2020).
Our contributions. Our contribution is three-fold. We first empirically demonstrate the relation between layer's sensitivity towards pruning and its associated ReLU sensitivity. Based our ob- servations, we introduce an automated layer-wise ReLU sensitivity evaluation strategy and propose SENet, three-stage training process to yield secure and efficient networks for PI that guarantees meeting target ReLU budget without any hyperpparameter--deeeneee iterative training. In particu- lar, for a given global ReLU budget, we first determine a sensitivity-driven layer-wise non-linearity (ReLU) unit budget. Given this budget, we then present a layer-wise ReLU allocation mask search. For each layer, we evaluate a binary mask tensor with the size of the corresponding activation map for which a 1 or 0 signifies the presence or absence of ReLU umit, respectively. Finally, we use the trained mask to create a partial ReLU (PR) model with ReLU present only at fixed parts of the non-linearity layers, and fine-tune it via distillation from an iso-architecture trained AR model. Importantly, we support ReLU mask allocation both at the granularity of individual pixels and acti- vation channels.
To further reduce both linear and non-linear (ReLU) layer compute costs, we extend our approach to SENet++. SENet++ uses single training loop to train a model of different channel dropout rates (DRs) d, (d+ Σ 1.0) of the weight tensor, where each d, yields a sub-model with a MAC- ReLU budget smaller than or same as that of the original one. In particular, inspired by the idea of ordered dropout (Horvath et al. 202T), we train a PR model with multiple dropout rates (Horvath) iet al. 2021). where each dropout rate corresponds to a scaled channel sub-model having number of channels per layer x the d,- This essentially allows the server yield multiple sub-models for different compute requirements that too via a single training loop, without costly memory footprint. Table compares the important characteristics of our methods with existing alternatives.
We conduct extensive experiments and ablations on various models including variants of ResNet, Wide Residual Networks, and VGG on -00 CIFAR-100, Tiny-ImageNet, and ImageNet datasets. Experimental results show that SENet can yield SOTA iccuracy-ReLU trade-off with an improved accuracy of up to 2.32% for similar ReLU budgets. SENet++ (d, 0.5) can further improve the MAC and ReLU cost of SENet, with an additional saving of 4x and ~2x, respectively.
2 PRELIMINARIES AND RELATED WORK
Cryptographic primitives. We briefly describe the relevant cryptographic primitives in this section.
Additive secret sharing. Given an element xr an ASS of x is the pair ((z) (4)2) = (r-r,r), where r is random element and x = (x)1 + (z)2. Since r is random, the value x cannot be revealed by single share, so that the value x is hidden.
2

---

Published as a conference paper at ICLR 2023
2009) is public key encryption scheme that supports homomorphic operations on the ciphertexts Here, encryption function E generates the ciphertext t of a plaintext m where t = Em. pk), and a decryption function D obtains the plaintext m via m = Dkl,sk), where pk and sk are corresponding public and secret key, respectively. In Pl, the results of linear operations can be obtained homomorphically through my o m2 = D(t, * t2, sk), where represents linear operation, * is its corresponding homomorphic operation, t and to are the ciphertexts of my and m2, respectively.
Homomorphic encryption. HE (Gentry
Garbled circuits. GC (Yoo 1986) allows two parties to jointly compute a Boolcan function ff over their private inputs without revealing their inputs to each other. The Boolean function f is repre- sented as a Boolean circuit C. Here, garbler creates an encoded Boolean circuit C and set of input-correspondent labels through a procedure Garble(C) to send C and the labels to the other party who acts as an evaluator. The evaluator further sends the output label upon evaluation via EVal(C). Finally, the garbler decrypts the labels to get the plain results to share with the evaluator.
Private inference. Similar to (Mishra et al. 2020), in this paper, we focus on semi-honest client- server PI scenario where client, holding private data, intends to use inference service from a server having a private model. Specifically, the semi-hones parties strictly follow the protocol but try to reveal their collaborator's private data by inspecting the information they received. On the other hand, a malicious client could deviate from the protocol.
To defend against various threats existing cryptographic protocols (Mishra et al. 2020 Ghodsi et al., 2020 Lou et al. 2021) rely the popular online-offline topology (Mishra 22.20) where the client data independent component is computed in the offline phase (Juvekar et al. [2018] Mishra
et al. 2020 Ghodsi et al.) [021] Lehmkuhl et al.) 2021). For the linear operations, DELPHI MMishra ct al., 2020) and MInIONN (Liu al.] 2017) move the heavy primitives in HE and ASS to offline
enabling fast linear operations during PI online stage. However, the compute-heavy Eval( of GC keeps the ReLU cost high even at the online stage.
ReLU reduction for efficient PI. Existing works use model designing with reduced ReLU counts via either search for efficient models (Mishra et al.) 2020; Ghodsi et al.) 2020 Lou et al., 202T] Cho et al.] (021)) or manual re-design from an existing model (Jha et al., (021). In particu- lar, SAFENet (Lou et al., 202T) enables more fine-grained channel-wise substitution and mixed- precision activation approximation. CryptoNAS (Ghodsi et al.) 2020) re-designs the neural archi- tectures through evolutionary NAS techniques to minimize ReLU operations. Sphynx (Cho et al. 202T further improves the search by leveraging differentiable macro-search NAS (Liu et al. 201 Sa) in yielding efficient PI models. DeepReDuce (Jha et al., (202I). on the other hand, reduced ReLU models via manual effort of finding and dropping redundant ReLU layers starting from an exist- ing standard model. Finally, a recent work (Cho et al. (022) leveraged 11-reuularization to remove ReLU at the pixel level to yield SOTA accuracy vs non-lineane lineliiearity trade-off. However, the extreme hyperparameter dependence of such methods often provide sub-optimal solution and does not nec- essarily guarantee meeting a target ReLU budget. Moreover, a resource-limine limited server (Mishra et al.) 2020 may not afford costly iterative training (Cho al. 2022) in reducing the ReLU count.
MOTIVATIONAL STUDY: RELATION BETWEEN RELU IMPORTANCE AND PRUNING SENSITIVITY
Existing work to find the importance of a ReLU layer (Tha et al. 2021). ) requires manual effort and is extremely time consuming. In contrast, model pruning literature (Lee et al.) 2018 Kundu et al.) 202 leveraged various metrics to efficiently identify layer's ss sensitivity towards target pruning ratio. In particular, a layer's s praming sensitivity can be quantitatively defined as the accuracy reduc- tion caused by pruning certain ratio of parameters from it (Ding et al. 2019). In particular, recent literature leveraged sparse learning (Ding et al. 2019 Kundu al.
1021a) and used a trained sparse o5 non-zera layer parametera
model evaluate the sensitivity of layer (na:) as the ratio EGEST
totel layer paremeters
Despite significant progress in weight pruning sensitivity, due to the absence of any trainable parameter in the ReLU layer, its sensitivity for a given ReLU budget is yet to be explored. We hypothesize that there may be correlation between layer's 's pruning sensitivity (Kundu et al.) 2021a) and the importance of ReLU and have conducted the following experiments to explore this.
Let us assume an L-layer DNN model & parameterized by O E E. that learns function f.. where m represents the total number of model parameters.
3

---

Published as a conference paper at ICLR 2023
The goal of DNN parameter pruning is to identify and remove the unimportant parameters from DNN and yield a reduced parameter model that has comparable performance to the baseline un- pruned model. As part of the pruning process for given parameter density d. each parameter is associated with an auxiliary indicator variable e belonging mask tensor E {0, 1 } such that only those remain non zero whose corresponding c = I. With these notations, we can formulate the training optimization
min (( o c)), s.t. I|cllo S d x
(1)
where L()) represents the loss function, which for image classification tasks is generally the
cross-entropy (CE) loss. We used a sparse learning framework (Kundu et al. 2021a) to train
a ResNetl8 8 on CIFAR-100 for target d = 0.1 and com- puted the pruning sensitivity of each layer. In particular, as shown in Fig. _ earlier layers have higher pruning sen- sitivity than later ones. This means that achieve close to baseline performance, the model trains later layers' pa- rameters towards zero more than those of earlier layers.
ResNet18 on CIFAR-100
1.0
0.8
20.6
0.4
Pruning sensitivity
Normalized ReLU importance
000.0
We then compared this trend with that of the impor- tance of different ReLU layers as defined in (Jha et al.) 2021]. In particular, we first identified five different modules of ReLU placement in a ResNet18, the pre- basic-block (BB) stem. BBI, BB2, BB3, and BB4. We then created five ResNet18 variants with ReLU non- linearity present only at one of the modules while re- placing non-linearity at the other modules with identity layers. We identify the modules yielding higher accu- racy to be the ones with higher ReLU importance (Jhal al. 202T). We then normalized the importance of a ReLU stage with accuracy Acc as the ratio (Acc ACCmin) (AACC/oz models with all and no ReLUs, respectively.
4
B 101112 131421111111111 Layer
Figure 2: Layer-wise pruning sensitiv- ity (d 0.1) vs. normalized ReLU im- portance. The later layers are less sensi- tive to pruning, and, thus, can afford sig- nificantly more velu-valued weights as opposed to the earlier ones. On the con- trary. later ReLU stages generally have higher importance.
AcCmin) ). Here AcCmaz and AcCmin correspond to the accuracy of
As depicted in Figure 2 the results show that the ReL.U importance and parameter pruning sensitivity of a layer are inversely correlated. This inverse correlation may imply that a pruned layer can afford to have more zero-valued weights when the associated ReI.U layer forces most of the computed activation values to zero.
4 SENET TRAINING METHODOLOGY
As highlighted earlier, for a large number of ReLU layers L,, the manual evaluation and analysis of the candidate architectures become inefficient and time consuming. Moreover, the manual assign- ment of ReLU the pixel level becomes even more intractable because the number of pixels that must be considered, explodes. To that end, we now present SENet, a three-stage automated ReLU trimming strategy that can yield models for given reduced ReLU budget.
4.1 SENSITIVITY ANALYSIS
Inspired by our observations in Section we define the ReLU sensitivity of layer 7 as
Na (1 170²)
is important to emphasize that, unlike ReLU importance, ReLU sensitivity does not require train- ing many candidate models. However, na can only be evaluated for a specific d. We empirically observe that > 0.3 tends to yield uniform sensitivity across layers due to a large parameter budget. In contrast, ultra-low density d < 0.1, costs non-negligible accuracy drops (Liu et al. 2018b Kundu al. 2022a). Based on these observations, we propose quantify ReLU sensitivity with proxy
density of d 0.1.
Moreover, to avoid the compute-heavy pruning process, we leverage the idea of sensitivity evaluation before training (Lee et al., 20188. On a sampled mini batch from training data D. the sensitivity of the jth connection with associated indication variable and vector as ej and ej. can be evaluated as,
4

---

Published as a conference paper at ICLR 2023
Algorithm 1: Layer-wise #ReLU Allocation Algorithm
Data: Global ReLU budget model parameters è, model parameter proxy donsity d.
number of ReLU layers Lr, active Rel.U indicator a E frr
Output: Per-layer ReLU
Ta < evalåctSene (9, for
Mad E Not
end
initVale (Fremiin Ptotali "totall Finat)
while Flotal do
For Lo
10
our www.gnnelllrooorrtaeeo.. IFremain Va!
Vinat: Troral assignipdateReluu (ra Teurs [total)
12
end
13
end
14
Tremove f Ntotal -r
15
while Premave > do t.
Your remnvesselaerpppoiionn fg)
18
Finat P Premore remove@pdateRellur finat F cupssrrmocd Premoce)
19
end
21
return Fiinal
DC)()o(O:D) g,(fo(O:D)) aL(so(c@ 0;D))
(3)
dc,
c=1
lim C D)) L(fa((c de,) θ 6-+
where e is a vector containing all indicator variables. The 0C is an infinitesimal version of AL; de, measuring the impact of a change in Cj from - 6. It be computed using one forward pass for all j at once. We normalize the connection sensitivities, rank them, and identify the top d- d-fraction of connections. We then define the layer sensitivity net the fraction of connections of each layer that are in the top d-fraction. For given global ReLU budget r, we then assign the # ReLU for each layer proportional to its normalized ReLU sensitivity. The details are shown in Algorithm (Fig. 3 as point (). Note r'milll in Algorithm represents the allocated #ReLUs of layer at the end of stage 1, with rnnal representing the set #ReLUs for all the ReLU layers.
4.2 RELU MASK IDENTIFICAION
After layer-wise #ReLU allocation, we identify the ReLU locations in each layer's activation map- In particular, for non-linear layer I, we assign mask tensor M E {0, 1 1 xw' xe' where ht, w and c' represents the height, width, and the number of channels in the activation map. For layer 1, we initialize M with 2 finel assigned I's with random locations. Then we perform a distillation- based training of the PR model performing ReLU ops only at the locations the masks with 1, while distilling knowledge from an AR model of the same architecture (see Fig. 3 point 2). At the end of each epoch, for each layer I, we rank the top-r' locations based on the highest absolute difference between the PR and AR model's post-Rel.U activation output (averaged over all the mini-batches) for that layer, and update the M with I's at these locations. This, on average, de-emphasizes the locations where the Post-ReLU activations in both the PR and AR models are positive. We terminate evalluauatt
mask evaluation once the ReLU m evaluation reaches the maximum mask training epochs or when the normalized hamming distance between masks generated after two consecutive epochs is below a certain pre-defined value. Notably, there has been significant research in identifying important trainable parameters (Savarese et al., 2020) Kusupati et al., 2020); Kundu et al. 0020:20222 Babakniya et al. 2022) through various proxies including magnitude, gradient, Hessian, however, due to the absence of any trainable parameter the ReLU layer, such methods can't be deployed in identifying important ReLU units of layer.
'The identified mask tensor has non zeros irregularly placed. This can be easily extended to the generation of the structured mask, allowing the assignment and removal of mask values the granularity of channels instead of activation scalar (Kundu 2021a).
5

---

Published as a conference paper at ICLR 2023
Layeris ric PuLU Iuaged
Retli meah laelan
Dist lodor
+ Masked RGU
Aethertonn
Rew Budges
The tuansd mudk
Model
Topa Attretinn
Mart evikaation
Fids traveng
Consakricn arruut
ReLl
TrO01 meeieed I11 model
Weigen
Charn
Figure 3: Different stages of the proposed training methodology for efficient private inference with dynamic channel reduction. For example, the model here supports two channel SFs, S and S2- Note, similar to (Horvath al., 2021), for each SF support (BN) layer to maintain separate statistics.
separate batch-normaliizttion
Channel-wise ReLU mask identification. The mask identification technique described above, cre- ates irregular ReLU masks. To support coarser level of granularity where the ReLU removal happens "channel-wise", we now present simple yet effective extension of the mask identification. For a layer 1, we first translate the total non-zero ReLU counts to total non-zero ReLU channels as rf = [ [" k'w We then follow the same procedure as irregular mask identification, however, only keep top-r channels non-zero.
4.3 MAXIMIZING ACTIVATION SIMILARITY VIA DISTILLATION
Once the mask for each layer is frozen, we start our final training phase in which we maximize the similarity between activation functions of our PR and AR models, Fig. 3 point @). In particular, we initialize a PR model with the weights and mask best PR model of stage 2 and allow only the parameters to train. We train the PR model with distillation via KL-divergence loss (Hinton et al.) 2013 Kundu & Sundaresan 202T) from pre-trained AR along with a CE-loss. Moreover, introduce an AR-PR Post-ReLLU activation mismatch (PRAM) penalty into the loss function. This loss drives the PR model to have activation maps that similar that of the AR model.
More formally, let yo and ym represent the m pair of vectorized post-ReLU activation maps of same layer for prr and Par, respectively. Our loss function for the fine-tuning phase is given as
L = (1 A) Cpr (U. + LALKL (a 20a
(0(²²)
B
Tr pr
T
m
or
(4)
P
2
mET
KL-diw loss
PRAM loss
where represents the softmax function with p being its temperature. À balances the importance between the CE and KL divergence loss components, and 3 is the weight for the PRAM loss. Similar to (Zagoruyko & Komodakis, 2016a), we use the 42-norm of the normalized activation maps to compute this loss.
4.4 SENET++: SUPPORT FOR ORDERED CHANNEL DROPPING
To yield further compute-communication benefits, we now present extension of SENet, namely SENet++, that can perform the ReLU reduction while also supporting inference with reduced model sizes. In particular, we leverage the idea of ordered dropout (OD) (Horvath et al.) 2021) to simulta- neously train multiple sub-models with different fractions channels. The OD method is parame- terized by a candidate dropout set D,, with dropout rate values d, E (0, 1]. At selected d, for any layer the model uses a d,-sub-model with only the channels with indices {0, I,.., dd, C.
active, effectively pruning the remaining [[d, C2], CT],... Ct - 1} channels. Hence, during training, the selection of a d,-sub-model with d, < 1.0 E Dr, is form of channel pruning, while d, = 1.0 trains the full model. For each mini-bater of data, we perform forward pass once for each value of d, in D,., accumulating the loss. We then perform a backward pass in which the model parameters updated based on the gradients computed the accumulated loss. We first train AR model with a dropout set D,.. For the ReLU budget evaluation, we consider only the model with d, = 1.0, and finalize the mask by following the methods in Sections 4.1 and 4.2 During the maximizing of activation similarity stage, we fine-tune the PR model supporting the same set D, as that of the AR model. In particular, the loss function for the fine-tuning is the same as 4 for d, = 1.0. For d, < 1.0, exclude the PRAM loss because we empirically observed that adding the PRAM loss

---

Published as a conference paper at ICLR 2023
for each sub-mode on average does not improve accuracy. During inference, +ENet++ models can be dynamically switched to support reduced channel widths, reducing the number of both ReLUs and MACs compared to the baseline model.
EXPERIMENTS
5.1 EXPERIMENTAL SETUP
Models and Datasets. To evaluate the efficacy of the SENet yielded models, we performed ex- tensive experiments on three popular datasets, CIFAR-16 10, CIFAR-100 (Krizhevsky et al. 2009), Tiny-ImageNet (Hnssen 2015), and ImageNef with three different model variants, namely ResNet (ResNet18, ResNet34) (He et al. 2016). wide residual network (WRN22-8) (Zagoruyko & Ko- modakis 2016b), and VGG (VGGI6) (Simonyan & Zisserman 2014). We used PoTorch API to define and train our models on an Nvidia RTX 2080 Ti GPU.
Training Hyperparameters. We performed standard data augmentation (horizontal flip and random cropping with reflective padding) and the SGD opti- mizer for all training. We trained the baseline all- ReLU model for 240, 120, and 60 epochs for CIFAR, Iinn-lmageNet, and ImageNet respectively, with a starting learning rate (LR) of 0.05 that decays by factor of 0.1 at the 62.5%, 75%, and 87.5% training epochs completion points. For all the training we used an weight decay coefficient of 10-4 4. For target ReLU budget, we performed the mask evaluation for 150, 100, and 30 epochs, respectively, for the three dataset types with the the training prematurely terminates when less than 5% of the total #ReLU masks change their po- sitions. Finally. we performed the Rest-ReLU activation similarity improvement for 180, 120, and 50 epochs, for CIFAR, Tiny- ImageNet, and ImageNet respectively. Also, unless stated otherwise, we use À = 0.9, and 3 = 1000 for the loss described in Eq. 44 Further details our training hyper-parameter choices are provided in the Appendix. In Table we report the accuracy averaged over three runs.
Table 2: Runtime and communication costs of linear and ReLU operations for 15-bit fixed-point model parametets/inputs and 31-bit ReL.Us (Mishra al.) 2020).
"Operation
Made
Rumtime
Comm, KB)
Linex
Ozlins
32.6
0.03
Oaline
0.248
0,000563
KaLU Oiline
149
17.3
Online
2,048
0.05, meaning
5.2 SENET RESULTS
As shown in Table 3 SENet yields models that have higher accuracy than existing alternatives by a sigmificant margin while often requiring fewer ReLUs. For example, at a small ReLU budget of < 100k, models yield up to 4.15% and 7.8% higher accuracy, on 10 and CIFAR-100, respectively. At ReLU budget of < 500k, our improve- ment is up to 0.50% and 2.38%, respectively, on the two datasets. We further evaluate the communication saving due to the non- linearity reduction by taking the per ReL.U communication cost mentioned in Table 2 In particular, the communication saving re- ported in the 8th column of Table i com- puted as the ratio of communication costs associated with an AR model that of the corresponding PR model with reduced Re- LUs. We did not report any saving for the custom models, they do not have any corresponding AR baseline model. On Tiny-ImageNet, SENet models can provide up to 0.3% higher performance while requiring 3.08x fewer ReLUs (Table 3). More importantly, even for a high resolution dataset like ImageNet, SENet models can yield close to the baseline performance, depicting the efficacy of our proposed training method.
Table 3: Results on Tiny-ImageNet and ImageNet.
Model
Baseline
Method
Test
Acc%t/
Comm.
Ace
(k)
Ace
A1k Rel.U
Savings
Tiny ImageNet
142
SENN
58.9
0.414
x15.7
ResNe118 66.1
298
84.96
0.218
7.5x
303
Mxcppellinee/laa Jaa of al 2011
61.85
0.137
6.7x
917
64.66
0.071
X2.4
imageNee
ResNor18
71.94
600
SENar
70.28
0.117
3.86x
950
71.16
0.075
2.43x
Table 4: Results with ReLU reduction at the granu- larity of activation channel evaluated on CIFAR-100
Model
Baseline
Method
Test
Accri
Coom..
Acc%
(k)
Acc%:
11k ReL. Savings
180
SENet
0.44
7.7x
WRN22
SUNet
0.33
200
JNNL
Cho
2023
77.46
0.38
6.9x
"On ImageNet, for comprehensive training with limited resources, sample 100 classes from the Ima- geNet dataset with 500 and 50 training and test examples per class, respectively.

---

Published as a conference paper at ICLR 2023
Table 5: Performance of SENet and other methods various datasets and models.
min S rS max
Model
Baseline Ace%
*ReLU (k)
Method
lest Aceti
Accks 11k Rel.U
Comm.
Sarings
Datnsct: CIFAR-ID
VGG16
93.8
12.6
91.6
7.33
23.6x
SLNNtiours)
6,0x
ReaNGTTK
95.2
93.60
6.8%
ResNet18
95.2
12.9
SNL Cho al 2022
85,23
6.84
433
WoG16
95.8
36.8
Drephee Rost Dltl 1 mir 20.1
88.5
2.41
8X
VGC16
93.8
126
(ENat
93.42
0.74
2.3x
KmNNeTk
100k S 500k
DecpRaDiace et
92.5
VOGI6
93.8
126
SATENes (Lu et 2021
88.0
0.7
3.3x
Custom Net
96.0
100
CyptiNAAS Calnas of 2020
92.18
0:32
500
94.41
0.19
Ditiret: CIFAR-100
ResNet18
78.05
24.6
70.39
2.87
21.8x
76.28
11.2x
SENeo(ours)
6.8%
S S 100k
ResNet34
78.42
50.1
74.84
1.5
x
80
76.66
0.96
12. 1
ResNei18
78.05
28.7
DeeppeDioe
Uha
2021
68.6
2.39
19.
49.2
69.5
1.41
11:3x
74.03
Sphyax Cho
2021
69.57
ResNeilk
78
150
76.32
0.52
3.7x
KEINe154
78.425
200
78.5
0.41
4.3X
WRN22
50.82
130
SENet(ours)
79.12
0.44
1.7X
2:00
79.81
0.33
5.8x
300
80.64
4.6x
WRN22.8
7.7x
100k + 600k
ResNe1K
78.05
DeeweeDoe
76.22
Custom Ket
14.93
112
Spiyax Cho of 2011
72.3
0.714
230
74.93
0.32
Custom Nei
79.07
T00
CoytoNAS Ghodei et al. 200
08.87
0.69
Results with activation channel level ReLU reduction. As shown in Table 4 while trimming ReLUs at a higher granularity of activation channel level, SENet models suffer a little more drop in accuracy compared to that at pixel level. For example, at ReLU budget of 240k, channel-level ReLU removal yields an accuracy of 79.3% compared to 79.81% of pixel-level, However, compared to existing alternatives, SENet can achieve improved performance of up to 1.85% for similar ReLUs.
5.3 SENET++ RESULTS
For SENet++, we performed experiments with D, [0.5, 1.0), meaning each training loop can yield models with two different channel dropout rates. The 0.5-sub-model enjoys ~4x MACs reduction compared to the full model. Moreover, as shown in Fig. 4 the 0.5-sub-model also requires signif- icantly less #eeLUs due to reduced model size. In particular, the smaller models have # #ReLUS reduced by factor of 2.05 2.08x, and 1.88x on CIFAR-IO, CIFAR- 100, and Tiny-ImageNet, respectively, compared to the PR full models, averaged over four experiments with different ReLU budgets for each dataset. Lastly, the similar performance of the SENet and SENei+ models at d, = 1.0 with similar ReLU budgets, clearly depicts the ability of +ENet+ + to yield multiple sub- models without sacrificing any accuracy for the full model.
5.4 ANALYSIS OF LINEAR AND RELU INFERENCE LATENCY
Table 2 shows the GC-based online ReLU operation latency is <433 higher than linear oper- ation (multiply and accumulate), making the ReLU operation latency the dominant latency compo- nent. Inspired by this observation, we quantify the online Pl latency as that of the N ReLUU opera- tions for a model with ReLU budget of N. In particular, based on this evaluation. Fig. 5[a) shows the superiority of SENet++ of up to ~9.6x (~1.92 x) reduced online ReLU latency on CIFAR-10 (CIFAR-100). With negligibly less accuracy this latency improvement thermore, when dr < 1.0. SENet++ requires fewer MACs and the lincar operation latency can be significantly reduced, as demonstrated in Fig. ())..
be up ~21x, Fur-
5.5 ABLATION STUDIES
Importance of ReLU sensitivity. To understand the importance of layer-wise ReLU sensi- tivity evaluations at given ReLU budget, we conducted experiments with evenly allocated
8

---

Published as a conference paper at ICLR 2023
Dataset: CIFAR-10
Dataset: CIFAR:100
Dataser: Tiny Imagelet
25
65
277
bp
333
875
281
02
59
291
:50
Siker
Sthuts 1---1
SEree
1.01
Stree
-1x-10
[3)
RaLU
[b]
RoLll
(c)
ReLu
Figure 4: Performance + on three datasets for various # U budgets. The points labeled A. B. C, and D corresponds to experiments of different target LUs for the full model (d, = 1.0). For SENet++ note that single training loop yields two points with the same label corresponding to the two different dropout rates.
bseed 1.1
Daniee 1INS 11
Detaie:: CEAR-222
DeephuDure
SINC--1tx=1.11
Zhd +2-
santhee
510
3mp
340
33
Ocass cneecy
Onlne fell. Iatery (r5)
Figure 5: Performance comparison of SENet++ (with d, 1.0 and 0.5) vs. existing alternatives (a) with VGG16 and ResNet18 in terms of ReLU latency. The labels A. B. C. D correspond to experiments of different target #ReLUU for the full model (d, = 1.0). For SENet++, note that a single training loop yields two points with the same label corresponding the two different dropout rates. (b) Comparison between Duce and SENet++ for a target # ReLU budget of 550k with ResNet18 on CIFAR-100.
ReLUs. Specifically, for ResNet18, for ReLU bud- get of 25% of that of the original model, we randomly removed 75% ReLUs from each PR layer with identity elements to create the ReLU mask, and trained the PR model with this mask. We further trained two other PR ResNet18 with similar and lower # ReLU budgets with the per-layer ReLUs assigned following the proposed sensitivity. shown in Table (6 the sensitivity-driiee PR models can yield signifi- cantly improved performance of ~5.6 for similar ReLU budget, demonstrating the importance of proposed ReLU sensitivity.
Table 6: Importance of ReLU sensitivity.
Model
Hascline
RReLU
ReLUU Sensitivity
Test
Comma. a1k Rel. Savings
Accii/
Acefs
(4)
Acc%
139.2
0.503
x1
ResNor18 78.05
135
75.88
0.56
x4.12
70.4
73.03
1.08
x7.9
Choice of the hyperparameter X and B. To determine the influence of the AR teacher's influence on the PR model's 's learning, conducted the final stage distillation with X [0.1, 0.3, 0.5, 0.7, 0.9] and B E [100, 300, 500, 700, 1000]. As shown in Fig. 6 the performance of the student PR model improves with the increasing influence of the teacher both in terms of high À and / values. However, we also observe, the performance im- provement tends to saturate at B & 1000. Note, keep À 0.5 and 3 1000, for the and X ablations, respectively.
Dataset: CIFAR-100
OFAA-100
Tost Accuracy
37.0
77.0
0.0
22
0.6
0.6
0.5
200
400
500
900
1000
Lambdo
Figure 6: Ablation studies with different À and B values for the loss term in Eq.
4
6 CONCLUSIONS
In this paper, we introduced the notion of ReLU sensitivity for non-linear layers of a DNN model. Based on this notion, we present an automated ReLU allocation and training algorithm for models with limited ReLU budgets that targets latency and communication-efficli works can achieve similar to SOTA accuracy while significantly reducing the # ReLUs by up to 9.6x on CIFAR-10, 10, enabling dramatic reduction of the latency and communication costs of PI. Extending this idea of efficient Pl to vision transformer models an interesting future research.
Pl. The resulting net-
ACKNOWLEDGMENT
This work was supported in parts by Intel, DARPA under the agreement numbers HR00 12190120, and 148650-18-1-7817.

---

Published as a conference paper at ICLR 2023
REFERENCES
Sara Babakniya, Souvik Kundu, Saurav Prakash, Yue Niu, and Salman Avestimehr. Federated sparse training: Lottery aware model compression for resource constrained edge. arXiv preprint 1.Xiv:2208.13092, 2022.
Zvika Brakerski and Vinod Vaikuntanathan. Efficient fully homomorphic encryption from (standard) lwe. SIAM Journal on computing, 43(2):831-87) 2014.
Minsu Cho, Zahra Ghodsi, Brandon Reagen, Siddharth Garg, and Chinmay Hegde. Sphynx: Relu- efficient network design for private inference. arXiv preprint arXiv:2 (0div:206...1. 106. /755, 2021.
Minsu Cho, Ameya Joshi, Brandon Reagen, Siddharth Garg, and Chinmay Hegde. Selective network linearization for efficient private inference. In International Conference on Machine Learning, pp- 3947-3961. PMLR, 2022.
Xiaohan Ding, Xiangxin Zhou, Yuchen Guo, Jungong Han, Ji Liu, al. Global sparse momentum sgd for pruning very deep neural networks. Advances in Neural Information Processing Systems, 32, 2019.
Craig Gentry. A fully homomorphic encryption scheme. Stanford university, 2009.
Zahra Ghodsi, Akshaj Kumar Veldanda, Brandon Reagen, and Siddharth Garg. Cryptonas: Private inference a relu budget. Advances in Neural Information Processing Systems, 33:16961-
16971, 2020.
Zahra Ghodsi, Nandan Kumar Jha, Brandon Reagen, and Siddharth Garg. Circa: Stochastic relus for private deep learning. Advances in Neural Information Processing Systems, 2021.
Oded Goldreich, Silvio Micali, and Avi Wigderson. How to play any mental game, or a completeness theorem for protocols with honest majority. In Providing Sound Foundations for Cryptography: On the Work of Shafi Goldwasser and Silvio Micali, pp. 307-328 2019.
Lucas Hansen. Tiny ImageNet challenge submission. CS 23IN, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Geoffrey Hinton, Li Deng, Dong Yu, George E Dahl, Abdel-rahman Mohamed, Navdeep Jaitly, Andrew Senior, Vincent Vanhoucke, Patrick Nguyen, Tara N Sainath, et al. Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups 1EEE Signal processing magazine, 29(6):22-97, 2012.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network, 2015.
Samuel Horvath, Stefanos Laskaridis, Mario Almeida, Ilias Leontiadis, Stylianos Venieris, and Nicholas Lane. Fjord: Fair and accurate federated learning under heterogeneous targets with ordered dropout. Advances in Neural Information Processing Systems, 34, 2021.
Zhicong Huang, Wen-jie Lu, Cheng Hong, and Jiansheng Ding. Cheetah: Lean and fast secure two-party deep neural network inference. IACR Cryptol. ePrint Arch., 2022:207, 2022.
Nandan Kumar Jha, Zahra Ghodsi, Siddharth Garg, and Brandon Reagen. Deepreduce: Relu reduc- tion for fast private inference. In International Conference Machine Learning, pp. 4839 4849. PMLR, 2021.
Chiraag Juvekar, Vinod Vaikuntanathan, and Anantha Chandrakasan. [GAZELLE]: low latency framework for secure neural network inference. In 27th USENIX Security Symposium (USENIX Security 18), pp. 1651-1669, 2018
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Souvik Kundu and Sairam Sundaresan. Attentionlite: Towards efficient self-attention models for vision. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 2225-2229. IEFE, 2021.
10

---

Published as a conference paper at ICLR 2023
Souvik Kundu, Mahdi Nazemi, Massoud Pedram, Keith M Chugg, and Peter A Beerel. Pre-defined sparsity for low-conplexity convolutional neural networks. IEEE Trunsactions on Computers, 69 (7):1045-1058, 2020.
Souvik Kundu, Mahdi Nazemi, Peter A Beerel, and Massoud Pedram. Dnr: A tunable robust pruning framework through dynamic network rewiring of dnns. In Proceedings of the 26th Asia and South Pacific Design Automation Conference, pp. 344-350, 2021a.
Souvik Kundu, Qirui Sun, Yao Fu, Massoud Pedram, and Peter Beerel. Analyzing the confidentiality of undistillable teachers in knowledge distillation. Advances in Neural Information Processing Systems, 34, 2021b.
Souvik Kundu, Yao Fu, Bill Ye, Peter A Beerel, and Massoud Pedram. Towards adversary aware non-iterative model pruning through d ynamic etwork ewiring of dnns. ACM Transactions Embedded Computing Systems (TECS). 2022a.
Souvik Kundu, Sairam Sundaresan, Massoud Pedram, and Peter A Beerel. A fast and efficient conditional learning for tunable trade-off between accuracy and robustness. arXiv preprint arXiv:2204.0000 00426, 2022b.
Souvik Kundu, Shikai Wang, Qirui Sun, Peter Beerel, and Massoud Pedram. Bmpq: bit-gradient sensitivity-driver mixed- preciiion quantization of dnns from scratch. In 2022 Design, Automation & Test in Europe Conference & Exhibition (DATE), pp. 588-591. IEEE, 2022c.
Aditya Kusupati, Vivek Ramanujan, Raghav Somani, Mitchell Wortsman, Prateek Jain, Sham Kakade, and Ali Farhadi. Soft threshold weight reparameterization for learnable sparsity. In International Conference Machine Learning, pp.
PMLR, 2020.
Namhoon Lee, Thalaiyasingam Ajanthan, and Philip HS Tor: Snip: Single-shot network pruning based on connection sensitivity. arXiv preprint arXiv:1810.02:3 2018.
Ryan Lehmkuhl, Pratyush Mishra, Akshayaram Srinivasan, and Raluca Ada Popa. Muse: Secure inference resilient to malicious clients. In 30th USENIX Security Symposium (USENIX Security 21). pp. 2201-2218. USENIX Association, aug 2021. ISBN 378-1-939133-22-3.
Geert Litjens, Thijs Kooi, Babak Ehteshami Bejnordi, Arnaud Arindra Adiyoso Setio, Francesco Ciompi, Mohsen Ghafoooran, Jeroen Awm Van Der Laak, Bram Van Ginneken, and Clara I Sánchez. survey on deep learning in medical image analysis. Medical image analysis, 42: 60-88, 2017.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. arXiv preprint arXiv: 180.000506 2018a.
Jian Liu, Mika Juuti, Yao Lu, and Nadarajah Asokan. Oblivious neural network predictions via minionn transformations. In Proceedings of the 2017 ACM SIGSAC conference on computer and communications security, pp- 619-631, 2017.
Ning Liu, Xiaolong Ma. Zhiyuan Xu, Yanzhi Wang. Jian Tang, and Jieping Ye. Autocompress: An automatic dnn structured pruning framework for ultra-high compression rates. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 4876-4883, 2020.
Zhuang Liu, Mingjic Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. Rethinking the value of network pruning. arXiv preprint arXiv: 1rX.v28810005200, 2018b.
Qian Lou, Yilin Shen, Hongxia Jin, and Lei Jiang. Safenet: A secure, accurate and fast neural network inference. In International Conference Learning Representations, 2021.
Pratyush Mishra, Ryan Lehmkuhl, Akshayaram Srinivasan, Wenting Zheng, and Raluca Ada Popa. Delphi: A cryptogaaphic inference service for neural networks. In 29th USENIX Security Sympo- sium (USENIX Security 20), pp.
2020.
Brandon Reagen, Woo-Seok Choi, Yeongil Ko, Vincent T Lee, Hsien-Hsin S Lee, Ye-Yeon Wei, and David Brooks. Cheetah: Optimizing and accelerating homomorphic encryption for private inference. In 2021 IEEE International Symposium on High- Perforrerfe Computer Architecture (HPCA), pp. 26-39. IEEF, 2021.
11

---

Published as a conference paper at ICLR 2023
Pedro Savarese, Hugo Silva, and Michael Maire. Winning the lottery with continuous sparsification. Advances in Neural Information Processing Systems, 33:1 13:1018-11190 2020.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1456, 2014.
Haotao Wang. Tianlong Chen, Shupeng Gui, TingKuei Hu, Ji Liu, and Zhangyang Wang. Once-for- all adversarial training: In-situ tradeoff between robustness and accuracy for free. Advances in Neural Information Processing Systems, 33:7449-7461, 2020.
Eric Wong, Leslie Rice, and Zico Kolter. Fast is better than free: Revisiting adversarial training.
arXiv preprint FrXiv:2001003990 2020.
Andrew Chi-Chih Yao. How to generate and exchange secrets. In 27th Annual Symposium on Foundations of Computer Science (sfes 1986), pp. 162-167. IEEE, 1986.
Hongxu Yin, Pavlo Molchanov, Jose M Alvarez, Zhizhong Li, Arun Mallya, Derek Hoiem, Ni- raj K Jha, and Jan Kautz. Dreaming to distill: Data-free knowledge transfer via deepinversion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 8715-8724, 2020.
Sergey Zagoruyko and Nikos Komodakis. Paying More Attention Attention: Improving the Performance Convolutional Neural Networks via Attention Transfer. arXiv preprint arXiv: 2016a.
Sergey Zagoruyko and Nikos Komodakis. arXiv:1605.07146. 2016b.
Wide residual networks. arXiv preprint
A APPENDIX
A.1 TRAINING HYPERPARAMETERS AND MODELS
The training hyperparameter details for each stage is provided in table Also, for the KL- divergence loss in stage 2 and 3, we fix the temperature value = 4.0. To evaluate Tlp we set of 1000 randomly selected training image samples.
details the loss function used for different OD rate values for the SENet+ (at the fine-tuning stage of the PR model i.e. training stage 3). Note, the training stage I where the AR model gets trained, we standard CE loss for all the sub-models with different OD rate values. The detailed fine-tune training algorithm is provided in 2 Note, for SENet, the algorithm remains the same with D, [1.0], meaning supporting only the full model with all the channels presen
compared to that of ImageNet, we have used the variant of ResNet18 and ResNet34 models that are suitable for supporting lower resolution datasets. In particular, we replaced the 7 x kernel, stride and padding 3 of the first layer with a x 3 kernel having stride and padding of 1 each. We would also like highlight that this is popular practice and can be seen in various other peer-reviewed manuscripts (Wang et al., 2020 Liu et al. 20200) Wong et al. (020)) More importantly, both the existing state-of-the-arr methods (Cho et al. 2022 Tha et al. 2021) used similar ResNet models as
Model Selection. For lower resolution images (CIFAR- 10, CIFAR- 100,
ours, for their evaluations a reduced ReL. budget.
Table 7: Hyperparameter settings of SENe//SENet+ training method.
Model(s)
Dataset
Epoch
batch
Initial LRR
Momen-
Optim-
Weight
stagel
stages
stagel
stage2
stage3
izer
decay
ResNetT8,
CIFAR-10
240
150
180
128
0.05
0.05
0.01
0.
SGD
0.0005
VGG16
ReaNet 18, 34)
CIFAR-100
240
150
180
128
0.05
0.05
0.01
0.9
SGD
0.0005
WRN22-8
Re 8
Tiny- ImageNet
120
100
120
32
0.05
0.05
0.01
0.9
SGD
0.0005
ResNet18
ImageNet
0.05
0.01
SGD
0.0005
We have open-sourced the validation codes with the supplementary.
12

---

Published as a conference paper at ICLR 2023
Table 8: Loss function used for sub-models with different OD rates in SENet++ training stage 3. OD (d.)
(L-A)C +ALK + LPRAM
1.0
(I-ALLOB + ALKL
Algorithm 2: SENeL+ Fine-Tune Training Stage
Data: Trained AR model parameters AAR. PR model parameters Opn, mini-batch size B, learned ReLU mask from stage 2 II. OD set D,. Output: Trained PR model with RcLU count + Pp
(OAR, II)
for i < 0 to cp do
for j + 0 ng do
for dropout rate d, in sorted D, do sampleSHN(d, //sample the BN corresponding to dr LCE computeCELoss (.x.88. Yo.s, You II)
LKL + comput if d, = 1.0 then
LPRAM + computerRaMLoss (a
11
+ (1 - X)CCE + ALKL + CPRAM
else
-)- + ALKL
end
15
accumulatedion accumulateGirad((] end
17
updateraram (pp Vc)
18
end
19
end
A.2 MODEL LATENCY ESTIMATION
Similar to the existing literature Jha et al. 2021 Cho et al. 2022 Lou et al. 2021), we assume the popular PI framework described in Delphi (Mishra et al. (022), and leverage their reported per ReLU operation latency estimate the total online cryptographic private inference latency. In particular, assuming sequential execution, the total latency can be estimated as
T = (NMAC tmac + NRLL tretu)
(5)
where AC and NRLLU corresponds the total number of MAC and ReLU operations required to perform single forward pass. lvelu represents per ReLU cipher-text execution time and its value is assumed 85.3yS. Note, Per ReLU cipher text execution wall-clock time is taken from (Mishra al. 2020) and the execution of 1000 ReLUs takes proportional time as reported in (Cho et al. Thus similar to (Cho et al. 2022 Tha et al. (0211, we extrapolated per- RerL wall clock time extract total ReLU latency for cryptographic inference. Similarly, we evaluated the per linear operation latency (man to be 0.248pS (Mishra et al. 2020). Now, as cryptographic ReLU latency is around 343 x costlier than that for linear ops, similar earlier literature (Jha et al.) [202]] Cho et al. 2022 Lou al. 22 for comparable NMAC and NRELU we can approximate the total latency with that of the non-linear latency. Note, the values of the Rel.U and linear operation latency are taken from (Mishra et 2020), however, we understand with recent improvement of operation latency the "per-operation delay' can reduce (Huang et al. 2022).
2022
A.3 DISCUSSION ON THE RELATION BETWEEN RELU IMPORTANCE AND PRUNING SENSITIVITY
Fig. of the main manuscript demonstrated the inverse trend between ReLU importance and param- eter pruning sensitivity. Here, exemplary, used extremely low target parameter density = 0.1 to compute the parameter sensitivity. This choice forces an aggressive drop in pruning sensitivity of later layers because they correspond to the majority of the model parameters. In con- trast, when we set less aggressive compression with higher parameter density d of 0.3 0.5, we observe a more gradual reduction in weight pruning sensitivity at later layers similar to change ob- served with ReLU importance. More importantly, shown in Fig. 7 the inverse trend between
13

---

Published as a conference paper at ICLR 2023
ReLU importance and pruning sensitivity still be clearly observed irrespective of the choice of while abruptness sensitivity change remains function of target d.
ResNet18 on CIFAR-100
1.0
0.0
aggressiveness
9.6
Reduces
0.4
Pruning senetivity (d 0.11
Pruning seraiiiity (d 0.3)
0-
sensitivity (d-
0.0
10
13
16
Layer
Figure 7: Ablation with different d yielding different layer-wise parameter pruning sensitivity. can be clearly observed that as the d increases, the aggressiveness of sensitivity change reduces and becomes similar to what observe for the ReLU importance plot.
A.4 MORE RESULTS AND ANALYSIS
1.0
RelU Servibuit
0.8
100k ReLU
0.6
0.2
(A)
60000
ReLU 49.5k
count
550000
RALU, 100k
RelU,557K
30000
20000
10000
2
6
10
11
13
14
35
36
37
Layer
(h)
Figure 8: (a) ReLU sensitivity per non-layer layer meeting difffernnt target ReLU budget, (b) ReLU count for different layers, evaluated following the sensitivity. We used ResNet18 on CIFAR- for this analysis.
Layer-wise ReLU Sensitivity. Fig. g(a) shows the ReLU sensitivity per layer. As elaborated in the Section 3, the ReLU sensitivity more the later layers, making our ReLU sensitivity follow similar trend as ReLU importance in DecpReDuce (Jha et al., (021). Fig. 8 shows the layer wise ReLU count for different ReLU budget, allocation driven by sensitivity (Fig. B(a). With the increasing count of ReLU budget, the assignment ReLU happens more aggressively at the later layers, compared to the earlier ones.
SENet++: Training with More Than Dropout Rates. In the original manuscript we showed results with two dropout rates (D, = [0.5, 1.0),, while training with different dropouts, and yield models with different channel width factors. We show results with D, [0.25, 0.5. 0.75, 1.0). In particular, Fig. D and 10 show the results with models yielded via training on four different d, choices. As shown in the Fig. π( and [0)a). the performance of the models at d, 1.0 and d, 0.5, similar for both two and four d, choices, making SENet++ efficient algorithm in yielding multiple reduced FLOPs/ReLU models. Moreover, the effective ReLU reduction remains proportional to the corresponding ordered dropout rate (OD rate) (Fig. (b) and b)). Fig()c) shows the effective FLOPs reduction for the CONV layers while performing inference at reduced OD rate model selection.
A.5 FURTHER ABLATION STUDIES
Ablation Studies for Stage 2 Importance. To understand the importance of the ReLU mask iden- tification stage, we now present results for SENet with and without that stage. In particular, for the model without stage 2, we randomly assign the ReLU mask by following the ReLU layer sensitivity, meaning layers having higher sensitivity will have non-zero mask values of similar proportions
14

---

Published as a conference paper at ICLR 2023
[0.25, 0.5 0.75 1.0)
-0
[0.5. 1.01
COWV redunocd
Aecuray
10.25,055.
30
00 rete (d))
00 rate (d6)
Figure 9: (a) Accuracy OD rate, (b) ReLU count OD rate for SENet+ training with different OD rate supports (two and four), (c) CONV layer FLOPs reduction factor for models at different OD rate values. We used ResNet18 on CIFAR- 100 for this evaluation.
94
120
100
-0-
[0.5.100]
Accu acy
90
02
70
60
- [0.25, 0.5, 0.75, 1.0]
[0.5,
90
30
0.6
0.8
0D rate (O6)
00 rate (0.)
Figure 10: (a) Accuracy vs. OD rate, (b) ReLU count vs. OD rate for SENet++ training with
different OD rate supports (two and four). We used VGG16 on CIFAR-10 for this evaluation.
random locations of the corresponding ReLU mask tensors. Table 9 clearly demonstraates the ben- efits of stage 2 as the model can provide improved performance of 1 ..2%% compared to the one trained without stage 2.
Table Importance ReLU mask identification stage (stage 2).
We used CIFAR- 100 dataset.
Model
Baseline
#RcLU
ReLU mask
Test
Acc%
Acc%
(k)
identification (stage 2)
Acc%
#1k ReLU
24.6
X
59.12 12
2.37
ResNet18
78.05
24.6
70.59
2.87
Ablation Studies for Stage 3 Importance. Table 10 shows the importance of fine-tuning stage (stage 3). In particular, the accuracy difference of model before and after stage 3 training can vary up to 6.3%.
Table 10: Importance of Activation similarity maximization (stage 3).
We used CIFAR- 100 dataset.
Model
Baseline
#ReLU
Activation similarity
Test
Acc%
Ace%
(k)
maximization (stage 3)
Acc%
#1k ReLU
24.6
X
64.10
2.57
ResNet18
78.05
24.6
70.59
2.87
Ablation Studies with the PRAM Loss Component. Table TT shows the importance of PRAM loss component during the fine-tuning stage (stage 3). In particular, the accuracy can improve to 0.69% on CIFAR-10 evaluated with ResNet18 for 150k ReL.U budget.
Table 11: Importance of PRAM loss at final fine- fine-tuning stage. We used CIFAR-I0 R-10 dataset.
Model
Baseline
#ReLU
With PRAM
Test
Acc%/
Acc%
(k)
loss
Acc%
#1k ReLU
150
X
94.12
0.62
ResNet18
150
94.91
0.63
15


Reviews:
[{"review_id": "QL36UpxHbiL", "reviewer": "Reviewer_uQYb", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies how to reduce ReLU operation more efficiently to reduce the communication and latency overhead of privacy inference. They demonstrate the relation between a layer’s sensitivity towards pruning and its associated ReLU sensitivity and introduce an automated layer-wise ReLU sensitivity evaluation strategy.  They propose SENet, a three stage automated ReLU trimming strategy that can  yield models for a given reduced ReLU budget. ", "summary_of_the_review": "See above ", "strength_and_weaknesses": "Strength: \n1. This paper studies a very important question. \n2. They study the relation between ReLU importance and pruning sensitivity. \n3. They propose a three-stage training process, SENet, to yield secure and efficient networks for PI  with the ReLU budget guarantee. \n4. They propose SENet++ to further reduce both linear and ReLU layer compute cost, \n5. They conduct extensive experiments on various models and show the improvements. \nWeakness: \nI am not an expert in this domain. The methodology and experiments sound reasonable to me. A minor suggestion is that since many models leverage the Vision Transformer as the backbone, I believe adding the experiments on ViT will be much more interesting. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "AOuJYWdgVp", "reviewer": "Reviewer_e8en", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a new SOTA method SENet++ for the reduction of ReLUs for Private Inference (PI). The method is based on a novel measure of ReLU sensitivity (of a layer), a learnable mask applied at each layer, a distillation technique, and ordered channel dropout, to remove the least important ReLU operations while maintaining high accuracy. Extensive experiments with comparisons to existing methods verify the superiority of the proposed method in terms of communication-saving and accuracy maintenance.", "summary_of_the_review": "This paper presents a promising solution for ReLU reduction. The paper is well-written and the experiments are fairly thorough, though missing an important ablation study. The improvement of the proposed SENet++ over existing methods seems quite significant.\n\n--------\nScore has been raised to 8 after the rebual.", "strength_and_weaknesses": "Strengths:\n1. A well-written paper with clear reasoning and structured presentation.\n2. Proposes a novel linearization method for secure PI\n3. The method seems to be extremely effective, compared to existing methods.\n4. Extensive empirical evaluations and understandings.\n\nWeaknesses:\n1. Some of the techniques in SENet++ are adapted or directly transferred from existing works. For example, sensitivity evaluation\nbefore training from Lee et al. (2018), ordered dropout from Horvath et al. (2021). This reduces part of the technical contribution of this paper.\n2.  Missing an ablation study of the key components of SENet++. How much each component contributes to the final performance and f they can be integrated into existing methods to improve their performance.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "7oanmgpBpFx", "reviewer": "Reviewer_oSdq", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a new approach to reduce the number of ReLU activation functions in a given model. In private inference applications, latency plays a crucial role and is highly dependent on the number of ReLUs. The proposed approach is based on the insight that parameter pruning sensitivity and ReLU importance are negatively correlated. It then uses a three-step algorithm to select a ReLU mask and distill the knowledge from a teacher model. The paper further proposes an additional improvement to not only reduce the number of activation functions but also the overall model size by leveraging ordered dropout.\n", "summary_of_the_review": "I like the motivation taken by parameter pruning and moved to \"ReLU pruning\". This offers a novel perspective on the problem of nonlinearities in the context of efficient private inference. However, the paper lacks clarity and quality of writing. Also, the comparison to previous approaches seems inconsistent and makes it hard to evaluate the overall effectiveness of the proposed solution.", "strength_and_weaknesses": "Strength:\n+ The inspiration based on weight pruning provides an interesting view of the topic.\n+ Some experimental results support the improvements made by the proposed solution\n+ The introduction and background and related work sections are well-written and motivated\n+ Experiments were performed on various datasets and model architectures\n\nWeaknesses:\n- The presentation and writing of the paper should be improved (see more details in the quality section)\n- Some parts of the paper are hard to follow, especially the second part of section 3 and parts of section 4.\n- The experimental evaluation and comparison to previous approaches are not consistent. For example, the number of ReLUs allowed varies in Table 5 between SENet and other approaches. This makes it hard to assess the effectiveness of the approach.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster