DeepReShape: Redesigning  Neural Networks for Efficient Private Inference
Authors: Nandan Kumar Jha
Brandon Reagen
Venue: Accepted by TMLR ()

Abstract: Prior work on Private Inference (PI)---inferences performed directly on encrypted input---has focused on minimizing a network's ReLUs, which have been assumed to dominate PI latency rather than FLOPs. Recent work has shown that FLOPs for PI can no longer be ignored and incur high latency penalties. In this paper, we develop DeepReShape, a technique that optimizes neural network architectures under PI's constraints, optimizing for both ReLUs {\em and} FLOPs for the first time. The key insight is strategically allocating channels to position the network's ReLUs in order of their criticality to network accuracy, simultaneously optimizes ReLU and FLOPs efficiency. DeepReShape automates network development with an efficient process, and we call generated networks HybReNets. We evaluate DeepReShape using standard PI benchmarks and demonstrate a  2.1% accuracy gain with a 5.2$\times$ runtime improvement at iso-ReLU on CIFAR-100 and an 8.7$\times$ runtime improvement at iso-accuracy on TinyImageNet. Furthermore, we investigate the significance of network selection in prior ReLU optimizations and shed light on the key network attributes for superior PI performance.

Published in Transactions on Machine Learning Research (06/2024)
DeepReShape: Redesigning Neural Networks for Efficient Private Inference
Nandan Kumar Jha New York University
mj20499mmm.edl
Brandon Reagen
bjr bjr5@ny..ccm edu
New York University
Reviewed on OpenReview: https: // openreview. net/ forun? id= SuCBWULIta
Abstract
Prior work on Private Inference (PI) inferences performed directly on encrypted input has focused on minimizing a network's ReLUs, which have been assumed to dominate PI latency rather than FLOPs. Recent work has shown that FLOPs for PI and incur high latency penalties. In this paper, we develop DeepReShape, technique that optimizes neural network architectures under PI's constraints, optimizing for both ReLUs and FLOPs for the first time. The key insight is strategically allocating channels to position the network's 's ReLUs in order of their criticality to network accuracy, simultaneously optimizes ReLU and FLOPs efficiency. DecpReShape automates network development with an efficient process, and call generated networks HybReNets. We evaluate DeepReShape using standard PI benchmarks and demonstrate 2.1% accuracy gain with 5.2x runtime improvement at iso-ReLU on CIFAR-100 and an 8.7x runtime improvement at iso-accuracy on TinyImageNett Furthermore, we investigate the significance of network selection in prior ReLU optimizations and shed light on the key network attributes for superior PI performance.
longer be ignored
1 Introduction
Motivation The increasing trend cloud-based machine learning inferences has raised significant privacy concerns, leading to the development of private inference (PI). allows clients to send encrypted inputs cloud service provider, which performs computations without decrypting the data, thereby enabling inferences without revealing the data. Despite its benefits, P1 introduces substantial computational and storage overheads (Mishra et al. 2020 Rathee al. 2020) due to the use of complex cryptographic primitives Mohassel & Rindal 2018
DDmmleer et al. 2015
Patra et al. 2021
Current PI frameworks attempt to mitigate these overheads by adopting hybrid cryptographic protocols and using additive secret sharing for linear layers (Mishra et al. 2020). This approach offloads homomorphic encryption tasks to an input-independent offline phase, achieving near plaintext speed for linear layers during the online phase. However, it fails to address the overhcads of nonlincar functions (c.g., RcLU), which remain orders of magnitude slower than linear operations (Ghodsi al. 2020
In Pl, Garbled Circuits (GCs) key cryptographic primitive that allows two parties to jointly compute arbitrary Boolean functions without revealing their data Yao 1986)-and are used for private computation of nonlinear functions. In GC, noulinear functions are first decomposed into a binary circuit (AND and XOR gates), which are then encrypted into truth tables (i.e., Garbled tables) for bitwise processing of inputs (Ball ct al. 2016 2019). The key challenge in GC the storage burden: a single ReLU operation GC requires 18 KiB of storage (Mishra et al. 2020), and networks with millions of ReLUs (e.g., ResNet50) can demand Additionally, computing all
-re
approximately ~00 GiB of storage for single inference (Rathee et al. 2020

---

Published in Transactions on Machine Learning Research (06/2024)
80
SENGITCLR 23)
2.3x FLOPs reduction
80
SNLACMLL'2 (2)
CIEAR- 100 Accuracy
76
CIFAR-100 (00 Accuracy
[lbReeNet(Our))
78
72
76-
68
X
64
SENGUCCLR'23)
74.
60
DeppReDueedCMMI
SNI (ICM (22)
72-
2.1% Acc. gain
56
HybReNet(Ouns)
70-
12.5x FLOPs reduction
8
16
32
64
128
256
64
128
256
512
1024
2048
#ReLUs (K)
FLOPs (M)
(a) ReL.Us-Accurae Pareto
(b) FLOPs saving
Figure 1: HebReNet outperforms state-of-the-art (SOTA)
Rel.U-optimization methods SENets(Kundu et al.
20230, (NL(Chh et al. 2022b), and DeepRe et al.) 2021), achieving higher accuracy (CIFAR-100)
and significant reduction in FLOPs while using fewer RcLUs (Table i illustrates the Pareto points specifics).
ReLUs in GC for a network like ResNet18 takes 21 minutes on TinyImageNet dataset (Garimella et al., 2023 Therefore, ReLUs are considered a primary source of storage and latency overheads in PI.
Prior work on PI-specific network optimization primarily focused on reducing nonlinear computation overheads, assuming linear operations (FLOPs) are effectively free. For instance, CryptoNAS (Ghodsi et al. 2020) and Sphynx (Cho et al.
2022a) employed neural architecture search for designing ReL baseline networks without considering FLOPs implications. Similarly, PI-specific ReLU pruning methods (Cho et al. 2022b; Jha et al. 2021] made overly optimistic assumptions that all FLOPs can be processed offline without affecting real-time performance. The cxisting state-of-tho-ars (SOTA) in PI (Kundu et al. 2023) claimed that FLOPs cost is 343x less significant than ReLU cost. However, recent. research (Garimella el al. 2023) has challenged these assumptions, demonstrating that FLOPs introduce significant latency penalties in end-to-end system-level PI performancd
Consequently, there is an emerging need to develop network design principles and optimization ion techniques that address both ReLU and FLOPs constraints in PI. This raises two critical questions: Can we leverage existing FLOPs reduction techniques and integrate them with Pl-specific ReLU pruning methods? Second, how effective is it to employ Pl-specific ReLU pruning techniques on FLOPs efficient networks, such as MobileNets (Howard et al. 2017 Sandler et al. 2018
Challenges Balancing ReLU efficiency with FLOPs efficiency is crucial for Pl-specific network design optimization methods. + (Kundu et al. 2023) integrate FLOPs reduction technique with their ReLU pruning method and achieves (up to) 4x FLOPs reduction; however, at the expense of ReLU efficiency. The impact of existing FLOPs reduction methods on ReLU efficiency has not been extensively explored, and Jha b showed that FLOPs pruning hots tend to result in lower ReLU efficiency.
et al.
Furthermore, employing ReLU pruning on FLOPs-optimized networks results in inferior ReLU efficiency. For example, when ReLU pruning (Jha et al. 2021) employed on MobileNets, their ReLU efficiency remains consistently lower compared to standard networks (e.g., ResNet118 used in PI (see Figure 5(a)). Similarly,
SOTA FLOPs efficient networks such as RegNet (Radosavovic et al. 2020) and ConvNeXt- V2 WWoo et al.
2023 exhibit suboptimal ReLU efficiency compared to the Pl-tailored networks (sce Figure 11
This conflict between ReLU and FLOPs efficiency arises from the distinct layer-specific distribution of ReLUs and FLOPs in the network and their impact on network accuracy. In conventional CNNs, ReLUs are concentrated in the early layers, while ReLUs critical for the network's accuracy reside in deeper layers (Jhal
et al. 2021). ReLU pruning often removes many ReLUs from these early layers (Cho et al. 2022b Jha et al. 2021
while FLOPs pruning targets the deeper layers due to their higher channel counts (He et al. 2020
'In real-world scenarios, there is invariably some degree of inference arrival, and even at very low arrival rates, processing FLOPs offline becomes impractical due to limited resources and insufficieut time. Consequently, FLOPs start affecting real-time performance, becoming more pronounced for networks with higher FLOPss The FLOPSs penalties can only be disregarded when there is zero inference arrival rate or when homomorphic accelerator offering more than 1000 speedup is employed.
2

---

Published in Transactions on Machine Learning Research (06/2024)
Morcover, designing ReLU efficient networks requires different network hyper-parametert than those needed for FLOPs efficient networks (See Table 15
Another significant challenge in designing Pl-tailored networks is identifying critical network attributes for PI efficiency. The effectiveness of PI-specific ReLU optimization techniques largely depends on the choice of input networks, leading significant performance disparities not solely ascribed to FLOPPs or accuracy discrepancies (refer to $3.2). Prior work on ReLU optimization offers limited insight into their network selection - SENets (Kundu et al., 2023 and SNL (Cho et al. 2022b) used WideResNet-2225 for higher ReLU counts and ResNet18 for low ReLU counts. This leaves gap in understanding whether networks with specific characteristics can maintain superior performance across various ReLU counts or if targeted ReLU counts dictate the desired network attributes.
The limitations of the existing ReLU-optimization techniques also impede the advancement of PI. Coarse- grained ReLU optimizations (Jha al., 2021) encounter scalability issues, as their computational complexity varies linearly with the number of stages in network. While fine-grained ReLU optimization (Cho al., 2022b Kundu et al. 2023) shows potential, its effectiveness is confined to specific ReLU distributions and tends to underperform in networks with higher ReLU counts or altered ReLU distribution (refer to $3.3)
Our techniques and insights To simultaneously optimize both the ReLU and FLOPs efficiency, we begin by eritically evaluating existing design principles and posing a fundamental question: What essential insights need to be integrated into the design framework for achieving FLOPs efficiency without compromising the ReLU efficiency? Our sensitivity analysis of different network stages on ReLU and FLOPs efficiency reveals two key observations:
Increasing the network's width while positioning the network's ReLU based on their criticality network's accuracy allows FLOPs reduction without sacrificing ReLU efficiency (Figure
2. Widening channels in each network stages has distinct effect on network's overall ReLU and FLOPs efficiency (Figure 3ie, f)).
These insights led us to propose ReLU-equalization, novel design principle that redistributes ReLUs in a conventional network by their order of criticality for network's accuracy (Figure B), inherently accounting for the distinct effect of network stages on ReLU and FLOPs efficiency.
Our investigation into key network attributes for PI efficiency indicates that specific characteristics are essential for superior performance at different ReLU counts. We discovered that wider networks improve PT performance at higher ReLU counts. Whereas, at lower ReLU counts, the proportion of least-critical ReLUs in the network is crucial, especially when ReLU pruning is employed. Leveraging this insight, we achieve a significant, up to 45x, FLOPs reduction at lower ReLU counts.
Building on the these insights, we develop ShapR,,hape, a framework to redesign the classical networks, with an efficient. process of computational complexity ((),, and synthesize Pl-efficient nelworks HybReNet. Our approach results in a substantial FLOPs reduction with fewer ReLUs, outperforming the SOTA in P1 (Kundul ct al. 2023). Preciscly, we achieve 2.3x ReLU and 3.4x FLOPs reduction at iso-accuracy, and 2.1% accuracy gain with a 12.5x FLOPs reduction at iso-ReLU on CIFAR-100 (see Figure 1.. On TinylmageNet, we achieve 12.4x FLOPs reduction at iso-accuracy compared to SOTA (see Table 7
Contributions Our key contributions are summarized as follows.
Extensive characterivation to identify the key network attributes for PT efficiency and demonstrate their applicability across a wide range of ReLU counts.
A novel design principle ReLU-equalzzation, and design of the HybReNel family of networks tailored to P ReLU dropping technique to systematically
constraints. Moreover, reduce the ReLU count by 16x, allowing efficient ReLU optimization even at very low ReLU counts.
devise ReLU-re a channel-wise
3. Rigorous evaluation of our proposed techniques against SOTA PI methods (Kundu et al. 2023 Cho et al.
2022bb and SOTA FLOPs efficient models (Woo et al. 2023 Radosavovic et al., 2020).
Scope of the paper This paper addressos the challenges of strategically dropping ReLUs from the convolutional neural networks (CNNs) without resorting to any approximated computations for nonlinearity. We exclude the models with complex nonlinearities, such as transformer-based models and FLOPS efficient.
3

---

Published in Transactions on Machine Learning Research (06/2024)
models like EfficientNet and MobileNetV3 + often relying approximated nonlinear computations in PI. Also, we exclude the CrypTen-based PT in CNNs (Tan et al. 2021 Peng et al. 2023), as it operates under different security assumptions
Organization of the paper Section 2 provides the relevant background on PI protocols, threat models, and network architecture, along with an overview of channel scaling methods and a categorization of PT-specific ReLU pruning methods. Section 3 comprehensively evaluates baseline network design and ReLU optimization strategies within the context of PI, outlining their limitations and our key observations. Section 4 introduces the DeepReShape method, followed by Section [ which presents our experimental findings, and Section 6 summarizing the related work. Finally, we discuss the broader impact, limitations and future work Section 7
2 Preliminary
Private inference protocols and threat model We use Delphi (Mishra et al. 2020) two-party protocols, as used in Jha et al. (2021) Cho et al. (2022b), for private inference. In particular, for linear layers, Delphi
performs heeayy heavy homomorphic operations (Gentry et al., 2009; Fan & Vercauteren 2012 Brakerski 2017) in the offline phase (preprocessing) and additive secret sharing (Shamir 1979
let al., 2014 Cheon et al.
in the online phase, once the client's input is available. Whereas, for nonlinear (ReLU) layers, it uses garbled
circuits Yao 1986 Ball et al. 2019). Further, similar to Liu et al. [0177] Juvekar et al. (2018) Mishra et al.
(2020): Rathee et al. 2020 we assume an honest-but-curious adversary where parties follow the protocols
and learn nothing beyond their output shares.
Note that the different sets of protocols for PI significantly affect the cost dynamics (communiaation, storage, and latency) for linear and nonlinear layers, thereby influencing network optimization goals. For instance, CoPriv ŞZeng et al., 2023bb uses oblivious transfer (OT) for nonlinear operations and primarily optimizes convolution operations (i.e., FLOPs). Unlike OT, GCs offer constant round complexity and typically distribute more computational load to the server for garbling the circuit, reducing the client's computational burden DDemller et al. 2015 Patra et al. 2021). h In this work, we compare against prior approaches that use GCs
for ReLUs, similar to the cryplographic selup of Delphi MMishra et al,, 2020
Architectural building blocks Figure 2 illustrates a schematic view of a standard four-stage network with design hyperparameters. Similar to ResNet (He et al. 2016), it has stem cell (to increase the channol count from 3 to m), followed by the network's main body (composed of linear and nonlinear layers, performing most of the computation); followed by a head (a fully connected layer) yielding the scores for the output classes. The network's main body is composed of a sequence of four stages, and the spatial dimensions of feature maps (d& x dx) are progressively reduced by x in each stage (except Stagel), and feature dimensions remain constant within a stage. We keep the structure of the stem cell and head fixed and change the structure of the network's body using design hyperparameters.
Notations and definitions Each stage composed of identical blocke repeated Φ1, Φ2 Φ3, and Φ4 times in Stagel, Stage2, Stage3, and Stage4 (respectively), and known as stage compute ratios. The output channels in stem cell (m) are known as base channels, and the number of channels progressively increases by factor of a, B, and y in Stage2, Stage3, and Stage4 (respectively), and we termed il slagewise channel reulliplicution factors. The spatial size of the kernel is denoted as f x f (e.g., 3 x 3). These width and depth hyperparameters primarily determine the distribution of ReLUs and FLOPs in the network.
?private inference on transformer-Bee based models entail fundamentally different challenges (Chen et al. 2022b Hao et al Akimoto et al.) 2023 Zheng et al. 2023 Hou et al. 2028 Gupta et al., 2023). CNNs predominantly employ crypto-frinnll nonlincarities, e.g., ReLUs (and MaxPPol. at all used); while, transformers utilize complex nonlinearities like Softmax, GcLU, and Norm. ReLUs in PI are preciscly computed using Carbled-circuit (Mishra et al. 2020), w whereas transformers often resort to approximations for their nonlinear computations due to performance objectives and numerical stability (Wagg et al.
2022
2022
Li et sl. 2023 Zeng et al. 2023a Zhang et al. 2023 Likewise, models such as Efficient.ets (Tan & Le 2019 2021 and
MobileNet
[Hwaard et. al.] 2019) incorporate Swish and Sigmoid nonlinearities ies to augment network expressiveness. These
nonlincarities approximated as discrect piecewise polyuomials (Tan al. 2022
&CrypTen resembles a thrce-party framework since it adopts a Trusted Third Party (TTP) to produce beaver triples during
the offline phase Knott al. (2021). Consequently, the actual FLOPs overheads do not appear in end-to-end PI latency.
4 'Except the first block (in all but Stagel) which performs downsampling of feature maps by 2x.
4

---

Published in Transactions on Machine Learning Research (06/2024)
3
Stage1
Stage2
Stage3
Stage4 XX (adfm)
Stagel
Stage2
Stuge3
Stage4
feature maps
0XX (m)
x (om)
#PParems * BReLU
mlgg)
aml
aBmel 16f
do
Hebha Behammo morui
#FLOPs & BeeA
mfa
am
body
de
agy X
de
Table Network's complexity (FLOPs and Params) per unit of nonlincarity varies with network's width, and independent of the net- work's depth. Consequently, Wider network need fewer ReLUs for a given complexity, com- pared to their deeper counterparts.
head
xd. X
11
Figure 2: Depiction of architectural hyperparameters and feature dimensions in a four stage network. For ResNet18 m = 64, - = and a=
Channel scaling methods Broadly, channel scaling methods can be categorized into three categories (see Table 2 First, Uniform channel scaling, where a, 3, and are set to 2 and channels are scaled either by scaling base channel counts (e.g., m=64 to m=128 in ResNets) or by constant multiplication factor in all the network stages (c.g., k=10 in WidcResNNt22x10). We refer their network variants as BascCh, often usod for FLOPs efficiency. Second, homogeneous channel scaling, where o, ,3, and y set identical, and channels in successive stages of the network scaled by homogencously augmenting these factors. For instance, a; B, and y are set to 4 in CryptoNAS (Ghodsi et al. (020)) and Sphyux (Cho et al. 2022a)) for designing ReLU efficient baseline networks. We termed their network variants as StageCh. Third, heterogeneous channel scaling, where a, 3, and 7 are non-identical, and provides greater flexibility for balancing FLOOPs and ReLU efficiency by scaling the channels successive stages of the network differently.
Criticality of ReLUs in network We employ the criticality metric C from Jhaa al. (202)) to quantify the significance of ReLUs' within a network stage for overall accuracy. Higher C2 values indicate more critical
ReLUs, while the least significant ReLUs are assigned value of zero (scc Table I0 and 11.. Empirically, for four-stage network like 18 and ils variants (BaseCh and StageCh), the ReLUs in Stagel contribute the least and are the least critical, while those in Stage3 are the most critical for the network's accuracy.
Table 2: Comparison of channel scaling methods: Uniform channel scaling special case of homogeneous channel scaling where all stagewise channel multiplication factors (a=B=y) is identical and set to 2, and channels in network's 's stages are scaled by constant factor (c.g., 1=10 in in WideRessNe12x100) In contrast, heterogeneous channel scaling differs by having non-identical factors, offering greater flexibility for balancing FLOPs and ReLU efficiency and meet the PI constraints.
Channel Scaling Methods
Uniform
Homogeneous
Heterogeneous
Width pareerpprramters
a=8=y=2
a=B=y
-(a=3=>)
Network Variants Naming
BaseCh
StageCh
CeopieNeetPrroosseee
Example Networks
WidcResNet
CryptoNAS
HybReNets
3x3, mxk 3x3, mxk
3x3, m 3x3, m
3x3, m 3x3, m
Stagel
xΦ1
x
xΦ1
3x3, 2mxk
3x3, 4m 3x3, 4m
3x3, am 3x3, am
Stage2
3x3, 2mxk
x
X
x
3x3, 4mxk
3x3, 16m
3x3, 3(am) 3x3, 3(am)
Stage3
X
X
x
3x3, 4mxk
3x3, 16m
3x3, 7(3m)
Stage4
X
3x3, 7(33m)
Coarse-grained vs fine-grained ReLU optimization The coarse-grained ReLU optimization method (Jha et al. 2021) removes ReLUs at the level of an entire stage or a layer in the network. Whereas fine-grained ReLU optimizations (Cho et al.) 2022b] Kundu et al. 2023) target individual channels or activation. These approaches differ in performance, scalability, and configurability for achieving specific ReLU count. The latter allows achieving any desired independent ReLU count automatically, while the former requires manual
5

---

Published in Transactions on Machine Learning Research (06/2024)
adjustments based on the network's overall ReLU count and distribution. Nonetheless, the coarse-grained method demonstrates flexibility and adapting to various network configurations. In contrast, the fine-grained method exhibits less efficient adaptation and can lead to suboptimal performance (see 3.3)
3 Network Design and Optimization for Efficient Private Inference
In this section, we critically evaluate the current practices in baseline network design for efficient PI ($3.1 examines the selection of input networks for various ReL.unprunng methods (3.2) and highlights the limitations of fine- ane-grained ReLU optimization methods ($3.)) We further present our key observations,
underscoring the significance of network architecture and ReLUs' distribution for and-to-end PI performance
and motivate the need for redesigning the classical networks for efficient PI.
3.1 Addressing Pitfalls of Baseline Network Design for Efficient Private Inference
We begin by evaluating uniform and homogeneous channel scaling methods and their effectiveness in designing bascline networks for efficient PI. Subsequently, we investigate the impact of various channel scaling mothods on the ReLUs' distribution within network and motivate the need for heterogeneous channel scaling for optimizing FLOPs and ReLU counts simultaneously.
The conventional uniform channel scaling Icads to suboptimal ReLU efficiency Table 1 shows that the (slagewise) complexity of the network, quantified as #FLOPs and #Params (oaaansssRRsssaaa (Radosavovic et al., 2019), per units of ReLU noulinearity scales linearly with base channel count m, while a, 3, and y introduce multiplicative effect. This implics that for a given network complexity, a network widened by augmenting a, B, and y requires fewer ReLUs than the one widened by augmenting m. The uniform channel scaling in BaseCh networks, including WideResNet, often resorts to conservative (a, B, 7) (2, 2, 2), which limits the potential ReLU efficiency benefit from wider networks.
Homogeneous channel scaling offers superior ReLU efficiency until accuracy plateaus In contrast to BaseCh networks, homogeneous channel scaling in StageCh networks significantly improves ReLU efficiency by removing the constraint on (a, B, 7) (Figure 3 B()).. Nonetheless, the superiority of StageCh networks remains evident until reaching accuracy saturation, which varies with network configuration. In particular, as shown in Figure b), accuracy saturation for StageCh networks of ResNet18, ResNet20, ResNet32, and ResNet56 models begins at (a, 8, 7) (4, 4, 4), (5, 5, 5), (5, 5, 5), and (6, 6, 6), respectively, suggesting deeper StageCh network plateau at higher (ax, B, 7) values. This observations challenge the assertion made in Ghodsi et al. (020)). that model capacity per ReLU peaks at (a, 7) = (4, 4, 4). Thus, determining the accuracy saturation point a priori is challenging, raising an open question: To what extent can a network benefit from increased width for superior ReLU efficiency? Morcover, can employing RcLU optimization on StageCh networks elfectively address accuracy saluration?
Homogeneous channel scaling alters the ReLUs' distribution distinctively than uniform scaling We investigate the effect of uniform and homogeneous channel scaling on the ReLU distribution of networks. Unlike uniform scaling, which scales all layer ReLUs uniformly, homogeneous scaling leads to distinct ReLU distribution, with deeper layers exhibiting more significant changes. As depicted in Figure 3 (c,d), there is a noticeable decrease in the proportion of Stagel ReLUs, while Stage4 witnesses a significant increase. Given the ReLUs' criticality analysis in Table I0 this implies that the proportion of least-critical ReLUs is decreasing while the distribution of ReLUs among the other stages does not strictly adhere to their criticality order. This leads us the following observation:
Observation 1: Homogeneous channel scaling reduces the percentage of least-critical ReLUs in the network.
Hetcrogencous channel scaling is required for optimizing ReLU and FLOPs efficiency simulta- neously To answer the question of potential benefits from wider norksrks, we perform a sensitivity analysis and evaluate the influence of each stagewise channel multiplication factor on the network's ReLU and FLOPs efficiency. We systematically vary one factor at a time, starting from 2, while other factors are held constant at 2, in ResNet18 with m=16. We observe that augmenting a and values improves ReLU efficiency; notably, the latter optimizes the performance marginally better than the former until a saturation point is
6

---

Published in Transactions on Machine Learning Research (06/2024)
CIFAR- 100 Accumcy (%)
81
81
Al 2.6%
Stem
Suge2
Stgee
CIFAR- 10:100 Accuracy
4.2%
78
100
Suagel
Suget
78
Total RcLU ($
80
A1
75
75
ResNec18(SSagsCh)
60
((Hose()))
72
ResNa200Siage33) Ressee20(BaseCh)
72
ResNet12(Nngoo
40
20
128
250
512
1024
2048
128
256
512
1024
2045
#ReLUs (K)
#RcLUs (K)
mm:66
m=
m=
m=
128
m m56
(a) ReLU officiency: StageCh vs BaseCh
(b) Accuracy saturation in StageCh
(c) ReLUss distribution: BaseCh networks
Stem
CIFAR 100 Accunacy(S)
CIFAR 100 Accunccy
100
78-
78
Total R (%6
80
76
76
60
74
Alpha
74
Alpha
40
72
Beta
72
Beta
20
70/28
256 #ReLUss (K)
Gamma
70
Gamma
1024
2,2.2
3,3.3 4,4.4 5,3.5
6.6.6
7,7.7 8.8.8
(M)
(d) ReLUs' distribution: StageCh networks (e) Sensitivity analysis: ReLU efficiency (f) Sensitivity analysis: FLOPs efficiency
Figure 3: (a) Homogencous channel scaling in StageCh networks enables superior ReLU efficiency compared
to uniform channel scaling in BaseCh networks; however, (b) the accuracy in StageCh networks tends to plateau unpredictably. (c.d) Unlike uniform channel scaling, homogeneous scaling reduces the proportion of least-critical ReLUs in StageCh networks. (e,f) Each network stage affects ReLU and FLOPs efficiency differently, requiring heterogencous channel scaling for optimizing both ReLUs and FLOPs for efficient PI.
reached (see ()). Whereas, FLOPs efficiency is most effectively improved by augmenting a, outperforming 3 enhancements while augmenting values yields the worst FLOP efficiency (see 3 d)). This suggests that
FL.OPs in the deeper layers of StageCh networks can be regulated without impacting ReLU efficiency.
We note that the semi-automated designed networks RegNets (Radosavovic et al. 2020 employ heterogeneous channel scaling. However, they confine 1.5 <(a, 3, 7) S optimize FLOPs efficiency, which in turn
limits their ReLU efficiency (see Figure 11 [c)) Thus, despite line of seminal work the network's 's width
expansion Zagoruyko & Komodakis 2016 Radosavovic al. 2019; Lee et al. 2019 Dollár et al. 2021 the approaches to leverage the potential benelits of increased width for simultaneously optimizing ReLUss and FLOPs efficiency remains an open challenge. The above analyses lead us to the following observation:
Observation 2: Each network stage heterogeneously impacts both ReLU and FLOPs efficiency, a muancd aspect largely overlooked by prior channel scaling methods, rendering them inadequate for the simultaneous optimizing ReLUs and FLOPs counts for efficient private inference.
Strategically scaling channels by arranging ReLUs in their criticality order can regulate the FLOPs in deeper layers without compromising ReLU efficiency Following from the observations and 2, we propose to scale the channels until all ReLUs are aligned in the criticality order. Thus, Stage3 dominates the distribution as it has the most critical ReLUs, followed by Stage2, Stage4, and Stagel (Table 10 Unlike StageCh networks, widening beyond the point where the ReLUs are aligned in their criticality order does not alter their relative distribution (Figure 4 This leads to higher a and / values, which boost ReLU efficiency, with restrictive 7 (r <4) regulating FLOPs in deeper layers, promoting FLOP efficiency.
Consequently, approach of heterogeneous channel scaling achieves ReLU efficiency par with StageCh networks with fewer FLOPs. Figure (bcc demonstrates that the ReLUs' criticality-aware ResNet18 network 5x5x3x maintains similar ReLU efficiency with a 2x reduction in FLOPss compared to the StageCh network 5x5x5x. This FLOP reduction is consistently attained across the entire spectrum of ReLU counts, employing both fine-grained and coarse-graines ReLU optimization. These results lead to the following observation:
Observation 3: ReLUs' criticality-aware network widening method optimizes FLOPs efficiency without. sacrificing the ReLU efficiency, which meets the demands of efficient PI.
7

---

Published in Transactions on Machine Learning Research (06/2024)
Stem Stage
Stage Stage
Stage
67.5%
71.3%
76.4%
2048
80.1%
2048
1024
#F1 .T. (M)
72.3%
77.7%
19.4%
Total (
80
2x FLOPs
Stagee))
312
HLOPP
60
reduction
5 Sc5x3n(Papoeed)
256
66.5%
70.6%
77.0%
40
128
20
1024
67.5%
71.1%
74.1%
76.2%
64
66.2%
32
64
128
32
64
128
2,5,3 3,5,3 4,5,3 5,5,3 6,5,3 7,5,3
#RcLUs (K)
#ReLUs (K)
(a) HRN-5xxxxx (Proposed)
(b) FLOPs saving w/ SNL
(c) FLOPs saving w/ DeepReDuce
Figure 4: (a) Unlike StageCh networks, once the network's Rs are aligned in their criticality order, here at
point (a, B, (), 5, 3), increasing a does not alter their relative distribution. (b,c) ReLUs' criticality-awaro network widening method saves 2x FLOPs by regulating the FLOPs in deeper layers while maintaining ReLU efficiency over a wide range of ReLU counts.
3.2 Addressing Fallacies in Network Selection for ReLU Optimization
In this section, we explore the crucial aspects of selecting appropriate input networks for various ReLU pruming methods and perform a detailed experimental analysis to identify network attributes crucial for PI efficiency across different ReLU counts. This study aims to bridge the knowledge gap for designing efficient. baseline networks tailored to ReLU pruning methods.
Selecting the appropriate input network for ReLU optimization methods is far from intuitive Table 3 lists input networks used in previous ReLU optimization methods with their relevant characteristics, while Figure demonstrates how different input networks affect the performance of coarse (DeepReDuce)) and fine-grained (SNL.) ReLU optimization methods. For the former, accuracy differences of 12.9% and 11.6% are observed at higher and lower iso-ReLU counts. These differences cannot be ascribed to the FLOPs or accuracy of the baseline network alone. For instance, ResNet18 outperforms WidcRosNNt222cs despite having 4.4x fewer FLOPs and lower baseline accuracy, and ResNet32 outperforms VGG16 even though the latter has 4.76x more FLOPs and higher baseline accuracy.
Likewise, fine-grained ReLU optimization (SNL) exhibits significant accuracy differonces when employed on ResNets and WideResNets, especially al lower ReLU counts, as shown in Figure 5 b). While WideResNet models outperform beyond 200K ReLUs, there are 3.2% and 4.6% accuracy gaps at 25K and 15K ReLUs between ResNet18 and WideResNet16xx. The above empirical observation led to the following observation:
Observation 4: Performance of ReLU optimization methods, whether coarse or fine-grained, strongly correlates with the choice of input networks, leading to substantial performance disparities.
RoLU optimiistin method
Input netwurks
RedNell
WHNLIN
MobicNaVVi
Delphi Migan et al.
2:20
ResNet52
VOGlé
MobileNat22
DeepReDure Dlar
SAFEN
MIPT
RecNet52, VGG16
ResNet18
CIFARA 100 Accunicy
80
75
Sequeee Setiin (0)
SNI. Cho it mi
ReaNa18, WRN22
70
AAco=12.99
70
SENNe
2024
ResNet18,
2R22
3
AAce. Moc.=11.. 68
ReaNet82
ResNo118
WRN22.8
VGCI6
50 3 &
ResNer18
FLOPs
ReLUs
903
7MM
M09M
557K
2661M
1380
BHM
Acc
1.6%
WRN22x4
Ase
71.67%
79.0%%
$1.2756
75.0%%
16
32
64
128
256
512
60
16
32
64
WRN1688
128
236
Table 3: Baseline networks used for
#ReLUs (K)
BRel.Us (K)
advancing ReLU-Accuracy Parcto (CIFAR-100) in prior Pl-specific ReLU optimization methods.
(a) Dece at iso-
(b) SNL at iso-ReLU
Figure 5: ReLU optimization, whether coarse or fine-grained, perfor- mance exhibits significant disparities based on the input networks.
Key network attributes for PI efficiency vary across targeted ReLU counts To identify the key network altributes for PI efficiency across a wide range of ReLU counts, we examine three ResNet18 variants with identical ReLU counts but different ReLUs' distribution and FLOPs counts (Table 4 These are realized by channel reallocation, and the configurations 2x2x2x(m=32),, 4x4x4x(m=16), and 3x7x2x(m=16) correspond to stagewise channel counts as 122,64,188,266] [16, 64, 256, 1024], and [16, 48, 336, 672] respectively. We analyze their performance using the DeepReDuce and SNL ReL.U optinization, as shown in Figure 6
8

---

Published in Transactions on Machine Learning Research (06/2024)
A consistent trend emerges from both ReLU optimization methods: Wider models 4x4x4x(m=16) and 3x7x2x(m=16) outperform 2x2x2x at higher ReI.U counts; however, even with R 4x fewer FL.OPs, (m=32) excel at lower ReLU counts, This superior performance stems from the higher percentage (58.82%) of Icast-critical (Stagel) ReLUs in 2x2x2x(m=32). When targeting low ReLU counts, RcLU optimization ion methods primarily drop ReL.ss from Stagel (Jha el al. 2021 Cho et al. 2022b; Kundu ct al. 2023). Thus, networks with a higher percentage of Stagel ReLUs preserve more ReLUs from critical stages, mitigating accuracy degradation. Furthermore, this emphasizes the importance of strategically allocating channels, even when aiming for higher ReLU counts: 3x7x2x(m=16) matches the ReLU efficiency of (x=x4x(m=16) with 30% fewer FLOPs by allocating more channels to Stage3 and fewer to Stage4.
30
Model
Am:(R)
FLOPs
ReLA's
Stagrwise RoLUs' distilbutio
CIFAR 100 Sccuracy
76
CIFAR- 100 Acceracy
Stagel
S.aye2
Stagsl
Stagel
2x2c2x((0-222)
75.60
141M
27tK
20.53%
3,8%%
72
78.16
6013
279K
29.415
29.5895
23.50%
23.03%
3xhshe(w=:11) 16)
78:08
100M
260K
31.30%
18.9095
33.00%
16.545
(42
Table 4: A case study to investigate the Capacity- Siticality-Trraeeof Three Iso-ReLU ResNet18 networks with different ReLUs' distribution and FLOPs count, achieved by reallocating channels per stage. The baseline accuracy is for CIFAR- 100 dataset.
16
#ReLUs (K)
64
128
BRoLUs
128
(a) DeepReDucee at iso- ReL.U
(b) SNT. at Re-RR
Figure 6: Capacity-Critiiclitty Criticality-TTraded results: Figures
(a) and (b) show the ReLU-Accuracy tradcoff for networks in Table using DeepReDuce and SNL.
The above findings offer insight into the network selection for prior ReL.U optimization methods. Specifically, the choice of WRN22x8 (with 48.2% Stagel ReLUs) for higher ReLU counts while 18 for lower ReLU counts in fino-grained ReLU optimization (Cho et al.. 2022b Kundu et al. 2023). Morcover, it also explains the accuracy trends depicted in Figure 5(b), the higher the Stagel ReLU proportion (58.8% for ResNet18, 47.7% for WRN22x4, and 43.9% for WRN16x8), the higher the accuracy at lower ReLU counts.
Interestingly, we note that the above networks with a higher percentage of least-critical (Stagel) ReLUs inherently have fewer overall ReLUs (e.g.. 1392.6K for WRN22x8 and 557K ResNet18). This might suggest that these networks utilize their ReLUS more effectively, especially when there are fewer ReLUs, leading them to excel at lower ReLU counts. However, counter-example in Appendix E.2 reaffirms our conclusion for the key factor driving PI performance at lower ReLU counts. We further investigate the Capacity-Critticality Criticality- Tradeoff in Appendix E.1 and the additional results are shown in Figure 18 These analyses lead to the following observation:
Observation 5:
Wider networks are superior only at higher ReLU counts, while networks with higher percentage of least-critical ReLUs outperform at lower ReLU counts CrippacttyCCitiiialttyyy-tTe Tradeofff.
3.3 Mitigating the Limitations of Fine-grained ReLU Optimization
We now investigate the limitations of fine-grained ReLU optimization methods, often outperforming coarse- grained methods in conventional networks, and discuss the strategies to mitigate these limitations. This study aims to assess the efficacy of fine-grained methods beyond the conventional networks, especially with atypical ReLU distributions, for instance, when heterogeneous channel scaling is employed for simultaneously optimizing ReLU and FLOPs (see observation 3).
Fino-grained ReLU optimization is not always the best choice While fine-grained ReLU optimization has demonstrated its effectiveness in classical networks such 18 and Ned, especially when Stagel dominates the network's ReLU distribution (Cho et al. 2022b Kundu et al. 2023). its advantages are not universal. To better understand its range of efficacy, we compared it against DeepReDuce on PI-amenable wider models: 4x4x4x(m-16) and 3x7x2x(m=16)) (Table 4.
As shown in Figure a and 7(),, Dece outperforms SNL by a significant margin (up to 3%-4%). This suggests that the benefits of fine-grained ReLU optimization are highly dependent on specific ReLU distributions, and it reduces when Stagel does not dominate the network's ReLU distribution. This trend is also observed in ReLU criticality-aware networks, where Stage3 dominates the distribution of ReLUs (see
9

---

Published in Transactions on Machine Learning Research (06/2024)
This empirical evidence collectively suggests that fine-grained ReLU optimization might limit the benefits fils of increased network complexily introduced through stagewise channel multiplication enhancements. Nonetheless, the performance gap is less pronounced when the network's ' overall ReLU count is reduced by half by using UThinning (Jha et al. 2021). which drops the ReLUs from alternate layers.
Figure 20
(367.058)
RosNeeLS
x/ TL.
Veeiila
C100
[lseeliee
120K
76.02
140K
75:09
150K
120K
100K
40%
14.41
12.06
70K
1.48
Smmmon on1 liresd
70
+4.06%
+2.95%
+2.0%%
CIFAR-10 Asomay
@ 50
+3.22%%
26/2
+2.45%
(940 (SK)
RowNee3t
Vanilla w/ Th.
A
70.00
79.67
27.04
76.35
V2.66
76.55
77.67
76.26
77.52
15 47
78,09
71.55
14.17
70.32
$4.50
12.07
8:03
DocgReDDace
(varill))))
8
SNL (vanila))
DespRedieee
WiRN2te6 (1202.644K)
Yauillis
50.56
77.56
76.54
76.15
74,06
76.25
72.16
71.13
Thiniing ISNL.
Thimin INLL
k/ L.b.
A
78.50
41.9
75.01
133
75.6
1.77
75:
9.35
7%
3.07
77:12
1.84
2.78
94
72.74
1.61
UseLUs
#RoLUs (K)
(a) 4xdxxx((m=16)
(b) 3x7x2x(mm=1))
Table A significant accuracy boost (on CTFAR- 100) is achieved when ReLU-Thinning is employed prior to SNL, despite the less accurate ReLU- Thinned models. A = Acc(w/ Th.))Acc(Vanilla)
Figure DeepReDuce outperforms SNL. by a significant
margin (up to 4%) when altering network's ReLUs distri-
bution; however, using SNL on ReLU-Thimned networks reduces the accuracy gap.
Narrowing the search space improves the performance of fine-grained ReLU optimization To further examine the efficacy of ReI.U-Thinning for classical networks, we adopt a hybrid ReLU optimization approach, and ReLU-Thinning is employed before SNL optimization. Surprisingly, even when baseline Thinned models are less accurate, a significant accuracy boost (up to 3% at iso-ReL.Us) is obscrved, which is more pronounced for networks with higher #ReLUS (ResNeL34 and WRN22x8, in Table 5 Since ReL.U-Thnmiig drops the ReLUs from the alternate layers, irrespective of their criticality, its integration into existing ReLU optimization methodologies would not impact their overall computational complexity and remains effective for reducing the search space to identify critical ReLUs. This leads us to the following observation:
Observation 6: While altering the network's ReLU distribution can lead to suboptimal performance in fine-grained ReLU optimization, ReL d-Tnnningg emerges as an effective solution to bridge the performance gap, also beneficial for classical networks with higher overall ReLU counts.
4 DeepReShape
Drawing inspiration from the above observations and insights, we propose a novel design principle termed ReLU equalization (Figure { and re-design classical networks. This led to the development of family of models HybReNel, tailored to the needs of efficient PI (Table 16). Additionally, we propose R-LUsr, a (structured) channel-wise ReLU dropping method, enabling efficient PI at very low ReLU counts.
4.1 ReLU Equalization and Formation of HybReNet
Given a baseline input network, where ReLUs are not necessarily aligned in their criticality order, ReLU equalization redistributes the network's ReLUs their criticality order, meaning the (most) least critical stage has a (highest) lowest fraction of the network's total RcLU count (Figure B Equalization is achieved by an iterative process, as outlined in Algorithm [ In each iteration, the relative distribution of ReLUs in two stages is aligned in their criticality order by adjusting either their depth or width or both hyperparameters.
Specifically, for a network of D stages and predetermined criticality order, given compute ratios é1, #2,,.. Φn and stagewise channel multiplication factors \1, X2, ..., A(D-1) the ReLU equalization algorithm outputs a compound inequality after D-1 iterations. We now employ Algorithm on a standard four-stage ResNet18 model with the given criticality order as (from highest to lowest): Stage3 > Stage2 > Stage4 > Stagel (refer are adjusted, as
to Table 10p. During the equalization process, only the model's width wider models tend to be more ReLU efficient. Consequently, the algorithm yields the following expression:
10

---

Published in Transactions on Machine Learning Research (06/2024)
ReL distribution
Least-critical
ReLUs crilicaiity order
ReL U distributtin
Most-critical
(Slage3 Stage2 Stage4 Stage1)
Most-critical
ReLU-equalization
(Proposed)
Stage1
Stage2
Stage3
Stage4
Stage1
Stage2
Stage3
Stage4
Layer-index
Layer-index
Classical network (Baseline)
ReLU-equalized network (Proposed)
Figure 8: ratuon of NellU-quulizattin: Unlike classical networks (e-g-, ResNet), where ReLUss are not.
positioned is their criticality order, ReLU Veleuuullizztion aligns network's RsLUs in their criticality order.
Algorithm ReLU equalization
Input: Network Net with stages Sp: C' a sorted list of most least critical stage; stage-compute ratio @I: and stagewise channel multiplication factors A1,.. A(D-1)- Output: ReI.U-equalized versions of network Net.
1: for = to D-1 do
2:
b C[I] is most critical stage C[2) is second-most critical stage while (ReLUs(SS)) > #ReLUs(S.)) do ReLUUs in two stages are aligned in their criticality order
S = C[]
S. = c
Φ4
dxx
5:
Rearranging ReLUs by adjusting width and depth parameters
ge-1
6: end while
end for
return A set of on and d A(D-1 that satisfies the compound inequality: #ReLUs(C(11)
#ReLUs(C[2[) > ... #ReLUs(CIDD - 1]) > #ReLUs(CID))
#ReLUs(S.)) #
#ReI.Us(S))
ap 1
02 4 01 (B87
03
> Φ1
ReLU equalization through width (01 = 02 = 03 = 01 = 2, and a 2 2, 2, 2 2):
a3
aB7 >
aB 16, a 4, aB> 64. 4, By 16, and y Solving the above compound incqualitics provides the following (8, r) pairs and the range of a The (3,7) pairs are: (5,2) & a 2 7; (5,3) & 2 5; (6,2) 2 6; (7,2) & a 25
16
We obtain four pairs (3, r), each having a range of a value. We choose the smallest a needed for ReLU equalization, as increasing a beyond this point does not improve the performance when ReLU optimization is used; also, the relative distribution of ReLUs remains stable (see Appendix Ap. Thus, we achieve four baseline HybReNets: HRN-5x5x3x, HRN-5x7x2x, HRN-666x22, and HRN-7x5x2x. The architectural details of these four HRNs are presented in Table 16
4.2 ReLU-reuse
We further refine the baseline network's architecture to increase ReLU nonlinearity utilization by introducing ReI.U-reuse, which selectively applics ReLUs to contiguous subset of channels while the remaining channels reuse them. This approach differs from previous channel-wise ReLU optimizations, where channels are either uniformly scaled down throughout the network (Jha et al., 2021) | or only a subset of channels utilize ReLUs without reusing them (Cho et al.] 2022b). Our ReLU-reuse mechanism allows for efficient at extremely low ReLU counts (c.g., 3.2K RcLUs on CIFAR-100 datasct).
11

---

Published in Transactions on Machine Learning Research (06/2024)
FLOPs-balanced & ReLU- efficient baseline network for higher ReLU counts
FLOPs-balanced & ReLU- efficient baseline network for lower ReLU counts
FLOPs-balanced & ReLUU optimized networks for various ReLU counts
Input network with inferior ReLU-efficiency
512
1200
480
Input
#Channels
64
128
256
ReLU
#Channels
16
80
400
Capacity- Criticality- Tradeoff
#Channels
16
32
160
ReLU-
Culling. Thinning. and
with fewer (critical) ReLUs
Networks
network
equalization
optimization
S1 Sz S4
S1 S2 S4
-(2% 4%
S Sg Sy S
-(20x to 45x) FLOPs reduction & up to -5x ReLU reduction
ReLU-reuse
(proposed)
Up to -64x
disttribution
ReLUs distribution
accuracy
distribution
ReLU reduction
ReLUs'
Improvement at
ReLUs
Uss
4
3
S1
S4
S1 S4
Sj S2
Network with a given ReLUs' criticality order
Allocating channels to optimize FLOPs and ReLU efficiency simultaneously
Alloocating channels to maximize the proportion of least-critical ReLUs
Coarse-grained ReLU optimization
The DeepReShape network redesigning pipeline. ReLU's criticality aware strategic allocation of channels (gray boxes) outputs FT.OPs-balanced ReLUUefficiet baseline networks for various ReLU counts (blue boxes). Numbers in green denote criticality order (Stage3 is most critical).
Figure 10:
Specifically, feature maps of the layer are divided into N groups, and ReLUs are employed only in the last group (Figure 9). However, increasing the value of N results in a significant accuracy convolution being employed
Comy BN ReLLL
Dare BN
82,4,
Xg
(SN
-XX
AN
X
X,
Xg
X,
Xst
X1
X;
X2
du
X
Naive
Proposed
30
loss despite for cross-channel interaction. This is likely due to the loss of cross-channel information arising from more divisions in the feature maps (see our ablation study in Table 9). To address this issue, we devise a mechanism that decouples the number of divisions in feature maps from the ReLU reduction factor N. Precisely, one-fourth of channels are utilized for feature reuse, while Nth fraction of feature maps are activated using ReLUs, and the remaining feature maps are processed solcly with convolution operations, resulting in only three groups. It is important to note that using the ReLUs in the last group of feature maps increases the effective receplive field as these neurons can consider a larger subset of feature maps using the skip connections (Gao et al. 2019
3x3
ReLU Recustion
333
RaLu Reduction
m x do x
Y
X1
3/2
35
X5
X
X
Y;
mdf
ma
(loss of Information hightr N)
m4
(adlisioni Is
N
N
independent of N)
Figure 9: Proposed ReLU-reuse where ReLUs are selec-
tively reused across channels, reducing #ReLUs up to 16x.
4.3 Putting it All Together
We developed the DeepReShape framework to re-design the classical networks for efficiency PI across a wide range of ReLU counts. Figure 10 Given input network with specific ReLUs' criticality order, the ReLU- equalization step aligns the network's ReLU in their criticality order by adjusting width hyper-parameters. This step allows for maximizing ReLU efficiency without incurring superfluous FLOPPs by allocating fewer channels in the initial stages and increasing them in the deeper stages. In the second step, following the Uutfcalitt-Capacciyy Tradeoff, Tradeof, the width is adjusted such that Stagel dominates the ReLUs' distribution. This is achieved by a straight.foraad step: setting o=2 the ReL.U-equalized networks since decreasing a results in an incrcased percentage of Stagel ReLUs, and distribution of ReLUs in all but Stagel follow their criticality order (see Table 11.. This step allows for a substantial FLOP reduction, up to 45x, by allocating fewer channels in all the stages. We call the networks resulting from stepl and step2 HybReNets (HRNs). The bascline HRNs from step2 are: HRN-2x5x3x, HRN-2x7x2x, HRN-2x6X2x, and HRN-2x5x2x (Table 17
U-L---ttmmizatoon steps for HybReNets We choose to employ coarse-grained ReLU optimization steps in HRNs, as they outperform fine-grained ReLU optimization when the ReLU distribution undergoes changes in traditional networks, as shown in Figure 7 and Appendix F In particular, we climinate all the ReLUs from Stagel (ReLU Culling) if it dominates the network's overall ReLU distribution, e.g., HRNs with 2=2. For subsequent stages, we utilize ReI.U-Thinning, which removes ReLUs from alternate layers
12

---

Published in Transactions on Machine Learning Research (06/2024)
without considering their criticality. We further reduce the ReLU count by implementing ReLU-r with appropriate reduction factor (see Algorithm
Complexity analysis of HybReNet design For a D stage network with a predefieed criticality order for stagewise ReLUs, the process of ReLU equalization typically involves considering 2D-1 hyperparametens, including stage compute ratios and D-1 stagewise channel multiplication factors. However, for HRNs, this hyperparameter count is reduced to D-1 since ReLU equalization is achieved solely by modifying the network's width. Unlike SOTA network designing methods (Radosavovic et al. 2020 Liu et al., 2022), which build networks from scratch, the hyperparameters involved in ReLU equalization are determined by solving compound inoquality, climinating the need for additional network training. That is, to narrow down the design search space provided by bounds on a, 3, and 7, we select the minimum values of these hyper-parameters that satisfy the ReLU equalization conditions. Thus, our method leverages the existing network designs and optimizes them under PI constraints rather than designing them from scratch. Consequently, the complexity of designing HRNs can be characterized as (1). A detailed discussion is included in Appendix H.5
Additionally, employing coarse coarse-grained ReLU optimization does not exacerbate the complexity of HRNs. This is due to the positioning of ReLUs in HRNs based on their criticality order, which necessitates only single iteration (sec Algorithm 2]. In contrast, when ReLUs in the input network are organized without regard to their criticality order (e.g., classical networks such as ResNets and WedeResNets),, a single iteration produces suboptimal results, requiring D-1 iterations (Jha et al. 2021 Thus, the complexity of ReLU optimization for HRNss is reduced to ((1) from O(D).
5 Experimental Results
Analysis of HybReNets Pareto points Figure [ shows that HybReNet advances the ReLU-accuracy Pareto with a substantial reduction in FLOPS a factor overlooked in prior PI-specific network optimization. We present detailed analysis of network configurations and ReLU optimization steps and quantify their benefits for ReLUs and FLOP reduction. We use ResNet MaANNel1--basee 18-based HRN-5x5x3x for ReLU-accuracy comparison with SOTA PI methods in Figure ! as its FLOPs efficiency is superior to other HRNs (Table 16).
Table 6: Network configurations and ReLU optimization steps used for the Pareto points in Figure Accuracies (CIFAR-100) are separately shown for KD (Hinton et al. 2015) and DKD (Zhao et al. 2022 highlighting the benefits of improved architectural design and distillation method. denotes ReL.Urrusee
ReLU optimization steps
Accuracy(%
HybReNet
m
#ReLU
#FLOPs
Acc./ReLU
Culled
Thinned
Re2
KD
DKD
5x5x3x
16
NA
S1+S2+S3+S4
NA
163.3K
1055.4M
79.34
80.86
0.50
2x5x3x
32
S31
NA
104.4K
714.1M
77.63
79.96
0.77
2x5x3x
16
S1
S2+S3+S4
NA
52.2K
178.5M
74.98
77.14
1.48
2x5x3x
8
SI
52+53+54
NA
26. IK
44.6M
70.36
72.65
2.78
2x5x3x
16
S2+S3+S4
13.1K
121.6M
67.30
68.25
5.23
2x5x3x
16
S1
S21
8
6.5K
130.5M
62.68
63.29
9.70
2x5x3x
16
S1
S2+S3+S4
16
3.2K
137.2M
56.24
56.33
17.26
The key takcaway from Table 0 that tailoring the network features for PI constraint significantly roduces FLOPs and ReLUs. Specifically, lowering a value and base channel count led to 23.6x fewer FLOPs in HRN-2x5x3xx boost by employing a simple yet efficient logit-based distillation method DKD (Zhao et al. 2022). as the ReLU-reduced models greatly benefit from decoupling the target and non-target class distillation.
compared to HRN-5xxx3x(m=16)) Furthermore,
notice a significant accuracy
HybReNets outperform state-of-the-an in private inference Table presents competing design points for SENet (Kundu al. (023) and SNL (Cho al. (022b),, and we select HybReNet points (sce Table 6 and Table [3 for configuration and optimization details) offering both accuracy and latency benefits for fair comparison. The runtime breakdown is presented as homomorphic (HF) latency (Brakerski al. 2014
13

---

Published in Transactions on Machine Learning Research (06/2024)
Table 7: Comparison of HybReNet with SOTA in private inference: SENet (Kundu et al, 2023 and SNL (Cho et al. 2022bp. HybReNet exhibits superior ReLU and FLOPs efficiency and achieve a substantial reduction in latency. #Re and #FL denote ReLU and FLOPs counts; Acc. is top-1 accuracy; Lat. is the runtime for one private inference, including Homomorphic (HE) and Garbled-circuit (GC) latencies.
SOTA in Private Inference
HybReeNet (Ours)
Improvements
#Re
#FL
Aoc.
HE
GC
Lat.
#Re
#FL
Ace.
HE
GC
Lat.
#Re
#FL
Acc.
HE
GC
Lat.
300
2161
80.51
1001
33.7
1037
163
1055
80.56
770
18.4
788
188x
2.3x
0.3
1.3x
188x
1.3x
CIFAR-IN 100
SENets
240
2461
79.81
1004
27.0
1031
163
1055
80 86
770
18.4
758
1.5x
2.3x
1.1
1.3%
1.5x
1.3x
180
2461
79.12
1004
20.2
1024
163
1055
80,86
770
18.4
788
1.1x
2.3x
1.7
1.3x
1.1x
1.3 X
50
559
75.28
268
5.6
274
52
179
77.14
123
5.9
129
1.0x
3.1x
1.9
2.2x
0.9x
2.1x
25
559
70.59
268
2.8
271
26
45
72.65
19
2.9
52
0.9x
12.5x
2.1
5.5x
1.0x
5.2%
SNL SENels SNL
559
67.17
268
1.7
270
13
179
68. 25
123
1.5
124
1.1x
3.1x
1.1
2.2x
1.1x
2.2x
659
66.53
268
1.5
270
13
179
68. 25
123
124
1.0x
3.1x
1.7
2.2x
1.0x
2.2x
300
2227
64.96
927
33.7
961
327
1055
61.92
526
36.7
563
0.9x
2.1x
0.0
1.8x
0.9%
1.7x
Tinyl ImageNel
142
2227
58.90
927
16.0
104
179
58.
11.7
108
1.4x
12.4x
0.0
9.6x
1.4x
8.7x
189
9530
61.42
3690
55.0
37:15
653
1216
67.58
20/29
73.4
2102
0.7x
2.3x
3.2
1.8x
0.7x
188x
489
9830
61.42
3690
55.0
37:15
418
28-12
66.10
1307
45.0
1352
1.2x
3.5x
1.7
2.8x
1.2%
2.8x
298
2227
64.04
927
33.5
961
327
1055
64.92
526
36.7
563
0.9x
2.1x
0.9
1.8 x
0.9%
1.7x
100
2227
58.94
927
11.2
933
104
179
58.00 .0
97
11.7
108
1.0x
12.4x
0.0
9.6x
1.0x
8.7x
59
2227
54.40
927
6.6
934
52
712
54.46
329
5.9
335
1.1x
3.1x
0.1
2.8%
1.1x
2.8%
arises from linear operations (convolution and fully-commettdd layers), and Garbled-cireuit (GC) latency BBlll et al.
2019 resulting from ReLU computation. See the experiential setup details in Appendix
On CIF,R-100, SENet requires 300K ReLUs and 2461M FLOPs to reach 80.54% accuracy, whereas HRN- 5x5x3x achieves 80.86% accuracy with only 163K ReLUs and 1055M FLOOS,, providing 1.8x ReLU and 2.3x FLOPs saving. Similarly, at 25K ReLUs, our approach achieves 2.1% accuracy gain with 12.5x FLOP reduction, thereby saving 5.2x runtime. Even at an extremely low ReLU count of 13K, HRN is 1.7% more accurale and achieves 2.2x runtime saving, compared to the SNL.
On TinylmageNet, HybReNets outperform SENet at both 300K and 142K ReLUs, improving runtime by 1.7x and 8.7x, respectively. Compared to SNL 489K ReLUs, HybReNets are 3.2% (1.7%) more accurate with a L.8x (2.8x) reduction in runtime. At lower ReLU counts of 100K and 59K, HybReNets match the accuracy with SNL and achieve 12.4x and 3.1x FLOP reduction, which results in 8.7x and 2.8x runtime improvement, respectively.
Our primary insight from Table 7 is that FLOP reduction does not inherently guarantee a proportional reduction in HE latency, whereas a direct correlation exists between ReLU reduction and GC latency savings. In particular, a 12.55 FLOP reduction translates to 5.2x and 8.7x latency reduction on CIFAR-100 and TinyImageNer respectively. This is due to the fact HE latency has an intricate dependency on the input/output packing (Aharoni et al. 2023), , rotational complexity {Lou et al. 2020b [uang et al.) 2022 and slot utilization (Lee et al. 2022 We refer the readers to Juvekar et al. (018)) for details.
Generality case study on ResNet34 We select ResNet34 for the DeepReShape generality study for two key reasons: (1) its consistent use for the case study in prior Pl-specific network optimization studies (Jha
et al.
Cho et al.] 2022b; Kundu et al.] 2023), and (2) its stage compute ratio (01=3, 62=4, 6 and 01 =3 distinguishes it from R8sNet18, results in different sets of HRN networks, HRN-4x6x3x and
2021
HRN-4x9x2x, upon applying Algorithm D We use HRN-4x6x3x for comparison with SOTA in Table g
Network configuration and ReLU optimization details are presented in Table 14
HybReNet advances the ReL Pareto both CIFAR-100 and Tiny Net, shown in Figures (a, b). Table R quantifies the FLOPP-RRLU--Accreee benefits and runtime savings. On CIFA compared to SOTA, HybReNet improves runtime by 3.1x x with a significant gain in accuracy 9.8%, 7.2%, 5.9%, and 2.1% at 15K, 25K, 30K and 50K ReLUs (respectively). Further on TinylmageNet, SNL requires 300K ReLUs and 4646M FLOPs to reach 64% accuracy, whereas HybReNet matches this accuracy with 8.8x fewer FLOPs, leading to runtime improvement of 6.3x. Conclusively, highlights the effectiveness of DeepReShape and validates its generality for different network configurations and datasets.
14

---

Published in Transactions on Machine Learning Research (06/2024)
Tiny Imageee Accuracy (1)
70
83
80-
CIFAR 100 Acquracy
CIFAR-100 Accutacy
65
76
81
60
72
HRN-T
IRRN-fodets
SNLICCEB 23)
Com-Nec- V2
79
HIRN-50722
64
16
61
256
50
64
128
256
312
1024
78156
512
1024 2048 4096 #ReLUs (K)
8192
1638
#ReLUs (K)
#ReL.Us (K)
(a) ResNet34 on CIFAR-100
(b) ResNet34 on TinylmageNet
(c) HRN vs RegNet on CIFAR-100
Figure 11: HybReNets outperform SOTA ReLU-optimization methods applied to ResNet34 and also surpass
SOTA FLOPs efficient models: RegNets and ConvNeXt-V2 (Sec Table 14 for the Parcto points specifics.).
Table 8: ResNet34- HybReNets outperform SOTA P1 methods (Kundu et al. 2023 Cho et al. 2022b
employed on ResNet34, and also surpass the SOTA FLOPs efficient models ConyNeXt-Vi (Woo al. #Re and #FL denote ReLU and FLOPs counts; Acc. is top-1 accuracy; Lat. is the runtime for one PI.
2023
SOTA in Private Inference (on ResNet34)
HyhiteNet(Ourr
Improvements
#Re
#FL
Arc.
HE
GC
Lat.
#Re
#PL
Acc.
HE
GC
Lat.
#Re
#FL.
Acc.
HE
GC
Lat.
CouvNeXi SNL. SNL SENet
1162
78.80
459
22.5
482
134
527
79.56
404
15.1
419
1.5x
2.2
1.1x
1.5x
1.x
80
1162
76.66
459
9.0
468
67
132
76.91
140
7.5
148
1.2%
8.8x
0.3
3.3x
1.2x
3.2x
CIFAR-
50
1162
74.84
459
5.6
465
67
132
76.91
140
7.5
148
0.7x
8.8x
2.1
3.3x
0.7x
3.1x
30
1162
71.00
459
3.4
462
67
132
76.91
140
7.5
148
0.4%
8.8x
5.9
3.3x
0.4x
3.1x
25
1162
69.68
459
2.8
462
132
76.
140
7.5
148
0.4x
8.8x
7.2
3.3x
0.4x
3.1x
15
1162
67.08
159
1.7
76.91
7.5
148
0.2x
8.8x
9.8
3.3x
3.1x
4646
65.34
56.2
1766
537
2109
67.48
60.3
940
0.9%
1.9x
400
4646
65.32
1710
45.0
1755
5:7
2109
67.48
880
60.3
940
0.7%
2.2%
2.2
1.9x
0.7x
2.3%
ThylnageNet
300
4646
63.99
1710
33.7
1744
268
629
64.02
245
30.2
275
1.1%
8.8x
0.0
7.0x
1.1x
6.3%
200
1646
62.49
1710
22.5
1733
268
529
64.02
245
30.2
275
0.7%
8.8x
1.5
7.0x
0.7x
6.3x
1622
11801
09.85
1007
182.4
4219
1270
5211
70.29
3091
142.8
3233
1.3x
1.1x
0.1
1.3x
1.3x
13x
1278
9080
68.75
143.7
2512
952
4638
69.15
1837
107.1
1944
2.0x
1.3x
1.3x
721
3436
67.08
1307
81.0
1358
537
2109
67.48
880
60.3
940
1.3x
1.6x
1.5x
1.3x
1.6x
511
1935
65,72
738
00.8
799
402
1187
65.77
592
45.2
637
1.3x
1.6x
0.0
1.3x
1.3x
1.3x
451
1315
61.07
546
50.7
397
268
529
61.02
245
30.2
275
1.7x
2.5x
0.0
2.2x
1.7x
2.2x
HybReNet outperform SOTA FLOPs efficient vision models We perform comparative analysis of HybReNets with SOTA FLOPs efficient vision models: ConyNeXt-VV2 (Woo et al. 2023) and RegNet RRadsavvvvcc et al. 2020). These models possess distinct depth and width hyperparameters. providing an interesting case study, particularly when contrasted with conventional ReNets. See Appendix H.4 for details.
For fair comparison with baseline RegNet-X models, we do not employ any ReT.U-optimmzation steps on (ResNet18-basse based) HybReNets. Results are shown in Figure [[(c) where HRNS are evaluated with m {16, 32, 64 }. HRNs achieve comparable accuracy with substantially fewer ReLUs compared to RegNets. For instance, to achieve 78. 78.26% (80.63%) accuracy on CIFAR-100, RegNets require 1460K (6544K) ReLUs, while HRN-5x5x3x needs only 343K (1372K) ReLUs, leading to a 4.3x (4.7x) ReLU reduction.
Further, we compare the ComyNeXt-V2 models with HybReNets on TinyImageNet while employing ReLU optimization on them (see Table 14 for optimization details). The ReL.Uaccuracy Pareto shown Figure [b),, with a detailed comparison outlined Table B The competing HRNs achieve 1.3x to 1.7x ReLU savings; 1.4x to 2.5x FLOP reduction, which results in 1.3x to 2.3x runtime improvements.
ReLU-reuse is more effective for HybReNets and outperforms the SOTA channel-wise ReLU optimization We examine the efficacy of ReLUUr on networks with various ReLUs' distributions and compare their performance with conventional (chamnel/feature-maal (aaatuure-apsscaliin used in DeepReDuce for achieving very low ReLU counts. Results are shown in Figure 22 and Figure 23 (Appendix]]. Interestingly, we observed that the efficacy of ReLU-reuse is most pronounced networks where ReLUs are aligned in their criticality order, whether partially or entirely. In fact, networks with an even distribution of stagewise ReLUs exhibit more significant, accuracy improvements from Rel.U-reuse compared to traditional networks like NesNets.
15

---

Published in Transactions on Machine Learning Research (06/2024)
CIFAR-IO0 100 Accuracyss)
80
CIFAR- 1..00 Acquracy
80
CIFAR-100 12-100 Acuuacyy
80
76
72
72
72
68
SNLíchanne)
68
SNLichanne)
SNLichamel)
64
SNL(vanilla)
64
SNL(vanllla)
64
SNL(vanilla)
60
Ours(HRN who Re2)
60
Ours(HRN wio Rel)
60
Quis(HRN wio Re2)
56
OunsHHRN w/ Re2)
56
Ours(HRN w/ Re2)
56
Ours(HRN Rc2)
16
32
128
T6
32
64
128
32
64
128
#ReLUs (K)
#RcLUs (K)
#ReLUs (K)
(a) HRR-2xxx2x
(b) HRRN-266x3x
(c) HRN-2x6X2x
80
80
80
CIFAR- 100 Accurncy
CIFAR 100 Accuracy
CIFAR 100 Accuracy
76
76
76
72
72
68
SNL(chamel)
68
SNL(channel)
SNL/Chmmnel
64
SNL(vanilla)
64
SNL(vanlll))
SNL(vanilla)
60
OurSHHR who Rc2)
60
OursHHRN wio Rc2)
Ouis(HR wio Rc2)
OuassHRN w/ Re2)
Ours(HRN wio Re2)
60
Qurs(HRN w/ Re2)
56
16
64
128
56
16
32
64
128
8
16
32
64
#ReLUs (K)
#ReLUS (K)
HReLUs
(d) HRN-2x7x2x
(e) HRN-2x6x3x
(f) HRN-2x9x2x
Figure 12: ReLU-reuse (Re2) consistently outperforms the SOTA channel-wise ReLU dropping technique used in SNL across various ReLU counts. Substituting the conventional scaling method used in DeepReDuce (denoted as "w/o Re2") with Re2 results in an accuracy gain of 1% 3%, bringing the performance closer to the pixel-wise SNL (denoted as "SNL(vanilla"").
Further, we employ ReLU-reuse on HRNs with a=2, as per Algorithm 2 and compare their performance with SOTA channel-wise ReLU optimization method used in SNL. For a fair comparison, we use standard knowledge distillation (Hinton et al., 2015), as used in SNI rather than DKD (Zhao al. 2022.. Figure 12 demonstrates that Re2 results in significant accuracy improvement of up to 3%. This gain in accuracy enables HRNs to achieve performance on par with pixel-wise SNL.
A Ablation study for ReLU-reuse We conduct an ablation study on ResNet18 to investigate the benefits of two key techniques employed in ReLU-reuse: (1) shortcut connections between outputs and inputs of feature maps regardless of the ReLU reduction factor. We removed ReLUS from alternate layers using ReLU-Thinming and integrated ReLU-reuse in the others, and results are shown in Table The results show that shortcut connections boost accuracy at lower ReLU reduction factors, but their benefit diminishes with higher reduction factors. Specifically, accuracy drops by 1.5% when the reduction factor increases from 2x to 4x. This reduction is likely due to the significant loss of cross-channel information with more divisions in feature-map.
subsequent feature-subspaces (see Figure 9), and (2) using a fixed number of divisions
Table 9: Results for an ablation study where ReLUer is employed in alternate convolution layers in (i.e., ReLU-Thimed) ResNet18 (CIFAR-100). The constant number of divisions (i.e., 3) in the proposed approach of ReL.uredntiion offers scolability for higher ReLU reduction factors. The term reuse in the table refers to shortcut connections between feature-subepaces in N partitions (see Figure 9
N divisions
Proposed
ReLU-reduction factor
#ReLUs
w/o Reuse
w/ Reuse
(3 divisions)
2x ReLU reduction (N=2)
434.18K
77.61%
78.19%
77.83%
4x ReLU reduction (4=
372.74K
75.84%
76.87%
77. 77.60%
8x ReLU reduction (N=8)
342.02K
75.43%
75.66%
76.93%
16x ReLU reduction (N=16)
326.66K
75.33%
75.47%
76.38%
PRAM (Post-ReLU Activation Mismatch) loss in conjunction with standard KD (Hinton 2015 for additional boost in the accuracy of ReI.U-redueed models. In contrast, both SNL. Cha al. 2022b) and DeepReDDcee Tha et 2021 rely solely on standard KD.
SIt important note that SENets
Kundu et al.
2023
16

---

Published in Transactions on Machine Learning Research (06/2024)
On the other hand, fixed number of divisions in our proposed approach stabilizes the accuracy degradation even at higher ReLU reduction factors, emphasizing their scalability for achieving significantly lower ReLC reductions. Note that, at a reduction factor of 2, the U-ruse technique demonstrated slightly lower accuracy than the N division method with shortcut connections. This is because the latter consists of only two groups of feature maps, while the former has three, which resulted in more information loss.
The baseline HybReNets exhibits superior ReLU efficiency compared to the standard networks used in private inference
We evaluated the ReLU efficiency of baseline HRRS without lever- aging any coarse or fine-grained ReLU optimization methods, as well as knowledge distillation. We compared them with two widely used network architectures in PI: ResNet and WideResNets. Re-
CIFAR-100 100 Accuracy
82
80
78
76
WidsReNNe-Iakk
WidcResNer
sults are shown in Figure 13 The homogeneous channel scaling in
74
ResNet18 StageCh networks led to superior ReLU efficiency than
72
Has:ChcResNert88
WideResNets variants until accuracy in the former is saturated. Nonetheless, all the four HRNs HRNS-HHR--xxsee HRN-5x5x3x, HRN-5x7x2x, HRN--6x622x, and HRN-7x5x2x exceeds the ReLU efficiency of ResNet18 StageCh variants, demonstrating the benefits of strategically allocating channels in the subsoquent stages of the classical networks for PI.
70
256
512
HRNs (Proposed)
1024
2048
#Rel (K) Figure 13: ReLU efficiency comparison
with bascline HybReNets. For WideRes-
Nets l E {2, 4, 6, 8, 10, 12].
6 Related Work
PI-specific network optimization Delphi (Mishra et al. 2020 SAFENct (Lou et al. 2021 and Garimella
et al. 2021) substitute ReLUs with low-degree polynomials, while AutoFHE Ao & Boddeti 2024) performed
layerwise mixed-degree polynomial substitution. Ghodsi et al. (021 proposed stochastic ReLU, proba bilisii
approximation of ReLU functions, and co-optimized the garbled circuits. DeepReDuce (Jha et al. 2021
manual roarse-grained ReLU optimization method, drops ReLUs layerwise. SNL (Cho et al. 2022b) and
SENet (Kundu et al. 2023 are fine-grained ReLU optimization and drop the pixel-wise ReL.Us. CryptoNAS Ghodsi et al. 2020 and Sphyux (Cho et al. 2022ah use neural architecture search and employ a constant number of ReLUs per layer for designing ReLU-efficient networks, disregarding FLOPs implications. In contrast, our approach achieves ReLU and FLOP efficiency simultaneously. We refer the reader to Ng & Chow 2023 for detailed HE and GC-specific optimizations for private inference. A recent work Zeng et al.
(2023b used oblivious transfer for nonlinear operations and rotation-free homomorphic encryption (Huang ct al.
2022 for linear layers, and showed that communication cost is dominated by linear operations.
Benefits of width The impact of network width on reducing catastrophic forgetting was highlighted by Mirzadeh et al. (2022). The influence of network width on the smoothness of the loss surface was analyzed by Li et al. (2018). and it was found that an increase in width could mitigate erratic behavior in the loss landscape. A study by Golubeva et al. (2021) decoupled the effects of increased width from over-parameterization and found that the width of network primarily determines its predictive performance, with the number of parameters being secondary factor under mild assumptions. Nguuen et al. (2021) established that wider networks, when delivering similar levels of accuracy on the ImageNet dataset, show superior performance on inputs that reflect the scene rather than the objects.
Challenges and implications of nonlinear layers in diverse neural network applications Nonlinear layers not only present challenges in private inference; they introduce significant hurdles across various neural network applications. For instance, in the realm of optical neural networks, ReLUs exacerbate energy consumption and increase latency due to the costs associated with aptical-to-electricaa signal conversions, which in turn diminishes the overall system efficiency (Chang et al.) 2018 Li et al. 2022). When it comes to verifying adversarial robustness, the prevalence of ReLUs can make the process notably more time-intensive.
This increase in complexity arises from the higher proportion of unstable neurons (Xiao et al. 2019: Balunović
& Vechev 2020 Chen et al. 2022a).
Additionally, ReLUs considerably hinder the progress of verifiable machine learning because its arttheitcmc operations are incompatible with zero-knowledge proof systems (Sun & Zhang 2023). and prior work has
17

---

Published in Transactions on Machine Learning Research (06/2024)
resorted to employing polynomial approximations (Ali et al. 2020 Zhao et al. 2021 Eisenhofer et al. 2022 or have implemented methods based on lookup tables (Liu et al., 2021 Kang et al. 2022). Furthermore, the non-distributive nature of ReLU over rotation operations can break the equivariance property of Steerable CNNs (Franzen & Wand 2021], known for their parameter and computation efficioncy (Cohen & Welling
2017
Weiler al. 2018 Weiler & Cesa 2019); thus, limiting their architectural choices and applicability.
Thus, the ReLU optimization techniques of DeepReShape not only address the challenges in private inference but also hold promise for broader applications, suggesting its versatility and potential for widespread impact.
7 Discussion
Conclusion and broader impact Privacy-preserving computations demand substantial resources, particu- larly in terms of storage, communication bandwidth, and compute power. Using the garbled-circuit technique alone can consume hundreds of gigabytes of storage, while homomorphic computations might need hours to complete single private inference in real-world scenarios (Rathee et al.
2020, Garimella et al.
2023
Researcher have proposed specialized hardware accelerators (Samardzic et al. 2021 2022 Soni et al. 2023
Mo et al. 2023 Kim et al.] 2023 Agrawal et al. 2024 Putra et al. 2024] and (cryptographic) protocol improvements to tackle these challenges. Yct, these solutions present challenges of their own: hardware solutions may not always be sustainable in the long run (Gupta et al.) 2022), and protocol tweaks could potentially open doors to security vulnerabilities or raise compatibility concerns.
In this context, our rescarch shifts the focus towards algorithmic innovations and aims to address the unique challenge of reducing FLOPs without compromising ReLU efficiency. We proposed DeepReShape to optimize FLOP count while maintaining ReLU efficiency effectively. We achieve this by identifying superfluous FLOPP in conventional ReLU efficient networks and understanding that wide networks are mainly beneficial for higher ReLU counts, providing additional opportunities for FLOP reduction when targeting lower ReLU counts. By leveraging these insights, we achieve FLOP reduction up to 45x without any bells and whistles.
One significant advantage of algorithmic improvement is their adaptability across diverse hardware configu- rations and pryplogrephic protocols, thus broadening the potential impact of our algorithmic innovations. We showed that a substantial reduction in runtime, ~(5x to 10x), ), can be achieved simply by strategically allocating channels in the existing networks and employing straightforward ReLU optimization steps.
Furthermore, as discussed S nonlinear layers are a bottleneck also in other areas of machine learning privacy and security. Thus, our work on the simultaneous optimization of ReLU and FLOPs holds promise for broader applications in these fields.
Limitations Achieving a specific target ReLU count with HybReNets is challenging due to the coarse-grained nature of ReLU optimization steps. Fine-grained optimization leads to suboptimal performance in HybReNets because of changes in the ReLUs' distribution compared to conventional CNNs. Coarso-grained ReLU optimization steps either halve the network's ReLU count or remove all ReLUs from Stagel, depending on the ReLU distribution in the network (sce Algorithm 2). Consequently, the final ReLU count depends on the baseline network's initial ReLU count and their distribution within the network, influenced by the base channels and stage-wise channel multiplication factors.
Future work Developing Pl-efficient networks from scratch could yicld more optimized networks for PI performance; however, it requires exhaustive design space exploration and the training of multiple subnetworks to successively narrow the search space. This makes it computationally intensive process. Nonetheless, there is a significant potential for creating families of networks tailored for optimal Pl performance. Furthermore, additional roductions in FLOPs can be achieved by employing techniques such as lincar layer fusion, as demonstrated in (Jha al. 2021 Dror et. al. 2021 Zeng el al. 2023a
Acknowledgment
We would like to thank Karthik Garimella for his assistance in computing the runtime (HE and GC latency) for private inference. This research was developed with funding from the Defense Advanced Research
18

---

Published in Transactions on Machine Learning Research (06/2024)
Projects Agency (DARPA), under the Data Protection in Virtual Environments (DPRIVE) program, contract HR0011--1199-003. The views, opinions and/or findings expressed are those of the author and should not be interpreted as representing the official views or policies of the Department of Defense or the U.S. Government.
References
Rashmi Agrawal, Anantha Chandrakasan, and Ajay Joshi. Heap: A fully homomorphie eneryption accelerator with parallelized bootstrapping. 2024.
Ehud Aharoni, Allon Adir, Moran Baruch, Nir Drucker, Gilad Ezov, Ariel Farkash, Lev Greenberg, Ramy Masalha, Guy Moshkowich, Dov Murik, et al. Helayers: A tile tensors framework for large neural networks on encrypted data. Proceedings on privacy enhancing technologies, 2023.
Yoshimasa Akimoto, Kazuto Fukuchi, Youhci Akimoto, and Jun Sakuma. Privformer: Privacy-preservee transformer with mpc. In IEEE 8th European Symposium on Security and Privacy (FuroS8P) 2023.
Ramy E Ali, Jinhyun So, and A Salman Avestimehr. On polynomial approximations for privacy-presevving and verifiable relu networks. arXiv preprint Xii::001.055590 2020.
Wei Ao and Vishnu Naresh Boddeti. Autofhe: Automated adaption of cmns for efficient evaluation over fhe. In 33rd USENIX Security Symposium, 2024.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey Hinton. Layer normalization. orXiv preprint orXiv:1607.04, 2016.
Marshall Ball, Tal Malkin, and Mike Rosulek. Garbling gadgets for boolean and arithmetic circuits. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, 2016.
Marshall Ball, Brent Carmer, Tal Malkin, Mike Rosulck, and Nichole Schimanski. Garbled neural networks are practical. Cryplology ePrinl Avehive, 2019.
Mislav Balunovié and Martin Vechev. Adversarial training and provable defenses: Bridging the gap. In International Conference on Learning Representations, 2020.
Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine-learning practice and the classical bias-variance tradc-off. Proceedings of the National Academy of Sciences, 2019.
Zvika Brakerski, Craig Gentry, and Vinod Vaikuntanathan. (leveled) fully homomorphic encryption without bootstrapping. ACM Trunsactions on Compulation Theory, 2014.
Julie Chang, Vincent Sitzmann, Xiong Dun, Wolfgang Heidrich, and Gordon Wetzstein. Hybrid optical- electronic convolutional neural networks with optimized diffractive optics for image classification. Scientific reports, 2018.
Tianlong Chen, Huan Zhang, Zhenyu Zhang, Shiyu Chang, Sijia Liu, Pin- Yu Chen, and Zhangyang Wang. Linearity grafting: Relaxed neuron pruning helps certifiable robustness. In International Conference on Machine Learning, 2022a.
Tianyu Chen, Hangbo Bao, Shaohan Huang, Li Dong, Binxing Jiao, Daxin Jiang, Haoyi Zhou, Jianxin Li, and transformer inference with homomorphic eneryption. In Findings of
Furu Wei. THE-X: the Association for Computational Singuistics(ACLL 2022b.
Jung Hee Cheon, Andrey Kim, Miran Kim, and Yongsoo Song. Homomorphic encryption for arithmetic of approximate numbers. In International conference on the theory and application of cryptology and information securily, 2017.
Minsu Cho, Zahra Ghodsi, Brandon Reagen, Siddharth Garg, and Chinmay Hegde. Sphynx: Relu-efficient network design for private inference. IEFE Security & Privacy, 2022a.
19

---

Published in Transactions on Machine Learning Research (06/2024)
Minsu Cho, Ameya Joshi, Siddharth Garg, Brandon Reagen, and Chinmay Hegde. Selective network linearization for efficient private inference. In International Conference on Machine Learning, 2022b.
Taco S. Cohen and Max Welling. Steerable CNNs. In International Conference on Learning Representations, 2017.
Daniel Demmler, Thomas Schneider, and Michael Zohner. Aby-a framework for efficient mixed-protore secure two-party computation. In The Network and Distributed System Security Symposium, 2015.
Piotr Dollár, Mannat Singh, and Ross Girshick. Fast and accurate model scaling. In Proceedings of the IEFE/CVF Conference on Computer Vision and Pattern Recognition, 2021.
Amir Ben Dror, Niv Zehngut, Avraham Raviv, Exgeny Artyomov, Ran Vitek, and Roy Josef Jevnisek. Layer folding: Neural network depth reduction using activation linearization. In British Machine Vision Conference, 2021.
Thorsten Eisenhofer, Doreen Riepel, Varun Chandrasekaran, Esha Ghosh, Olga Ohrimenko, and Nicolas Papernot. Verifiable and provably secure machine unlearning. arXiv preprint 0tt::2210.91200 2022.
Junfeng Fan and Frederik Vercauteren. Somewhat practical fully homomorphic encryption. Cryptology e ePrint Archive, 2012.
Xiaoyu Fan, Kun Chen, Guosai Wang, Mingehun Zhuang, Yi Li, and Wei Xu. Nfgen: Automatic non-linear function evaluation code generator for general-purpose mpe platforms. In Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security, 2022.
Daniel Franzen and Michael Wand. General nonlinearities in squuveequivariant cnns. In Advances in Neural Information Processing Systems, 2021.
Shanghua Gao, Ming-Ming Cheng, Kai Zhao, Xin- Yu Zhang, Ming-Hsuan Hsuan Yang, and Philip HS Torr. Res2net: A new multi-scale backbone architecture. In IEEE transactions on pottern analysis and machine intelligence, 2019.
Karthik Garimella, Nandan Kumar Jha, and Brandon Reagen. Sisyphus: A cautionary tale of using low-degree polynomial activations in privacy-preee prreeerirrs deep learning. In ACM CCS Workshop on Private- preseving Machine Learning, 2021.
Karthik Garimella, Zahra Ghodsi, Nandan Kumar Jha, Siddharth Garg, and Brandon Reagen. Characterizing and optimizing end-to-end systems for private inference. Tn Proceedings of the 28th ACM International Conference on Architectural Support for Progromming Languages and Operating Systems, 2023.
Craig Gentry et al. A fully homomorphic encryption scheme. 2009.
Zahra Ghodsi, Akshaj Kumar Veldanda, Brandon Reagen, and Siddharth Garg. CryptoNAS: Private inference on a relu budget. In Advances in Neural Information Processing Systems, 2020.
Zahra Ghodsi, Nandan Kumar Jha, Brandon Reagen, and Siddharth Garg. Circa: Stochastic relus for private deep learning. In Advances in Neural Information Processing Systems, 2021.
Anna Golubeva, Guy Gur-Ari, and Behnam Neyshabur. Are wider nets better given the same number of parameters? In International Conference on Learning Representations, 2021.
Kanav Gupta, Neha Jawalkar, Ananta. Mukherjee, Nishanth Chandran, Divya Gupta, Ashish Panwar, and Rahul Sharma. Sigma: Secure gpt inference with function secret sharing. Oryptology ePrint Archive, 2023.
Udit Gupta, Mariam Elgamal, Gage Hills, Gu-Yeon Wei, Hsien-Hsii S Lee, David Brooks, and Carole-Jean Wu. Act: Designing sustainable computer systems with an architectural carbon modeling tool. In Proceedings of the 49th Annual International Symposium Computer Architecture, pp. 784-7984749 2022.
Meng Hao, Hongwei Li, Hanxiao Chen, Pengzhi Xing, Guowen Xu, and Tianwei Zhang. Iron: Private inference on transformers. In Advances in Neural Information Processing Systems, 2022.
20

---

Published in Transactions on Machine Learning Research (06/2024)
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEFE conference on computer vision and pollern recognilion, 2016.
Yang He, Yuhang Ding, Ping Liu, Linchao Zhu, Hanwang Zhang, and Yi Yang. Learning filter pruming criteria for deep convolutional neural networks acceleration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2020.
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). aiv preprint, 2016.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge a neural network. arXiv preprint Sal.1550..22344 2015.
Xiaoyang Hou, Jian Liu, Jingyu Li, Yuhan Li, Wen-jie Lu, Cheng Hong, and Kui Ren. Ciphergpt: Secure two-party gpt inference. Cryptology ePrint Archive, 2023.
Andrew Howard, Mark Sandler, Grace Chu, Hlang-Chiell Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudovan, et al. Searching for mobilenetv3. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019.
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. aixi preprint, 2017.
Zhicong Huang, Wen jie Lu, Cheng Hong, and Jiansheng Ding. Cheetah: Lean and fast secure Two-Party deep neural network inference. In 31st USENIX Security Sympostum, 2022.
Nandan Kumar Jha, Zahra Ghodsi, Siddharth Garg, and Brandon Reagen. DeepReDuce: Relu reduction for fast private inference. In International Conference on Machine Learning, 2021.
Chiraag Juvekar, Vinod Vaikuntanathan and Anantha Chandrakasan. Gazelle: A low latency framework for secure neural network inference. In 27th USENIX Security Symposium, 2018.
Daniel Kang, Tatsunori Hashimoto, Ion Stoica, and Yi Sun. Scaling up trustless dnn inference with zero-knowledge proofs. Xiv preprint arXic:22100080% 2022.
Jongmin Kim, Sangpyo Kim, Jacwan Choi, Jaiyoung Park, Donghwan Kim, and Jung Ho Ahn. Sharp: A short-word hierarchical accelerator for robust and practical fully homomorphic encryption. In Proceedings of the 50th Annual International Symposium on Computer Architecture, 2023.
Brian Knott, Shobha Venkataraman, Awni Hannun, Shubho Sengupta, Mark Ibrahim, and Laurens van der Maaten. Crypten: Secure multi-party computation meets machine learning. Advances in Neural Information Processing Systems, 2021.
Alex Krizhevsky, Vinod Nair, and GeolTrey Hinton. Cifar-10 (canadian institute for advanced research). URL http://www. cs. toronto.
cifar. html, 2010.
Souvik Kundu, Shumlin Lu, Yuke Zhang, Jacqueline Liu, and Peter A Beerel. Learning to linearize deep neural networks for secure and efficient private inference. In The Eleventh International Conference on Learning Represenlations, 2023.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7, 2015.
Eunsang Lee, Joon-Woo Woo Lee, Junghyun Lee, Young-Sik Kim, Yongjune Kim, Jong-Seon No, and Woosuk Choi. Tow-complexity deep convolutional neural networks on fully homomorphic encryption using multiplexed parallel convolutions. In International Conference on Machine Learning, 2022.
Jachoon Leo, Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl-Dickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve as linear models under gradient descent. A Aduances in neural information processing systems, 32, 2019.
21

---

Published in Transactions on Machine Learning Research (06/2024)
Dacheng Li, Hongyi Wang, Rulin Shao, Han Guo, Eric Xing, and Hao Zhang. MPCFORMER: FAST, PER- FORMANT AND PRIVATE TRANSFORMER INFERENCE WITH MPC. In The Fleventh International Conference on Learning Representations, 2023.
Gordon HY Li, Ryoto Sekine, Rajveer Nehra, Robert M Gray, Luis Ledezma, Qiushi Guo, and Alireza Marandi. All-optical ultrafast relu function for energy-efficient nanophotonic deep learning. Nanophotonics, 2022.
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. Advances in newral information processing systems, 2018.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. arv preprint, 2018.
Jian Liu, Mika Juuti, Yao Lu, and N Asokan. Oblivious neural network predictions via minionn transformations. In Proceedings of the ACM SIGSAC Conference Computer and Communications Securily, 2017.
Tianyi Liu, Xiang Xie, and Yupeng Zhang. Zkenn: Zero knowledge proofs for convolutional neural network predictions and accuracy. In Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Securily, pp. 2968-2988) 2021.
Zhuang Liu, Hanzi Mao, Chao-Yuna Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, 2022.
Nya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. ar preprint 2016.
Qian Lou, Song Bian, and Lei Jiang. Autoprivacy: Automated layer-wise parameter selection for secure neural network inference. In Advances in Neural Information Processing Systems, pp. 8638 8647, 2020a.
Qian Lou, Wen-jie Lui, Cheng Hong, and Lei Jiang. Falcon: fast spectral inference on encrypted data. Advances in Neural Information Processing Systems, 2020b.
Qian Lou, Yilin Shen, Hongxia Jin, and Lei Jiang. SAFENet: Asecure, accurate and fast neu-ral network inference. International Conference on Learning Representations, 2021.
Seyed Iman Mirzadeh, Arslan Chaudhry, Dong Yin, Huiyi Hu, Razvan Pascanu, Dilan Gorur, and Mehrdad Farajtabar. Wide neural networks forget less catastrophically. In International Conference on Machine Learning, 2022.
Pratyush Mishra, Ryan Lehmkuhl, Akshayaram Srinivasan, Wenting Zheng, and Raluca Ada Popa. Delphi: A cryptographic inference service for neural networks. In 29th USENIX Securily Symposium, 2020.
Jianqiao Mo, Jayanth Gopinath, and Brandon Reagen. Haac: A hardwaro-software co-design to accelerate garbled circuits. In Proceedings of the 50th Annual International Symposium on Computer Architecture, 2023.
Payman Mohassel and Peter Rindal. Aby3: A mixed protocol framework for machine learning. In Proceedings of the ACM SIGSAC Conference on Computer and Communications Security, 2018.
Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Nya Sutskever. Deep double descent: Where bigger models and more data hurt. Journal of Statistical Mechanics: Theory and Experiment, 2021.
Lucien KL Ng and Sherman SM Chow. Sok: Cryptographic neural-network computation. In 2023 IEEE Symposium on Security and Privacy (SP), 2023.
Thao Nguyen, Maithra Raghu, and Simon Kornblith. Do wide and deep networks learn the same things? uncovering how neural network representations vary with width and depth. In International Conference on Learning Representations, 2021.
22

---

Published in Transactions on Machine Learning Research (06/2024)
Arpita Patra, Thomas Schneider, Ajith Suresh, and Hossein Yalame. Aby2.0: Improved mixed-protocol secure two-party computation. In 30th USENIX Security Symposium, 2021.
Hongwu Peng, Shaoyi Huang, Tong Zhou, Yukui Luo, Chenghong Wang, Zigeng Wang, Jiahui Zhao, Xi Xie, Ang Li, Tony Geng, et al. Autorep: Automatic relu replacement for fast private network inference. Tn Proceedings of the IEEE/CVE International Conference Computer Vision (ICCV), 2023.
Adiwena Putra, Joo- Young Kim, et al. Morphling: A throughpat-maxiinin tfhe-based accelerator using transform-domain reuse. In IEEE International Symposium on Highh-Perforrancce Computer Architecture (HPCA), 2024.
Mija Radosavovic, Justin Johnson, Saining Xie, Wan-Yen Lo, and Piotr Dollár. On network design spaces for visual recognition. In Proceedings of the IEEE/CVE international conference on computer vision, 2019.
Lija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár. Designing network design spaces. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.
Deevashwer Rathee, Mayank Rathee, Nishant Kumar, Nishanth Chandran, Divya Gupta, Aseem Rastogi, and Rahul Sharma. Cryptflow2: Practical 2-party secure inference. In Proceedings of the ACM SIGSAC Conference on Compuler and Communications Securily, 2020.
Nikola Samardzic, Axel Feldmann, Aleksandar Krastev, Srinivas Devadas, Ronald Dreslinski, Christopher Peikert, and Daniel Sanchez. F1: A fast. and programmable accelerator for fully homomorphic encryption. In 54th Annual IEEE/ACM International Symposium on Microorchitecture, 2021.
Nikola. Samardzic, Axel Feldmann, Aleksandar Krastev, Nathan Manohar, Nicholas Genise, Srinivas Devadas, Karim Eldefrawy, Chris Peikert, and Daniel Sanchez. Craterlake: a hardware accelerator for efficient unbounded computation on encrypted data. In ISCA, 2022.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang- Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In The IEEE Conference on Compuler Vision and Pallern Recognition (CVPR), 2018.
SEAL. Microsoft SEAL (release 4.0). https://gitabb.oom/iiirrtttttt/ March 2022. Microsoft Research, Redmond, WA.
Adi Shamir. How to share a secret. Communications of the ACM, 1979.
Gowthami Somepalli, Liam Fowl, Arpil Bansal, Ping Yeh-Chiang, Yehuda Dar, Richard Baraniuk, Micah Goldblum, and Tom Goldstein. Can neural nets learn the same model twice? investigating reproducibility and double descent from the decision boundary perspective. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, 2022.
Deepraj Soni, Negar Neda, Naifeng Zhang, Benedict Reynwar, Homer Gamil, Benjamin Heyman, Mohammed Nabeel, Ahmad Al Badawi, Yuriy Polyakov, Kellie Canida, et al. Rpu: The ring processing unit. In IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS), 2023.
Haochen Sun and Hongyang Zhang. zkdl: Efficient zero-knowledge proofs of deep learning training. arXiv preprint XrXio:2307. 16273, 2023.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pallern recognilion, 2016.
Mingxing Tan and Quoc Lc. Efficientnet: Rethinking model scaling for convolutional neural networks. In International Conference on Machine Learning, 2019.
Mingxing Tan and Quoc Le. Efficientnet22: Smaller models and faster training. In International conference on machine learning, 2021.
23

---

Published in Transactions on Machine Learning Research (06/2024)
Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoc V Le. Mnasnet: Platform-aware neural architecture search for mobile. In Proceedings of the TEEE Conference Computer Vision and Pattern Recognition, 2019.
Sijun Tan, Brian Knott, Yuan Tian, and David J Wu. Cryptgpu: Fast privacy-preserving machine learning on the gpu. In IEEE Symposium on Securily and Privacy, 2021.
Yongqin Wang, G Edward Suh, Wenjie Xiong, Benjamin Lefaudeux, Brian Knott, Murali Annavaram, and Hsien-Hsin S Lee. Characterization of mpo-based private inference for transformer-basel models. In TEEE International Symposium Performance Analysis of Systems and Software (ISPASS), 2022.
Maurice Weiler and Gabricle Cosa. General o(2))-quivariant stecrable cnns. Advances in Neural Information
Processing Systems, 2019.
Maurice Weiler, Mario Geiger, Max Welling, Wouter Boomsma, and Taco S Cohen. 3d steerable cnns: Learning rotationally equivariant features in volumetric data. Advances in Neural Information Processing Systems, 2018.
Sanghyun Woo, Shoubhik Debnath, Ronghang Hu, Xinlei Chen, Zhuang Liu, In So Kweon, and Saining Xie. Convnext v2: Co-designing and scaling convnets with masked autoencoders. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
Kai Y. Xiao, Vincent Tjeng, Nur Muhammad (Mahi) Shafiullah, and Aleksander Madry, Training for faster adversarial robustness verification via inducing reLU stability. In International Conference on Learning Representations, 2019.
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE conference computer vision and pottern recognition, 2017.
Andrew Chi-Chih Yao. How to generale and exchange secrets. In 27th Annual Symposium Foundalions of Computer Science, 1986.
Leon Yao and John Miller. Tiny imagenet classification with convolutional neural networks. CS 231N, 2015.
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? Advances neural information processing systems, 27, 2014.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprinl, 2016.
Wenxuan Zeng, Meng Li, Wenjie Xiong, Wenjie Lu, Jin Tan, Runsheng Wang, and Ru Huang. Mpevit: Searching for mpe-friendly vision transformer with heterogencous attention. In Proceedings of the IEFE/CT International Conference on Computer Vision (ICCV), 2023a.
Wenxuan Zeng, Meng Li, Haichuan Yang, Wen-jic Lu, Runsheng Wang, and Ru Huang. Copriv: Net- work/protocol co-optimization for communication-eflcient private inference. In Advances Newral Information Processing Systems, 2023b.
Yuke Zhang, Dake Chen, Souvik Kundu, Chenghao Li, and Peter A. Beerel. Sal-vit: Towards latency efficient private inference on vit using selective attention search with learnable softmax approximation. In Proceedings of the IEEF/CVE /CVF International Conference on Compuler Vision (ICCV), 2023.
Borui Zhao, Quan Cui, Renjie Song, Yiyu Qiu, and Jiajun Liang. Decoupled knowledge distillation. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, 2022.
Lingchen Zhao, Qian Wang, Cong Wang, Qi Li, Chao Shen, and Bo Feng. Veriml: Enabling integrity assurances and fair payments for machine learning as a service. IEEE Transactions on Parallel and Distributed Systems, 2021.
24

---

Published in Transactions on Machine Learning Research (06/2024)
Mengxin Zheng, Qian Lou, and Lei Jiang. Primer: Fast private transformer inference on encrypted data. arXiv preprinl arXin:203. 13679, 2023.
Itamar Zimerman, Moran Baruch, Nir Drucker, Gilad Ezov, Omri Soceanu, and Lior Wolf. Converting transformers to polynomial form for secure inference over homomorphic eneryption. Tn International Conference on Machine Learning (ICML), 2024.
25

---

Published in Transactions on Machine Learning Research (06/2024)
Appendix
Table of Contents
A Design Rationale for Hyper-Paramelee Selection in HybReNet Networks
27
ReLUs' Criticality Order in StageCh, BaseCh and HybReNet Networks
28
Adapting HybReNet Design to Criticality Order Variations
29
D Depth-Based ReLU Equalization
30
CapacityCCriicalty-Traedddl
Investigating Capacity Tradeoff in HybReNets
30
E.2 Intuitive Explanation for Sapacity-Critical Tradeoff
32
F
Fine-grained ReLU Optimization on HybReNet Networks
Detailed Analysis of ReLU-Accuracy Pareto Points
33
Extended Discussion
34
H.1
Constraints for the Simultaneously Optimizing ReLU and FLOPs Efficiency
34
H.2
Achieving ReLU and FL.Pss Efficiency in Nybt by Regulating FLOPs in Deeper Layers
35
H.3
Explanation of Accuracy Saturation in SlageCh Networks Through Deep Double Descent
35
Why RegNet and ConvNeXt Models are Selected for Our Case Study?
36
H.5
Potential of ReLU Equalization as a Unified Network Design Principle
36
I
Performance Comparison of HybReNets Classical Networks for ReLU-reuse
37
Design of Experiments and Training Procedure
40
Network Architecture of HybReNets
26

---

Published in Transactions on Machine Learning Research (06/2024)
A Design Rationale for Hyper-Parameter Selection in HybReNet Networks
Stage
Stage
Total ReLU (56)
80
Total RcLU (5)
80
Total Rel.U (96)
80
60
60
60
40
40
40
20
2,7.2
3.7.2
4,7.2
5.7.
6.7.2
7.7.2
2.6.2 3,6,2 4,6,2 5.6.2 6,6.2 7,0,2 8,6,2
2,52
(a) HRN-5x7x2x (Proposed)
(b) HRNN66x6x2x(PPPoooed)
(c) (xx-xxxox)(
Figure 14: Analyzing ReLUs' distribution in HRNs by progressively increasing the values from a=2. Once the network achieves ReLU equalization (5, 7, 2) for HRN-5x7x2x, (6, 6, 2) for HRN-6x6x2x, and (7, 5, 2) for HRN-7x5x2x- the ReLUs' distribution remains stable with increasing value.
In this section, explain our design decisions for choosing specific a, 8, and in HybReNets. We selected the smallest a within a specified range for the given pairs of (3, 7) based on two primary considerations
Firstly, when the network attains ReLU equalization, the ReLU distribution becomes stable and remains constant as a grows. This stability due the fact that altering a has the least impact on the relative distribution of stagewise ReLUs compared increasing and y (Figure 14 Specifically, increasing a results in slight decrease in the proportion of Stage ReLUs and slight increase in the remaining stages.
Secondly, when ReLU optimization (Jha et al. 2021 is employed, increasing a in HRNs does not improve ReLU efficiency. Instead, it results in an inferior ReLU-accuracy tradeoff at lower ReLU counts (Figure 15
80
80
CIFAR-100 Accuracy
CIFAR- CIFAR-100 100 Accuracy
76
76
72
5x5x3x
72
5x7x2x
6x5x3x
6x7x2x
7x5x3x
7x7x2x
68
DeepReDuce
68
DeepReDuce
SNL
SNL
64
128
32
128
#ReLUs (K)
#ReLUs (K)
(a) HRN-5x5x3x
(b) HRN-5x7x2x
100 Accuracy (%)
80
CIFAR- ITAAR-10KA 100 Accuracy (%)
80
76
76
CIFAR- R--0000
72
72
8x5x2x
7x6x2x
9x5x2x
8x6x2x
68
DecpReDuce
68
DeepReDuce
SNL
SNL
32
64
128
256
32
64
128
256
#ReLUs (K)
#ReLUs (K)
(c) HRN-7x5x2x
(d) HRR-666X2x
Figure 15: Effect of increasing in HybReNets: The ReLU efficiency of networks with higher a does not
improve; in fact, it significantly reduces at lower ReLU counts.
27

---

Published in Transactions on Machine Learning Research (06/2024)
B ReLUs' Criticality Order in StageCh, BaseCh and HybReNet Networks
Table 10: Evaluating stage-wise RcLU criticality in R8sNet18 (R18) BascCh and StageCh networks on
CIFAR-100. Criticality metries (Ca) are determined using the method from Jha et al. and StageCh networks maintain the original ResNet18 criticality order: S3 S2 implies more critical ReLUs).
(022)) Both BaseCh
S4
>
S (Higher Ca
Netwooks
Stagel
Stage2
Stage3
Stage4
*BeLLis
Aoc175)
+KD((%)
C1
#RtLLs
Acc(%)
+KD(5))
+BeLL's
Aoc(35)
+KD(5)
+Stells
Ave(?5)
+KDD(5))
CL
T118(m= 20
52:67
32.17K
61.21
63.00
5.19K
59.20
6.07
1188m==22) 2x202s
163.84K
59.10
60:19
65.54KC
55,91
06.47
4.69
32.17K
65.7
67.28
16,38K
02.22
1.67
6188mm644 2x2x2x
327.686C
02.85
63:13
0.00
131.97%
67.18
08.32
3.69
65.54K
08.75
70.29
0.34
32.77%
62.63
63.47
0.27
Rt/8(mn (28)- 2cbbbe
655.26K
62.51
64:15
6:00
262
09.25
70.36
4.34
131,6K
71.25
12:04
3.61
65.54X
63.59
64.38
0.32
RI8(m=256) 20660-2200)22b222
1310 72K
64.81
05:22
0.00
$24.20%
71.05
72.43
4.05
252, 14K
72.09
75.77
5.79
131 07K
64.79
05.27
0.30
R:88mm Sxdeedee
S1,02K
52.77
55.07
6:00
19.15K
61.95
65.67
9:50
30,96K
60.23 21
07:96
11:57
27 85.
61.34
63:43
8.21
R18 10
62:30
65.51K
65.54K
9.88
R18(u=166-4 Sa-
11.22K
50.38
0.00
$1,92K
66.10
66.83
11.74
10K
70-17
70.64
14,45
128.00K
67.48
10.02
TL18(m= 0o0dos
81. .002
30.00
51
0:00
96. 30X
09.74
67.11
11.50
1477 46K
70.03 67
72.00
14.49
221 I8K
65.22
68.
10.21
R118(m= 60-Pe7
61,90K
50.00
49.07
0.00
114.00%
08.59
67.59
13.50
200.
08
73.53
10.74
351 23K
65.95
07.58
12.48
It remains intriguing to examine whether the ReLUs' criticality order in baseline networks, such as ResNet 18, remains consistent when the network width is modified, specifically in the BaseCh, StageCh, and HRN variations. To explore this, we computed the stagewise criticality metric for ResNet18 BaseCh and StageCh networks (Table 10,, and HRN nerworks with values between 2 and 7 (Table 11 Interestingly, the criticality order of the standard ResNet18 remains preserved in BaseCh and StageCh modcls, as well as in all HRNs, HRN-2x6x22, and HRN-2x7x2x). Specifically, in
except for those with a=2 (HRN-2x5x3x, HRNs with a=2, the criticality order of Stage2 and Stage3 is shuffled, while the most and least critical stages remain unchanged (i.c., S3 > S2 > S4 > S1). To account for this altered criticality order, we recomputed o, 3, and y using Algorithm [ resulting in two HRNss: HRN-2x6x3x and HRN-2x9x2x. However, the criticality order these two HRNs did not adapt to the altered criticality order (highlighted in green in Table 11
Table 11: Evaluating stage-wise ReLU criticality in ResNet based HRN networks with a values from 2 to 7 on CIFAR-100. Criticality metrics (Cx) for each stage are determined using the method in Tha el al. 2002 b. Except for a=2, all HRN networks maintain the original ResNet18 criticality order (Sg > S2 > S. > S )- HRNs with the minimum a, B, and y required for full ReLU equalization are highlighted in gray. The HRRNs highlighted in green are designed for a different criticality order:Sz > S S2 S1.
Networks
Stagel
Stape
Stage3
Stagel
#RoLUs
Arc1%)
+KDI(%)
C2
*RRLUs
Arct%))
+KDD(%)
C2
*HooUU
Are(%)
(KD2
C.
#RoLLa
Anc(%))
+KDD1%)
C2
HRN-247-22
S1.52K
62:34
53.39
0.000
32.77K
61.03
61.59
0.42
57.34K
05:44
69.82
12:38
28.67K
022115
03.40
7.01
HIN--677725
S1.92K
51.61
53.29
11.00
49.155K
64.95
65.25
9.31
86.02K
63.85
70.77
12.881
43.01K
63.10
66.17
6.36
HRN--17722s
.9K
61.28
49.42
0.00
65.54k
65.93
66.47
12.72
114.00K
70.94
72.16
16.32
57.34K
63.70
64.77
11.50 11.56
HRN--17722
NT 12.92K
05.17
67.69
14:13
71:00
72:18
16.83
71.68K
64.10
65.35
12:60
HRN-6-7-22
19K
61.23
48.65
0.00
08.30K
60.58
68.06
14:20
172.03K
71.96
72.73
16.91
86,02K
65.15
65.75
12:64
HRN-767-22
$1.92K
60.11
52.40
0.00
11 114.69KK
66.92
68.29
13.40
200. 70K
71.00
73.16
14.32
100.35K
63.92
65.58
9.51
HRN-2-6:2b
99K
62.20
53.19
0.00
39,77IC
61.02
62.00
6.90
49.15K
07.55
69.51
12.43
24 33.588
01.64
63.25
8.04
HRN-346625
0.00
49.15K
78.73K
70.96
33.44
36.86K
62.77
64.09
HRN-1-6-22
92K
53.23
53.52
0.00
65.54K
65.74
66.05
9:45
98.30K
70.47
71.54
13:22
49.15K
63.59
64.82
8:70
HRN-5-6-22
$1.5 52K
60.79
51.64
8.00
61.92KC
66.80
67.27
13.48
122. 95K
70.33
71.50
14.18
61.44K
63.97
64.54
9.97
81 92K
30.01
36.59
0.00
96 36.30K
06.57
67.04
12:58
867
71.18
72:50
15.51
75 25.73K
64.15
65.30
10 08.8
HRN-776622
0.00
72.54
16.18
64.54
HRN-9-5-22
99K
62.03
53.05
0.00
32.77K
61.00
61.76
6.80
40 16.
00.64
11.75
20.48K
61:02
02.58
7.71
HRN-3-5-22
$1.5 125
63.45
52.61
8.00
10.15KC
64.07
65.71
9.97
61.4KK
68:40
69.03
12.58
30.72K
62.52
63.42
8.51
HRN 4x5x2br
92K
52.65
52.35
0.00
65.51K
65.60
66.89
10.96
81, BL.92K
60.51
70.85
13.61
40.96KC
63.14
63.94
8.96
HRN--555<2
02.00K
70.15
14.55
51.30K
04.67
10. 00:06
IRRN 6x
92K
19.06
52.10
8.00
96.30K
66.56
68.08
11.50
122.88K
71.33
71.85
14.10
61.4KK
63.59
04.89
9.50
TINN 7astede
81 192K
51.58
51.05
0.00
114.89
68.99
67.69
11:45
143,36K
70.79
72.87
14.20
71.65K
64.02
65:25
9.58
HRN 2x5x0x
92K
$2.96
50.65
0.00
32,77K
61.39
61.90
5.97
40. 00.95K
66.78
68.17
11.17
30.72K
62.01
63.85
7.99
HRN-3=65=x
52.60
15K
61.4KK
65.57
12:93
46.08K
03.96
04.92
8.74
HRRN Axdocit
92K
51.57
50.62
8.00
65.54K
05.86
66.05
11.52
81.92K
60.12
70.13
14:33
61.4KK
63.64
05.58
11.21
HIEN Sentrcte
$1 11.92K
30:22
52:41
0.00
S1 921
66.42
67.55
12:12
102. 102
70.15
70.07
13.012
76.50K
64.21
65.59
9.73
HUNN Gubscii
12:45
122.88K
70068
71.29
14.28
92.16K
64.57
05.87
11:22
HRNN 7edocct
$1.92K
50.31
0.00
114.69K
66.85
67.95
12.06
143.36K
71.20
71.87
15.23
107.52K
64.72
65.58
11.01
92K
$1.96
50.22
0.00
30.77K
81.15
61.65
6.60
73.75K
60.00 46
70.25
12 23.33
36 80K
62.5 53
64.25
5.57
HRRN-2-0--3
62.75
39.77KC
40.15K
76
36.
64.59
28

---

Published in Transactions on Machine Learning Research (06/2024)
C Adapting HybReNet Design to Criticality Order Variations
We conducted an exhaustive characterization of HRN networks designed for the prevalent criticality order: Stage3 Stage2 > Stage4 Stagel. However, we observed that the criticality order of Stage2 and Stage4 can change in some instances, such when using HRNs with a=2 or when applying ResNet18/ResNee T'iyylmagNNet (Jha et al. 2021). In these cases, the criticality order shifts to Stage3 > Staged > Stage2 > Stagel. This raises the question of whether running the criticality test for every baseline network on different datasets is necessary.
on
To address this, we compared the ReLU-accuracy performance of HRV networks designed with two different. criticality orders. Using the DeepReShape algorithm (Algorithm we designed HybReNets for the alternative criticality order of Stage3 > Stage4 > Stage? > Stagel.
# #ReLUs(S.) > #ReLUs(SA)) > ># #ReLUs(S2)) #ReLUs(S11)
a
03
04
d2
4 >
16
64
ReLU oqualization through width (0) 02 03 = 04 = 2, and a 2 2,3 2 2, 2):
a3
a
-
a
aB >
16, a 4, aB7 64, 4, By 16, and 7
16
64
Solving the above compound inequalities provides the following range of B and y at two different y y 3 > 8 a 4; and at y 3, B 5
Solving these inequalities provides the following ranges for 3 and y at different 7 values:
At 7 = 2: 3 > 8 and a
At 7 = 3: B > 5 and 4
HRNs with the minimum values of a, B, and y that satisfy the ReLU equalization for the altered criticality order (Stage3 > Stage4 > Stage? > Stagel) are HRN-5x6x3x and HRN-5x9x2x. For lower ReLU counts, we select. HRNs with a=2, (i.e., HRN-2x6x3x and HRN-2x9x2x). We compare the ReI.U-accuracy tradeoffs of these HRNs with those designed for the prevalent criticality order using both coarse-grained ReLU optimization (DeepReDuce) and fine-grained ReLU optimization (SNL).
The results (Figure 16 show that the performance of HRNs for both criticality orders is similar with coarse-grained optimization on CIFAR-100. However, with fine-grained optimization, there is a noticeable accuracy gap. Specifically, HRN-2xxx3x and HRN-2x7x2x outperform HRN-2x6x3x and HRN-2x9x2x by small but discernible margin. On TinylmageNet, HRN-5x6x33 and HRN-5x9x2x perform similarly, except that HRN-5x5x3x outperforms at some intermediate ReLU counts.
67
66
ggonges
Thyy Net Accuracy
64
372
62
HRN-2x6xck
HRN-2:613x
60
HRN-5x6ic3x
HRN-2x9<2x
2x922
58
HRNNsx
600
HRN-2x5x3x
66
HRN-2x5x3x
56
HRN-5x5x3x
56
HRN-2
61
128
66
16
HRN-2x7X2x
61
128
54
128
156
HRN-5x7x2x
512
#ReLUs (K)
#ReL.Us (K)
#Rel.Iis (K)
(a) Dece CIFAR-100
(b) SNL CIFAR-100
(c) DeepHeDuce on TinylmageNet
Figure 16:
Performance comparison of HRNs designed for the altered criticality order (Stage3 > Stage4 >
Stage2 > Stagel) with 3-6 & 9, and HRNs designed for the prevalent criticality order (Stage3 > Stage2
Staged > Stagel) with 3=5 & 7. Overall, the latter exhibit slightly better performance than the former.
29

---

Published in Transactions on Machine Learning Research (06/2024)
D Depth-Based ReLU Equalization
ReLU equalization through width in HybReNets has two effects: increasing the network's complexity per unit of nonlincarity (measured as parameters and FLOPs per unit of RcLU) and aligning the ReLU distribution according to their criticality order. To analyze these effects independently, we applied ReLU equalization through depth and augmented the base channel counts to increase parameters and FLOPs per unit ReLU.
82
82
CIFAR-100 100 Accuracy (%)
80
80
CIFAR 100 Accuracy
78-
78
HRN(1.7.5.2]
[..51.7..52]
76
HRN[1.6.6.2]
(6666,6,622
74-
HRN[1.5.7.2]
74
HRN[1.5.7.2]
HRN[1,5,5.3]
HR5,1,5,5,3]
72
ResNet188Stageee
72
ResNet 8(StageCh)
(Bssec))((Bseeh))
ResNar18(Basce)
70
70,2
256
512
1024
2048
64
128
256 512 1024 2048 4096 8192 16384 #FLOPs (M)
#ReLUs (K)
(a) ReL.U-efficiency comparison
(b) FLOPP=-fffiieeccy comparison
Figure 17: Efficacy of depth-based ReLU equalization, performed using depth typer-parametert i.c., stage- compute-ratios (modified values shown in brackets [Ø1, Ø2, Φ3. ø1l). These depth-based HRNs show similar or worse ReLU and FLOOPs efficiency compared to BaseCh networks, highlighting the effectiveness of width adjustments through a, 3, and 7 for ReLU equalization in width-based HybReNets.
We use ResNet18 with m=16 and fixed a=ß=y=2 while sctting the stage-computte-attis (Ø1, d2, Φ3, h4) design hyperparameters. Using Algorithm for ReLU equalization, solved compound inequalities to determine the depth hyperparameters. We determine depth hyperparameters (ø1, d2, Φ3, 64) E (11,5,5,3); (1,5,7,2); (1,6,6,2); (1,7,5,2) corresponding to the minimum values enabling ReLU equalization, resulting in network global depth parameters and FLOPs per unit of ReLU in BaseCh networks.
of 14. We then varied m E {16, 32, 64, 128, 256} to increase
The experimental results, shown Figure 17 compare the ReLU and FLOPs efficiency with BaseCh and StageCh networks. ReLU and FLOPs efficiency of the derived networks were either similar or worse than the BaseCh networks. For example, H,, exhibits inferior ReLU (FL.OPs) efficiency at higher ReLU (FLOPs) counts compared with BaseCh networks. This underscores the significance of ReLU equalization through width adjustment by altering a, 3, and 7, and demonstrates that ReLU equalization alone does not yield the desired benefits in HybReNets.
E Cappaity-Citticlltyy-raaad Tradeof
E.1
Investigating
Tradeoff in HybReNets
We conducted additional experiments on various HybReNets to further investigate the Critiityy Tradeoff phenomenon observed in Figure 6 We progressively reduced the values in all the HRNs, increasing the proportion of the network's ReLUs in Stagel (see Table I1] For example, HRN-6x6x2x, HRN-4x6x2x, and HRN-2x6x2x have Stagel ReLU fractions of 20.4%. 27.8%, and 43.5%. respectively.
We employed both the coarse-grained (DeepReDuce) and fine-grained (SNL) ReLU optimization methods on all the HRNs variants, and the results are shown in Figure [8] Consistent with trends in Figure D the wider versions of all four HRNs outperform at higher ReLU counts, while HRNs with a=2, having higher proportion of Stagel ReLUs, excel lower ReLU counts. For instance, HRN-6x6x2x and HRN-4x6x2x outperform HRN-2x6x2x at higher ReL.U counts, whereas HRN-2x6x2x excels at lower ReLU counts.
30

---

Published in Transactions on Machine Learning Research (06/2024)
80
CIFAR 100 Accuracy (5)
80
CJFAR 100 Accuracy
76
76
72
72
68
68
HRN--2x53x
HRN-2x7x2x
64
HRN-4x5x3x
64
HRN-4x7x2x
60
HRN-5x5x3x
HRN-5x7x2x
60
8
16
32
64
128
16
32
64
128
#ReLUs (K)
#ReLUs (K)
(a) DeepReDuce on HRN-5x5x3x
(b) DeepReDuce on HRN-5x7x2x
Accuracy de
880
Accuracy (96
800
76
76
72
72
100
100
38
68
HRN-2x6x2x
HRN-2x5x2x
64
HRN-4x6x2x
64
HRN-4x5x2x
HRN-6x6x2x
HRN-7x5x2x
360
060
8
16
32
64
128
8
16
32
64
128
#ReLUs (K)
#ReLUs (K)
(c) DecpReDuce HRN-6x6x2x
(d) DecpReDuce IIRN-7x6X2x
Accuracy (%))
Accuracy
76
76
72
72
CIFAR- 100
CIFAR 100
HRN-2x5x3x
HRN-2x7x2x
68
68
HRN-4x5x3x
HRN-4x7X2x
HRN-5x5x3x
HRN-5x7x2x
64
64
16
32
64
128
16
32
64
128
#ReLUS (K)
#ReL.Us (K)
(e) SNL on HRN-6x5x3x
(f) SNL on HRN-6x7x2x
CIFAR-100 100 Accuracy (%)
76
CIFAR: 100 Aecuracy
76
72
72
HRN-2x6X2x
HRN-2x5x2x
68
68
HRN-4x6x2x
HRN-4x5x2x
HRN-6R6X2x
HRN-7x5x2x
64
64
16
32
64
128
16
32
64
128
#ReLUs (K)
#ReLUs (K)
(g) SNL HRN-6x6x2x
(h) SNL HRN-7x5x2x
Figure 18:
Tradeoff in HRN networks for feeee/inegrriinee ReLU optimization DeepRe-
Duce/SNL. HRN networks with lower a posses higher proportion of Stagel (the least-critical) network's ReLUs, and exhibit superior performance at lower ReLU counts.
31

---

Published in Transactions on Machine Learning Research (06/2024)
E.2 Intuitive Explanation for Capacity-Criticality Tradeoff
In classical CNNs such as ResNet18 and WRN22x8, networks with higher proportion of least-critical (Stagel) ReLUs tend to have lower overall ReLU counts. For example, WRN22x8 and ResNet18, used in SNL (Cho et al. 2022b) and SENct (Kundu et al. 2023), have total RcLU counts of 1392.6K and 557K, respectively, with 58.8% and 48.2% of the network's s ReLUs in Stagel. This trend also consistent in HybReNets: HRN-6x662x, HRN-4x6x2x, and HRNN2x662x have ReLU counts of 401.4K, 294.9K, and 188.4K, with 43.5%, 27.8 27.8%, and 20.4% of ReLUs in Stagel, respectively. This raises the question: what drives better performance at lower ReLU counts the proportion of Stagel ReLUs the network's total ReLU count?
To investigate this, we compared the performance of networks with the opposite trend: networks with higher ReLU counts and a higher proportion of Stagel ReLUs. Specifically, we selected ResNet34 and ReLU efficient variant of ResNet18, ResNet18(m = 14x-xx4x4x having a uniform ReLU distribution and used in prior Pl-specific network optimization methods Ghodsi et al. (0200) Cho et al. (022a).. The results in Table 12 show that despite having 3.5x fewer RcLUs, compared ResNet34, Res 16)-4x4x4x starts performing worse below 50K ReLU counts. This due to lower percentage (29.41%) of Stagel ReLUs in
ResNet18(m 16)-4x4x4x compared to ResNet34 (47.46%)
Propertion of Total ReLUs
60
58.8%
+1.6%
Network
#ReLUs Stagel(%)
180K
100K
50K
15K
50
48.2%
51.2%
ResNet31
966.7K
47.46
76.35%
74.33%
72.07%
66.46%
4x4xx4
278.5K
20.41
77.09%
75.03%
71.38%
64.77%
40
25.7%
30
Table 12: Performance comparison (using SNL ReLU optimization) between ResNet34 and a ResNet18 vari- ant (60N))11((111664x45
20
Non-critical ReLUs
10
Critical ReLUs
having a uniform distri- bution of ReLUs, Stagel(%) indicates the fraction of the network's ReLUs in Stagel. Despite having 3.5x fewer ReLUs, the performance of ResNet18(m= 4x4x4x remains inferior to ResNet34 when the ReLU count falls below 50K. This suggests that the key de- terminant for superior performance at very low ReLU counts the fraction of least-critical ReL.Us, rather than the network's total ReLU count.
WRN-22x8
ResNet18 ResNet34 4x4x4x
Network
Figure 19: Networks with a higher fraction of least- critical (i.e., Stagel) ReLUs, such as ResNet18 and ResNet34, drop smaller fraction of their critical Re- LUs. In contrast, WRN22x8 and 4x4x4x drop larger fraction of their critical ReLUs to achieve a ReLU count of 25K. This occurs regardless of their initial ReLU counts in baseline network.
To understand why higher fraction of Stagel ReLUs leads to better performance at lower ReLU counts, we examined the ReLU dropping strategies used in prior ReLU optimization techniques (Kundu et al. 2023 Cho et al. b022b Tha et al., 2021b. They consistently showed that Stagel ReLUs are the least critical and are dropped first. to achieve very low ReLU counts. Consequently, networks with higher proportion of least-critical ReLUs drop smaller fraction of their critical RcLUs when aiming for very low ReLU counts. This is illustrated in Figure 19 where WRN22x8 and ResNet18(m = 16)-4x4x4x drop higher fraction of their critical ReLUs compared to ResNet18 and ResNet34 to reach a ReLU count of 25K. Dropping a higher fraction of critical ReLUs leads to significant accuracy loss and inferior performance. This explains why prior work (Kundu al. 2023 Cho et al. 2022b used ResNet18 as backbone network for ReLU optimization at lower ReLU counts and WRN22x8 al higher ReLU counts.
This observation aligns with findings from Yosinski et al. (2014), which showed that neurons in the middle layers of a network (Stage2 and Stage3) exhibit fragile co-adaptation, making them difficult to relearn. Therefore, dropping more ReLUs from these stages in networks with fewer least critical (Stagel) ReL.Us disrupts the fragile co-adaptation and significantly reduces performance.
F Fine-grained ReLU Optimization on HybReNet Networks
We employed SNL ReLU optimization (Cho al. 2022b) on HybRoNets to examine the efficacy of fino-grained ReLU optimization on these networks. The results, presented in Figure 20 show that HRNs with SNL ReLU optimization inferior to the (vanilla) SNL, the ReI.U-Accuracy Pareto points reported in Cho et al.
32

---

Published in Transactions on Machine Learning Research (06/2024)
2022bb (which used WRN22x8 (for #ReLUs >> 100K) and ResNet18 (for #ReLUs & 100K) as backbone networks). To further investigale, we first performed ReI.U-Thimming, where ReLUs from alternate layers are removed, and then employed SNL ReLU optimization. Employing SNL on ReLUU-Thinne HRNs results in an accuracy boost of up to 3% (on CIFAR-100) across all ReLU counts, achieving performance comparable to vanilla. SNL. This underscores the significance of ReLU Thinning even for fine-grained ReLU optimization.
These findings are consistent with observations made in Figure 7 and Table 5 demonstrating the limitations of SNL when the distribution of ReLUs is changed, the proportion of Stagel ReLUs decreases and that in other stages increases. This implies that the benefits of fine-grained ReLU optimization are contingent upon the higher proportion of least-critical ReLUs in the network.
78
78
CIFAR-100 100 Aecuracy
CIFAR-100 100 Accuracy
75
75
72
72
69
HRN(wl Th.
CIFARR-
69
HRN((wo Th.
HRN(W/ Th.
HRNWW TTL
66
66
SNL(vanilla)
SNL(vanilla)
16
32
64
128
256
16
32
64
128
256
#ReLUU (K)
#ReLUs (K)
(a) HRN-5x5xdx
(b) HRN-5x7x2x
CIFAR-101 100 Accuracy (%)
78
78
CIFAR-100 Accuracy
75
75
72
72
69
HRN(w ThISSNL
69
HRN(W ThISSNL
HRN Th HSNL
HRN Th TNNL
66
66
SNL(vanilla)
SNL(vanilla)
16
32
64
128
256
16
32
64
128
256
#ReLUs (K)
#ReLUs (K)
(c)
HRN-6x6x2x
(d) HRN-7x5x2x
Figure 20: HybReNets with fine-grained ReLU optimization: HRNs using SNL (fineggraine)) ReLU opti- mization exhibit inferior ReLU-Accuracy tradeoff compared to the vanilla SNL, employed on ResNet18 and WRN22x8 (Cho et al.) 2022b). Interestingly, using ReLU-Thinning (a coarse-grained optimization step used in DeepReDuce) before SNL optimization yields performance on par with the vanilla SNT. This highlights the significance of ReLU Thinning in ReLU optimization, even the context of fino-grained ReLU optimization.
To better understand the effectiveness of ReLU Thinning in the context of fine-grained ReLU optimization, analyzed the total mumber of ReLUs within network before and after applying the optimization. For instance, in the HRN-5x7x2x model, the SNL algorithm identified 50K essential ReLUs from an initial pool of 363.5K ReLUs, subsequently eliminating 312.5K ReLUs. However, when we ReLU-Thinmig is applied prior to SNL. optimization, the SNL algorithm finds 50K eritical ReLUs from a reduced pool of 181.25K ReLUs and dropped 131.25K ReLUs. This because Thinning eliminates half of the network's ReLUs from alternating layers, regardless of their criticality. Thus, RcLU Thinning effectively reduces the scarch space required to identify critical ReLUs, resulting in an accuracy boost of 2.44% on CIFAR-100.
G Detailed Analysis of ReLU-Accuracy Pareto Points
In this section, we describe the ReLU optimization steps used to construct ReLU-Accuracy Pareto frontier shown in Figure 11 (a,b) and Table 7 When employing ReI.U-reuse, fixed m=16; however, the network's
33

---

Published in Transactions on Machine Learning Research (06/2024)
FLOPs count reduces due to the group-wise convolution. For instance, HRN-2x6x3x with ReLU-reuse factor of four has 370.1M FLOPs, while the baseline HRN-2x6x3x (m=16) network has 527.4M FLOPs, when used on CIFAR-100 (see Table 14I. However, due to specific implementation constraints of group convolution in Microsoft SEAL (SEAL), we calculated the HE latency without considering the FLOP reduction from group convolution Rel.U-reuse.
Table 13: Network configurations and ReLU optimization steps employed for the HybReNet points in Table Accuracies (TinylmageNet) are separately shown for vanilla KD (Hinton et al.) 2015 and DKD (Zhao el al. 2022), highlighting the benefits of improved architectural design and distillation method. Re2 denoles ReLU- KeLU-rruse, reuse, and used for efficient PI very low ReLU counts.
ReLU optimization steps
Accuracy(%
HybReNet
#ReLU
#FLOPs
Acc./ReI.U
Culled
Thinned
Re2
KD
DKD
5x5x3x
16
NA
S1+
NA
653.3K
4216.4M
65.76
67.58
0.10
2x5x3x
32
S1
S2+S3+S4
NA
417.8K
2841.7M
64.11
66.10
0.16
5x5x3x
NA
S3
326.6K
1055.4M
61.92
64.92
0.20
2x5x3x
8
S1
S2+S3+S4
NA
104.4K
178.9M
56.37
58.90
0,56
2x5x3x
16
S1
$2 S3 S4
52.2K
486.3M
53.13
54.46
1.04
Table 14: Network configurations and ReLU optimization steps employed for the Pareto points in Figure 11 (a.b), along with and the HybReNet points used for the comparison with SOTA PI methods and ConvNeXi-V2 [Woo et al. 2023) as illustrated in Table 14 Re2 denotes ReLU-reuse, key method in achieving significantly reduced ReLU counts.
m ReLU optimization steps
Accuracy (%
Nets
#ReLU
#FLOPs
Acc./ReLU
Culled
Thinned
Re2
KD
DKD
4x6x3x
$3
NA
317.4K
2061.1M
80.14
81.36
0.26
CIFAR 100
2x6x3x
16
S1
S2-S3+S4
NA
134.1K
527.4M
78.80
79.56
0.59
HybReNet
2x6xxxx
8
S1
S2-S3
NA
67.1K
132.2M
74.84
76.91
1.15
2x6x3x
16
S2-S3+S4
33.5K
370.1M
70.93
73.89
2.21
2x6x3x
16
S1
S2+S3+S4
8
16.6K
394.8M
68.17
69.70
4.19
2x6x3x
$24$3+$1
8.3K
413.4M
62.44
64.65
7.78
4x6x3x
16
NA
S1+S2+S3+S4
NA
1269.8K
8244.2M
68.90
70.29
0.06
4x6x3x
12
NA
S14824S34S4
NA
952K
4638.8M
68.16
69.15
0.07
HybReNet
2x6x3x
16
S1
S2
NA 536.6K
2109.4M
66.29
67.48
0.13
2x6x3x
12
S1
S2-S3+S4
NA 402K
1187.5M 5M
64.51
63.77
0.16
ThyIIaggNNe
2x6x3x
S2+$3+S4
NA
268.3K
528.6M
60.97
64.02
0.24
2x6x3x
16
S1
S24$3+84
4
134.1K
1480. 1M
57.84
61.52
0.46
2x6x3x
S1
S2-$3+S4
8
67.1K
1579.2M
54.47
56.24
0.84
2x6x3x
16
S1
S2
16
33.5K
1653.5M
49.13
49.96
1.49
T
96
S1
S2-S3+S4
NA
1622K
11801M
68.32
69.85
0.04
ConvNeXt
S3
NA 1278K
9080.2M 2M
66.73
68.75
0.05
P
64
S1
S2-S3+84
NA 720.9K
3435.7 7M
65.42
67.08
0.09
F
48
S1
S2-S3+S4
NA 540.7K
1935M
64.23
65.72
0.12
A
40
S2-$3+S1
NA
450.6K
13
63.23
64.08
0.14
H Extended Discussion
H.1 Constraints for the Simultaneously Optimizing ReLU and FLOPs Efficiency
Classical CNNs often follow the convention of doubling the filter count when downsampling feature maps
by factor of two prevent representational bottlenecks (Szegedy et al. 2016). This results in a fixed stagewise channel multiplication factor of = for most. classical networks. Even in the design
34

---

Published in Transactions on Machine Learning Research (06/2024)
of state-of-theeart FLOPs-efficient vision models, such as RegNet (Radosavovic al. 2020 the stagewise channel multiplication factors are restricted to 1.5 S (a, 8) s 3. Conventionally, network design paradigms prioritize FLOPss efficiency, so the stagewise multiplication factor is typically conservative since the FLOPs count has a quadratic dependence on the channel counts.
To optimize both ReI.U and FLOPs efficiency under PT constraints, higher a and & values and a lower y value are required (see Figure B(e.)). Restricting a, 3, and y in classical FLOPs efficient networks (e.g., ResNet and WideResNets) primarily harms the network's ReLU efficiency. Also, the prior approach of designing Uefficifi networks by increasing a, B, and r homogeneously (e.g., a B 7 4 in Ghodsi et al. Cho et al. (2022a)) results in poor FLOPs efficiency due to higher 7.
2020)
H.2
Achieving ReLU and FLOPs Efficiency in HybReNets by Regulating FLOPs in Deeper Layers
ReLU equalization in a four-stage network imposes specific constraints on 3 and y values, with By < 16 and Y < 4. These constraints effectively limit the explosion of FLOPs in deeper layers, making HRN networks more FLOPs efficient than their StageCh counterparts, which use homogeneous sets of a, 3, and y. This homogeneity in StageCh networks leads to rapid increase in FLOPs due to the multiplicative effect of stagewise multiplication factors. Hence, even slight increase in these factors significantly increase FLOPs count, especially in deeper layers.
To illustrate this, we computed the normalized FLOPs in ResNet18-basce StageCh networks and compared a²8²
and them with HRNs. The normalized FLOPs in Stage3 and Stage4 of ResNet18 are expressed as 16
x²8822 respecivllly. Thus, a network with 7=2 would have equal FLOPs in Stage3 and Stage4. This is evident from the normalized FLOPs ratios in HRN-557x2x,, HRN-7x5x2x, and HRN-6x6x2x (Table 15)
In particular, the coustraints on 7 (i.e., 7 < 4) keep the ReLU count of Stage4 lower than that of Stage3 (the most critical stage), and restrict the FLOPSs in Stage4. While further limiting a and / values can reduce the network's FLOPs, this would also reduce the proportion of the most critical (Stage3) ReLUs. Therefore, unlike StageCh networks, criticality-aware design strcamlines both the ReLU and FLOPs in networks by preventing superfluous FLOPs in deeper layers and mazimizes the FLOPs ulilization for given ReLU cound.
Table 15: Stagewise FLOPs and ReLU trend variations with o, 3, and y in (ReLJ-efficieet) StageCh and HRNs. The least (most) critical ReLUs are colored in red (blue). Evidently, ReLU equalization in HRNs restricts the growth of FLOPs in deeper layers, achieving ReLU efficiency comparable to StageCh networks but with fewer FLOPs. This enables the right balance between ReLU and FLOPs efficiency in HRNs.
Stage1
Stape?
Stage3
Stage4
(
2c(o,8,7)<4
(.,88))
(0,8.))>4
FLOPs
a?
Layerwise FLOPs
cunsstant
incressing (t)
increusing (tt) incrrasing (ttt)
ReLUs
od
Layerwise ReLUs
decreasing (LJ)
decreasing (4)
constant
increasing (t)
2,
(.,
(o,
[4, 4)
(a.
6,
Stagel
Stage2
Stage3
Stagel
Stagel
Stage2
Stage8
Stagel
Stagel
Slage?
StageS
Stagel 4006
Skagel 64
Stage? 576
Stage8 3184
Staged
FLOPs
64
64
64
64
64
144
324
729
64
256
1024
40056 216
ReLUS
16
48
64
144
HRN-5xx7222
HRRN-THGS 7x5x2x
HRN-6xeix22
HIRN-566x33
Stagel 64
Stage2 80
Stagel 4900
Stage4 70
Stagel 64
Stage2
Stage3
Stage4 1900
Stagel 61
Stage2 96
Stagel 5181
Stagel 5181
Stagel 64
Stage2 100
Stage3
Stage4 75
FLOP= ReLUs
781
2500
140
112
140
70
64
144
72
80
100
H.3 Explanation of Accuracy Saturation in StageCh Networks Through Deep Double Descent
A distinct (rend in the ReLU-Accuracy tradeoff has been observed (see Figure 3 (a)b)) as the width of models increases by augmenting a, B, and y. Specifically, accuracy initially increases with increasing values of a, 3, and 7, reaches saturation point, then increases again at higher ReLU counts. This trend is more prominent in models with smaller depth, such as ResNet18, and disappears in deeper models such as ResNet56, where performance does not improve once accuracy saturated.
35

---

Published in Transactions on Machine Learning Research (06/2024)
The observed saturation trend in the ReLU-Accuracy tradeoff can be explained by the "model-wise deep double descent." phenomenon, which becomes more pronounced with higher label noise (Belkin et al. 2019: Nakkiran et al. 2021 Somepalli et al. 2022). In particular, with label noise, a U-shaped curve appears in the regime due to a bias-variance tradcoff. Whereas, in the over-parameterized regime, accuracy improves due to the regularization enabled by the strong inductive bias of the network. However, with zero label noise, the test error platcans around the interpolation threshold, resulting a flat. trend similar to the one shown in Figure a,b) instead of a U-shaped curve.
classical hunder-paramcteriinn
H.4 Why RegNet and ConvNeXt Models are Selected for Our Case Study?
RegNets are models designed using a semi-automated network design method, parameterized by the stage compute ratio (01, Ø2, Ø3, Φ1), base channel count (m), and stagewise channel multiplication factors within the range 1.5 I (a, 7) S 3. In contrast, ConvNeXts are redesigned ResNets with modified values of m and Φ1, Φ2, 03, d4. For example, the ConvNNett-T model has a stage compute ratio of [3, 3, 9, 3], different from the [3, 4, 6, 3] ratio used in ResNet34, and a base channel count. of m=96, higher than the m-64 used in ResNet34. These unconstrained design choices in RegNets and the modified depth and width in ConyNeXt models make them suitable for our case study, where we investigate their impact on RcLU distribution and the balance between ReLU and FLOPs efficiency.
The original ConvNeXt model uses the Pl-unfriendly GELU activation function Hendrycks & Gimpel (2016 and LayerNorm Ba et al. (2016), which require several approxinations in ciphertext evaluation (Fan et al. 2022 Zimerman et al.) 2024). Therefore, we adapted ResNet models to incorporate ConvNeXt features, specifically the stagewise channel allocation and the stage compute ratio from ConyNeXt models. For experiments with RegNets, we opted for the RegNet-x variant instead of RegNet-y, as the latter employs a sigmoid activation function in its aaneezo-and-excitttinn blocks.
H.5 Potential of ReLU Equalization as a Unified Network Design Principle
Deep learning has advanced significantly in recent years with the development of sophisticated neural network architectures. Traditionally, researchers have relied mamual network design techniques, such ResNet (He et al. 2016), ResNeXt (Xie et al. 2017), and ConvNeXt (Liu et al.
2022 J. Neural architecture scarch (NAS)
methods (Liu et al. 2018 Tan & Le 2019 Howard et al. 2019 Tan et al. 2019 have also shown promise.
However, these approaches have their own limitations. Manual techniques can become more complex and lead to suboptimal models as the number of design hyperparameters increases. NAS methods often lack interpretability and may not generalize beyond restricted settings. Moreover, both methods require substantial computational resources to find optimal design hyperparameters when designing networks from scratch.
A semi-automated design technique like RegNets (Radosavovie et al. 2020 offers an interpretable network design and aulomates the process of finding the optimal population of nelworks, generalizable across wide range of settings. However, it still requires training thousands of models to iteratively narrow down the search space, making it expensive when models are designed from scratch.
We analyzed the ReLUs' distribution in SOTA semi-automaate models (RegNets) and manually designed models (ConvNeXt) and made the following observations:
The specific values of width and depth hyperparameters, determined by a quantized linear function in RegNet models, position the networks' ReLUs in their criticality order (see Figure (a)
Similar to HRNs with a=2, in ConvNeXt models, the distribution of ReLUs, except for Stagel, follows their criticality order.
In particular, the proportion of the most critical (Stage3) ReLUs increases from 20.3% in ResNet34 to 30.2% in ConvNeXi-T and 34.8% in ConvNeXI-N model while reducing the proportion of less critical ReLUs (see Figure b)). The stagewise channel allocation in ConvNeXt models and HRNs (with a=2) shows that both models allocate an identical number of channels in the deeper stages. HRNs allocate fewer channels during the initial stages. These observations demonstrate the generality of ReLU equalization a design principle, even when designing FLOPSs efficient models.
36

---

Published in Transactions on Machine Learning Research (06/2024)
Stem
Stage 2 booceoo Stage
Stage 4
Stem
Stage 2
bosoboo
Stage
Stage
Stage
Stage 3
100
100
Total ReLU (%)
80
80
60
Total ReLU
40
40
20
20
0.2 0.. 0.8 3.6.2 4.0 12 16 32
R34 C-T C-A C-F C-P C-N HRN
(a) RegNet-X
(b) ResNet34 and ConvNeXt-V2
Figure 21: Analysis of RsLUss distribution in RegNet-X and ConvNeXt-V2 architectures, In all RegNet-X models, ReLUs are arranged preciscly according to their criticality order. ConvNeXt models, including their T, A, F, P, and variations (Woo et al. 2023), have a higher proportion of ReLUs in the most critical stage (Stage3) and fewer in the less critical stage (Stagel) due modified depth and width design hyper- papee-pprameters. In contrast, the distribution of ReLUs in the HRN-4x6x3x model (designed from the ResNet34 network) strictly adheres to their criticality order.
ConvNext V2-T m=96 [96, 192, 384, 768
HRN-2x6x2x m=32 [32, 64, 384, 768]
ConvNext V2-N m==00 [80, 160, 320, 640
HRN-2xxx2x m=32 [32, 64, 320, 640
ConvNext V2-P m==4 [64, 128, 256, 512
HRN-2x5x3x m=16 [16, 32, 160, 480
ConvNext V2-F m=48 [48, 96, 192, 384]
HRN-2x6X2x m=16 [16, 32, 192, 384
ConvNext V2-A m==00 [40, 80, 160, 320]
HRRN2x5x2x m=16 [16, 32, 160, 320]
ReLU equalization, the other hand, requires only prior knowledge of the stagewise criticality of baseline network, which is often consistent within specific. model family and requires training very few models. Moreover, ReLU equalization can be used to design both FLOPs and ReLU-efficien neural networks. Thus, ReLU cqualization offers a new perspective to simplify network design, improve interpretability, and achieve both FLOPs and ReLU efficiency.
Performance Comparison of HybReNets vs. Classical Networks for ReLU-reuse
We compared ReL U-reuse against the conventional scaling method (channel (chanell/eeature-mag scaling) used in DeepReduce (Jha et al. 2021) on both classical networks and HRNs. First, we applied ReLU-reuse to all convolutional layers the networks, reducing the ReLUs by a factor of N E {2, 4, 8, 16). For N ===2 2, we used the naive ReLU reduction method, as it outperformed the proposed ReLU-reuse (see Figure
Experimental results show that for ResNet18 BaseCh networks, ReLU-reuse performs inferior to conventional scaling methods, and this performance gap increases for networks with higher In contrast, for HRN networks, ReLU-reuse surpasses conventional scaling at higher ReLU reduction factors (i.e., at very low ReLU counts). However, at higher ReLU counts, particularly for reduction factor of two, the information loss incurred from feature maps division outweighs the benefits of ReLUer, leading to inferior performance. This observation holds even for networks with uniform ReLUs distribution, such as ResNet18(m==1))AAeee 4x4x44.
Furthermore, we repeat the exporiments on ReLU-Thinned networks, as conventional scaling performed on Thinned networks in Jha et al. (2021). We dropped ReLUs from alternate convolutional layers and applied ReLU-reuse in the remaining layers. The results are shown in Figure 23 Since ReLU-reuse is now only employed in half of the total number of layers, the cumulative information loss caused by the loss of cross-channel information in meature-map divisions is reduced. Consequently, the performance of R-LU-r is further improved. This improvement is noticeable from the change in the performance gap between ReLU-reuse and conventional scaling for all the networks, as shown in Figure 23
To summarize, the effectiveness of ReLU-reuse depends on the network architecture and the ReLU reduction factor. ReI.U-reuse is effective for networks with (partial/full) ReLU equalization, in contrast to classical
37

---

Published in Transactions on Machine Learning Research (06/2024)
70
ReLU-reuse
70
ReLU-reuse
Accuracy (%
65
Scaling
Accuracy (%)
Scaling
65
60
60
55
50
55
8
16
32
64
16
32
64
128
#ReLUs (K)
#ReLUs (K)
(a) (m))et18(m==16)
(b) ResNet18(m=32)
80
R-reuee
80
ReLUer
Accuracy (%
75
Scaling
Accuracy (%
Scaling
70
70
65
65
60
60
32
64
128
256
64
128
256
512
#ReLUs (K)
#ReLUs (K)
(c) RosNet18(m=61)
(d) RosNot18(m=125)
75.
ReLU-reuse
ReLUereuse
70
Scaling
Scaling
Accuracy (%)
70
Accuracy (%)
65
65
60
60
55
55
50
16
32
128
32
64
128
#ReLUs (K)
#ReLUs (K)
(o) ResNet18((=110)Axx4ak
(f) HR-N-xxx22x
80
75-
ReLU-reuse
ReLU-reuse
75
Scaling
Scaling
Accuracy (%)
70
Accuracy (%
70
65
60
60
55
55
16
32
64
128
16
32
64
128
256
#ReLUs (K)
#ReLUs (K)
(g) HRRN-xx5x3e
(h) HRN-6x6x2x
Figure 22: Performance comparison (CIFAR-100) of ReLU-reuse vs conventional scaling (used in coarse- grained ReLU optimization Tha et al. 2021 when ReLU-reuse is employed after every convolution layer.
38

---

Published in Transactions on Machine Learning Research (06/2024)
70
65
Accuracy (%
Accuracy (%
60
55
60
50
ReLU-reuse
ReLU-reuse
Scaling
Scaling
45
55
16
32
8
16
32
64
#ReLUs (K)
#ReLUs (K)
(a) RosNct18(m=16)
(b) ResNet18(m==2)
75
Accuracy (6))
75
70
Accurayy (%
70
65
ReLU-reuse
65
ReLU-reuse
60
Scaling
Scaling
16
32
64
128
60
32
64
128
256
#ReLUs (K)
#ReLUs (K)
(c) ResNet18(m==6))
(d) ResNet11
75
70
Accuracy (%)
Accuracy (%)
65
65
60
60
55
ReLU-reuse
55
ReLU-reuse
50
Scaling
Scaling
50
8
16
32
64
8
16
32
#ReLUs (K)
#ReLUs (K)
(o) ResNet18(m ( 16)--xxxxx
(f) 1x6x22xxx2x
75
75
Accuracy (%)
70
Accuracy (%)
70
65
65
60
ReLU-reuse
ReLU-reuse
55
Scaling
55
Scaling
16
32 #ReLUs (K)
32
64
#ReLUs (K)
(g) HRN-5x5x3x
(h) HRN-6x6x2x
Figure 23: Performance comparison (CIFAR-100) of ReLU-reuse conventional scaling (used in coarse- grained ReLU optimization (Jha et al. 2021 when ReLU-reuse is employed ReLU-Thinned networks. That is, first ReLU is dropped from every-alternate layers, and then ReLU-reuse is applied in remaining layers (see Algorithm
39

---

Published in Transactions on Machine Learning Research (06/2024)
networks, and scales well with higher ReLU reduction factors. Therefore, ReLU-Thinmed HRN networks combined with Rel.U-reuse significantly improve performance at very low ReLU counts, which we incorporate in the ReLU optimization pipeline designed for HybReNets (shown in Algorithm 2
Algorithm ReLU optimization steps employed in (HRRs)
Input: A network Net with D stages S1, S2, ... Sn and C, a sorted list. of stages from least to most critical Output: ReLU optimized versions of Net
if the least critical stage C[L] dominates the distribution of ReLUs then
2: Sk = C[]
D Get the least critical stage D Cull the least critical Stage Ss.
3: Net = Net- Sy.
4: end if
5: Net," Thin(Net)
b Thin the remaining stages Channel scaled by 0.5x
Net, ScaleCh (Netf. a=0.5)
NetR1 ReuseReLU (Nel,, Sc=4)
ReLU-reuse with scaling factor 4
Net its ReuseReLU (Net, Sc=8)
> ReLU-reuse with scaling factor 8
9: Net, RIG = ReuseReLU (Net, Sc=16)
> ReLU-reuse with scaling factor 16
10: Nels += Nel, Nell Net, Net,a, Net Ne
b Apply KD to each Net.
11: return Nets
J Design of Experiments and Training Procedure
Sweeping the width hyperparameters for ReLU efficiency experiments: To examine the effect of various network widening methods (primarily BaseCh and StageCh), we conducted experiments with ResNet18 as backbone architecture in Figure 3 (a.b). We reduced the base channel count in ResNet18 to m=16 (from m-64) to enable a fair comparison with other ResNet models, such ResNet20, ResNet32, and ResNet56, which have m=16. These ResNet models have [16, 32, 64] #channels in their successive stages, while that in (original) ResNet18 is [64, 128, 256, 512]. For BaseCh networks, we sweep m (16,32,,4,128822)) and for StageCh networks, we sweep (a, B, 7) = (2, 2, 2) to (8, 8, 8), homogeneously.
Training methodology and datasets: We performed our experiments on CIFAR-100 (Krizhevsky et al., Yao & Miller 2015), as prior PI-specific network optimization
2010) and TinyImageNet (Le & Vang. 2015
studics (Jha ct al. 2021 Cho et al. 2022b; Kundu et al. 2023) used these datasets. CIFAR-100 consists of 100 classes, each with 500 training images and 100 lest images of resolution 32x32. TinyImageNel includes
200 classes, each with 500 training images and 50 validation images with a resolution of 64x64.
For training on CIFAR-100 and Tiny-mageNet, we used a cosine annealing learning rate scheduler (Loshchilov & Hutter 2016) with an initial learning rate of 0.1, a mini-batch size of 128, a momentum of 0.9, and a weight decay factor of 5e-4. We trained the networks for 200 epochs on both datasets, with an additional 20
warmup epochs for Decoupled KD (Zhao et al. 2022
For DeepReDuce and KD experiments in Tables G Table 13 and Table 14 we employed Hinton's 's knowledge
distillation (inton et al. 2015) with a temperature of 4 and relative weight to cross-entropy loss as 0.9.
For SNL, we train the baseline networks using the aforementioned methodology; however, we used their default implemennttton for mask ien,ration, fine-tuning, and knowledge distillation. When employing Decoupled knowlodge distillation (Tables 6 13 and 14], we set the relative weight of target class KD as one and vary the weight of non-target class KD as {0.8, 1, 2, 6). For the KD experiment, we consistently employed ResNet18 as the teacher model for fair comparison across the studies.
We adopted the methodology from Garimella et al. 2023 to compute the runtime of a single (private) inference. Specifically, we used Mierosoft SEAL to compute the homomorphic encryption (HE) latency for convolution and fully connected operations, and DELPHI MMishra et al. 2020) to compute the garbled-circuit (GC) latency for ReLU operations. Our experimental sctup involved an AMD EPYC 7502 server with 2.5 GHz, 32 cores, and 256 GB RAM. The client and server were simulated as two separate processes operating the same machine. We set the number of threads to four to compute the GC latency.
Runtime measurement:
40

---

Published in Transactions on Machine Learning Research (06/2024)
Note that for HRNs with ReLU-reuse, calculated the HE latency without accounting for the FLOP reduction from group convolution in ReL.U-reuse due to the implementation constraints in Microsoft SEAL.
K
Network Architecture of HybReNets
Table 16 shows the architectural comparison between ResNet11, WRN22x8, and four specific HRNs: HRN- 5x5x3x, HRN-5x7x2x, HRN-6x6x2x, and HRN-7x5x2x. Compared to ResNet18, the HRNs allocate fewer channels in the initial stages and increase those in the deeper stages of the network. This strikes the right balance between RsLUs and FLOPs efficiency. Among the four HRNs, HRN-5x5x3x offers a slightly better balance between ReLU and FLOPs efficiency.
Table 16: Comparison of WideResNee22x88 and ResNet18 architectures, which are commonly used as baseline networks in Pl-specific ReLU optimization techniques (Kundu et al. 2023 Cho et al. 2022b Jha et al. 2021 with our proposed HybReNets (highlighted in bold). Unlike conventional WidcResNNts and ResNets, the strategic channel allocation in the subsequent stages of HybReNets streamlines the network's ReLUs and FLOPs, optimizing both ReLU and FLOPs efficiency simultaneously. The last rows compare their FLOPs
and ReLU counts, along with baseline accuracy CIFAR-100.
Stages
output
WRN22x8
ResNec18
HRN-5x6x3x
HRN-5x7x2x
HRN-6x6x2x
HRN-7x6x2x
Stem
32 x 32
[xx3, 16)
[3x3, 64]
[xx3, 16]
3x33, 16
[3x3, 16]
[3x3, 16
3x3, 128
3x3, 64
3x3, 16
3x3, 16
3x3. 16
3x3, 16
Stagge
32x32
3x3, 128
x3
3x3, 64
x2
3x3, 16
x2
3x3, 16
x2
3x3, 16
x2
3x3, 16
x2
3x3, 256
3x3, 128
3x3, 80
3x3, 80
3x3, 96
3x3, 112
Stage2
16x 16
3x3,
x3
3x3,
3x3, 80
x2
3x3,
x2
3x3,
3x3,
3x3, 512
3x3, 256
3x3, 400
3x3, 560
3x3, 576
3x3, 560
Stage3
3xx3, 512
x3
3x3, 256
3x3, 400
3x33, 560
3x3; 576
3x3, 560
Stage4
4xc4
3x3,
3x3, 512
x<2
3x3,
3x3, 1200
x2
3x3, 1120
x2
3x3, 1152
1152
x2
3x3,, 1120
11:20
x2
FC
1x1 1
[8 8, 512)
x 4, 512]
[1 1200]
[ 4, 1120]
[1 4, 1152]
[ 4, 1120]
#FLOPs
2461M
539M
1006M
1273M
1368M
1328M
#RoLUs
1393K
557K
343K
379K
401K
412K
Accuracy
81.27%
79.01%
78.40%
78.28%
78.52%
78.81%
Table 17 details the architectural specifics of HRNs with a=2, which are crucial for performing PI at lower ReLU counts. These HRNss developed using ResNet18 as backbone architecture, leading to stage compute ratios of [2, 2, 2, 2]. Similarly, HRNs designed using ResNet34 as backbone architecture have stage compute ratios [3, 4, 6, 3). same as ResNet34 (For instance, HRN-4x6x3x Figure 21 b)).
41

---

Published in Transactions on Machine Learning Research (06/2024)
Table 17: Baseline HybReNets, developed using Res.et118 as the backbone architecture, aim for efficient PI with lower ReLU counts. The final rows compare their FI.OPs, ReLU counts, and baseline accuracy on the CLEAR-100 dataset.
Stages
output size
HRN-2x5x3x
HRN-2x7x2x
HIRN-266X2X
HRN-2x5X2X
Stem
32 x 32
3x3, 16)
3x3, 16)
3x3,, 16
[33,, 16)
3x3, 16
3x3, 16
3x3, 16
3x3, 16
Stagel
32x32
x2
x2
x2
x2
3x3, 16
3x3, 16
3x3, 16
3x3, 16
3x3, 32
3x3, 32
3x3, 32
3x3, 32
Stage2
16x 16
x2
x2
x2
x2
3x3, 32
3x3, 32
3x3, 32
3x3, 32
3x3, 160
3x3, 224
3x3, 192
3x3, 160
Stage3
8x8
x2
x2
x2
x2
3x3, 160
3x3, 224
3x3, 192
3x3, 160
3x3, 480
3x3, 448
3x3, 384
3x3, 320
Stage4
4x4
x2
x2
x2
x2
3x3, 480
3x3, 448
3x3, 384
3x3, 320
FC
X 1
[8 8, 480]
4, 448]
[4 4, 384]
[4 X 4, 320
#FLOPs
179M
213M
163M
119M
#ReLUs
186K
201K
188K
176K
Accuracy
75.34%
75.73%
75.70%
75.03%
42


Reviews:
[{"review_id": "LomhiCqJfB", "reviewer": "Reviewer_CFqf", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths:\n\n- The paper provides a comprehensive experimental study on state-of-the-art efficient PI methods and pinpoints the existing fallacies. Researchers working in this area can greatly benefit from the benchmarking of this paper.\n\n- The proposed method is a straightforward implementation of the observations made by the paper.\n\n- Experimental results demonstrate that the proposed method significantly enhances the existing efficient PI methods.\n\n### Weaknesses:\n\n- Given that this reviewer does not work in the immediate research area of the paper, I found the paper very hard to follow and read. The major issues contributing to this are:\n\n   1. Since the beginning lines, the paper takes the knowledge about efficient PI as assumed knowledge. For instance, the paper never discusses why the ReLU count is important for efficient PI. As the paper builds upon the importance of lowering the ReLU counts, I firmly believe that it should provide a reasonable amount of discussion around why the number of ReLUs are critical for efficient PI.\n\n   2. Moreover, given the length of the paper, the structure of the writing is hard to follow. Unfortunately, instead of motivating each section properly, the paper just uses bold paragraph headings to alleviate this issue, which in my opinion does not adequately satisfy the importance of a proper motivation. Take the introduction for example. I believe that instead of having bold headings for each paragraph, it would be much easier to ensure that each paragraph follows the previous one and reads coherently. Unfortunately, this is not the case for this submission.\n\n  3. On a minor note, I strongly recommend to the authors to use the natbib package correctly for their citations. Citations that are not part of the sentence should always be used with a parenthesis using \\citep. For example, instead of writing: \"_CryptoNAS Ghodsi et al. (2020) and Sphynx Cho et al. (2022a) employ neural architecture search to design ReLU-efficient baseline networks and disregard FLOP implications._\" the sentence should be written as \"_CryptoNAS [Ghodsi et al., 2020] and Sphynx [Cho et al., 2022a] employ neural architecture search to design ReLU-efficient baseline networks and disregard FLOP implications._\" This can greatly enhance the readability of the paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper provides a comprehensive investigation of efficient private inference (PI) in the context of convolutional neural networks (CNN). To this end, the paper first delves into existing approaches proposed for efficient PI and argues that ReLU optimization-based methods neglect the importance of FLOP efficiency in their proposed algorithms. Among the observations made, the authors demonstrate that the distribution of ReLUs in different stages of a CNN can have a considerable impact on FLOP efficiency. Based on this key observation, the paper proposes a new method called \"ReLU Equalization\" to ensure that the distribution of ReLUs in different stages of the CNN follow their criticality order. Additionally, a new \"ReLU-reuse\" algorithm is proposed to further enhance the PI efficiency at low rate ReLUs. The experimental results on CIFAR-100 and TinyImageNet datasets demonstrate that the proposed CNNs, coined HybReNets, improve the ReLU and FLOP efficiency of PI models while maintaining a comparable accuracy to existing methods.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "As mentioned above, I believe that despite the contributions of the paper, the writing should be greatly improved. To this end, I believe that the authors should incorporate the feedback in the weaknesses section above. In terms of the importance of the contributions, I will wait to read the review of my peers before making up my mind.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "CB3NXmVGTJ", "reviewer": "Reviewer_7qPP", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. The ReLU equalization design is novel.\n\n2. The evaluation is comprehensive, and demonstrates the empirical effectiveness of the approach.\n\nWeaknesses:\n\nThe main weakness of this paper is that the paper writing is generally confusing. In particular, Section 3 includes a lot of experimental results, but sometimes the connection to the key design principle (i.e., ReLU equalization) is unclear, e.g., regarding the discussion of different input networks and fine-grained VS coarse-grained ReLU optimization.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes DeepReShape, which optimizes both ReLUs and FLOPs for private inference (PI). The main novel technique is ReLU equalization, where they scale the channels to align the ReLU counts in the criticality order. Based on this design principle, they design a family of networks called HybReNet. They compare HybReNet to SOTA models in PI and FLOP-efficient models, and show that HybReNet achieves more substantial FLOP reduction with fewer ReLUs, while achieving additional accuracy gain.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See the weaknesses above.", "broader_impact_concerns": "No concern.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "BM0Sq5DXF2", "reviewer": "Reviewer_T1uq", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n(1)    Extensive Experiments： This paper is filled with rich experiments and observations based on these experiments.\n\n(2)    Important Topic: The paper focuses on an important research area, efficient private inference.\n\n(3)    Well-motivated. This paper offers a comprehensive introduction and analysis of existing channel scaling and ReLU optimization methods.\n\n**Weaknesses**\n\n(1) The originality of this paper is constrained, with its primary contributions centered around heterogeneous channel scaling and hybrid ReLU optimization. However, it is important to note that heterogeneous channel scaling essentially involves adapting homogeneous channel scaling by introducing variations in alpha, beta, and gamma. The hybrid ReLU optimization is also just a combination of existing methods such as ReLU-Thinning and SNL optimization. \n\n(2) Additional experiments are required to strengthen the findings of this paper. While the current work utilizes DELPHI (2020) as the foundational 2PC private inference framework, it's noteworthy that Cheetah (2022) is generally acknowledged for its enhanced efficiency in both linear and nonlinear computations. Cheetah demonstrates a speed advantage of 12 to 30 times faster on end-to-end evaluations compared to DELPHI. Although the primary focus of this paper is on network design, it would be beneficial to demonstrate the compatibility of the proposed techniques with frameworks like Cheetah. \n\n(3) Algorithm 1, responsible for ReLU Equalization and the formation of HybReNet, exhibits limitations in delivering a holistic solution. Its primary output is constraints generated by allocating more ReLUs to critical stages. However, the current approach leaves the user to manually address these constraints, introducing a reliance on trial and error to optimize network design. This aspect raises concerns about the algorithm's user-friendliness and efficiency in providing an automated and robust solution for network optimization.\n\n(4) The authors claim that their proposed algorithms, specifically ReLU equalization and ReLU optimization, are asserted to have a constant time complexity of O(1), regardless of the number of stages N. However, to substantiate this claim convincingly, more in-depth analysis or additional data is needed.\n\nMoreover, the authors' assertion that the proposed approach requires no additional network training lacks clarity. This uncertainty arises because the changes in hyperparameters governing the network width imply a modification to the network structure. Consequently, it seems necessary to fine-tune this altered structure, casting doubt on the claim that additional training is not needed.\n\n\n(5)  Regarding the second technique, 'ReLU-reuse,' how it alleviates accuracy loss is not explicitly clarified, as ReLU continues to operate solely on the last group (X/N). Moreover, the paper lacks a clear explanation of why feature reuse holds significance within this particular context.\n\n(6)  Typos: On page 5,  ReLU efficiency (see 3(c)--> 3(e)). FLOP efficiency (see 3(d)-->3(f)).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces DeepReShape, an innovative network redesign technique that customizes architectures to meet PI constraints while optimizing for both ReLUs and FLOPs simultaneously. The key insight involves strategically allocating channels to align the network's ReLUs in their criticality order, thereby enhancing both ReLU and FLOP efficiency. DeepReShape streamlines network development through an efficient process, generating networks termed HybReNets. Evaluation on standard PI benchmarks showcases a 2.1% accuracy gain with a 5.2× runtime improvement at iso-ReLU on CIFAR-100 and an 8.7× runtime improvement at iso-accuracy on TinyImageNet.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See above weaknesses.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: ### Summary of the paper

This paper introduces DeepReShape, a method designed to optimize neural networks for more efficient private inference (PI) by addressing ReLUs and FLOPs, which are crucial for performance. Contrary to previous work that primarily focused on minimizing ReLUs due to their perceived impact on PI latency, this study reveals that FLOPs contribute significantly to latency. They propose an approach called DeepReShape, which suggests a strategic distribution of network channels to optimize the positioning of ReLUs based on their criticality, achieving improvements in ReLU and FLOP efficiency. The networks created using DeepReShape, or HybReNets, have enhanced performance in standard PI benchmarks. Specifically, on the CIFAR-100 dataset, HybReNets achieved a 2.1% accuracy increase and a 5.2 times faster runtime at comparable ReLU counts. On the TinyImageNet dataset, they demonstrated an 8.7 times faster runtime at similar accuracy levels. This research also examines the role of network selection in prior ReLU optimization efforts and highlights the critical attributes of networks for superior PI performance.

### Summary of the discussions
This paper was initially very poorly written, as also noted by reviewers CFqf and T1uq. However, the authors did a very good job during the rebuttal period and managed to improve the writing of the paper significantly. As it stands, I think this paper reads well. The authors also, in my opinion, managed to address the other concerns related to experimental methodologies and comparisons very well, and they made a significant update to the paper. However, the Reviewer T1uq is still concerned about the CoPriv method at NeurIPS’23, which uses OT for nonlinear operations and has different (network) optimization goals. CoPriv's proposed method is designed to align with studies predominantly using Garbled Circuits; therefore, it did not adopt the Cheetah framework," which is still confusing and concerning. Although GC and OT are different techniques, they both work for private inference (the goal of the related works is the same in protecting the user's privacy). Thus, it would be useful to compare both methods. I would recommend the authors to include and add some comparisons to the CoPriv method if they can.
Decision: Accept (poster)