PrivShap: A Finer-granularity Network Linearization Method for Private Inference
Authors: Xiangrui Xu
Zhenzhen Wang
Rui Ning
Chunsheng Xin
Hongyi Wu
Venue: Accepted by TMLR ()

Abstract: Private inference applies cryptographic techniques like homomorphic encryption, garble circuit and secret sharing to keep both sides privacy in a client-server setting during inference. It is often hindered by the high communication overheads, especially at non-linear activation layers such as ReLU. Hence ReLU pruning has been widely recognized as an efficient way to accelerate private inference. Existing approaches to ReLU pruning typically rely on coarse hypothesis, which assume an inverse correlation between the importance of ReLU and linear layers or shallow activation layers have less importance for universal models, to assign the budgets according to the layer while preserving the inference accuracy. However, these assumptions are based on limited empirical evidence and can fail to generalize to diverse model architectures. In this work, we introduce a finer-granularity ReLU budget assignment approach by assessing the layer-wise importance of ReLU with the Shapley value.

To address the computational burden of exact Shapley value calculation, we propose a tree-trimming algorithm for fast estimation. We provide both theoretical guarantees and empirical validation of our method.  Our extensive experiments show that we achieve better efficiency and accuracy than the state-of-the-art across diverse model architectures, activation functions, and datasets. Specifically, we only need $\sim$$2.5\times$ fewer ReLU operations to achieve a similar inference accuracy and gains up to $\sim$$8.13\%$ increase on inference accuracy with similar ReLU budgets.

Published in Transactions on Machine Learning Research (08/2025)
PrivShap: A Finer-granularity Network Method for Private Inference
Linearization
Xiangrui Xu Deportment of Computer Science Old Dominion University
amu0020odu. edu
Zhenzhen Wang
zwang2180kjhu. edu
Deportment Biomedical Engineering Johns Hopkins University
Rui Ning
rring @odu edu
Department of Computer Science Old Dománion University
Chunsheng Xin Department Computer Science lowa State Universily
cain Giastate.....mmm edu
Hongyi Wu
mhunuarizonaa edu
Department of Electrical and Computer Engineering University of Arizona
Reviewed OpenReview: https:// opeervveeeeerrvviee.. net/ ttt/formm? id= 7Tld 2mJr2m
Abstract
Private inference applies cryptographic techniques like homomorphic encryption, garble cir- cuit and secret sharing to keep both sides privacy in a client-server setting during inference. It is often hindered by the high commuicatton overhcads, especially at non-lincar activa- lion layers such as ReLU. Hence ReLU pruning has been widely recognized as an efficient way accelerate private inference. Existing approaches to ReLU pruning typically rely on coarse hypothesis, which assume inverse correlation between the importance of ReLU and linear layers shallow activation layers have less importance for universal models, to assign the budgets according to the layer while preserving the inference accuracy. How- ever, these assumptions are based on limited empirical evidence and can fail to generalize to diverse model architectures. In this work, we introduce a finer- mner-gaamllrity ReLU bud- gel assignment approach by assessing the layer-wise importance of ReLU with the Shapley value. To address the computational burden of exact Shapley value calculation, we propose tree-trimming algorithm for fast estimation. We provide both theoretical guarantees and empirical validation of our method. Our extensive experiments show that achieve better efficiency and accuracy than the state-of-the-art across diverse model architectures, activa- tion functions, and datasets. Specifically, we only need 225x fewer ReLU operations to achieve a similar inference accuracy and gains up to ~8.13% increase on inference accuracy with similar ReLU budgets.
1 Introduction
The growth of AI-driven client-server technologics Xing & Zhao (224a c) Wang et al. (0024): Zhu et al. (2025): Xu al. (2024a 2025); Bai al. (2025). has significantly advanced Machine Learning as Service
(MLaaS) Xu et al. (0024c b): Mishra et. al. (2020) Rathee et al. (2020) in academia. and industry. However,

---

Published in Transactions on Machine Learning Research (08/2025)
privacy concerns have led to the emergence of cryptographic techniques for private inference (PI), where the server processes encrypted client input while retaining the machine learning model, ensuring data privacy throughout the inference process.
Existing private inference frameworks often involve cryptographic techniques such as Homomorphic Encryp- tion (HE) Fan & Vercauteren (2012); Cheon et al. (2017), Garbled Circuits (GC) Bellare et al. (2012). and Secret Sharing (SS) to maintain the privacy of both the client and the server. For example, Crypt- Flow2 Rathee et al. (2020) uses HE for convolutional computations and SS for non-linearity computations,
while Gazelle Juvekar et al 2018) employs GC for ReLU operations. Compared to linear computations, the use of GC and SS in non-lincar functions like ReLU requires significantly higher communication costs and bandwidth. This increased overhead leads to longer communication latency during client-server interactions in privacy-preserving inference, making efficiency a significant concern.
This challenge has prompted a range of efforts develop more efficient privacy-preserving models by re- ducing the number of ReLU operations, thereby decreasing communication and lateney overhead in secure client-server interactions. A series of network linearization works that utilizing various methods are sum- maried in Table Notably, CryptoNAS Ghodsi et al. (202)) and Sphynx Cho et al. (2022a) use the neural architecture search (NAS) technique to find the optimum network architecture. However, such methods usu- ally lead to significant accuracy drops. To rectify such shortcomings, gradient-based and manual meth such as SNL Cho et al. (022b)) and DeepReDuce Jha et al. (2021), , are proposed to prune pixel-wise ReLU operations by adding I regularizations during training. However, these methods assume a uniform impor- tance distribution of ReLU operations across different layers, which leads to only a suboptimal balance point between accuracy and communicttin efficiency. Moreover, it requires extra-long training epochs to reach low ReLU operation budget. To find better trade-off between accuracy and efficiency, SENet [Kundu ct al. (2023) proposes the hypothesis that the importance of ReLU layers is inversely correlated with the corresponding linear layers based on the sensitivity measured by accuracy loss when pruned. Based on this hypothesis, they propose customized layerwise allocation of ReLU budgets based on inferred ReLU impor- lance, which is calculated by one minus the linear layer gradient. Arguably, this hypothesis only based on limited empirical evidence and not guaranteed to be generalizable to diverse model architectures and scttings.
Our contributions: We introduce PrivShap, a Shapley Value (SV)-based ReLU pruning method, which provides fine-grained and accurate ReLU importance estimation and assigns ReLU budgets based on this estimation. To the best of our knowledge, this is the first work to estimate the layer-wise importance of ReLU operations using Shapley values Shapley (1953), challenging the ReLU importance hypothesis presented in the SOTA Kundu et al. (2023). To alleviate the computational cost. of exact Shapley value calculation with O(2") time complexity, we propose double-trimming strategy for fast estimation. This method effectively reduces the number of coalitions by trimming the less important subsets of ReLU layer combinations that contribute marginally to the final SV results. Moreover, unlike related works that use I regularization to implicitly reduce ReLU operations, PrivShap employs a projected gradient descent pruning strategy to enforce the layer-wise ReLU operations within certain budgets. This approach enables more precise control over the optimization goal during pruning and helps maintain accuracy. We conduct extensive experiments on various models with various activation functions like SiLU SiLU/ReLU, including ResNet He et al. (2016) and Wide Residual Networks Zagoruyko (2016), on CIFAR-10, CIFAR100, and Tiny-ImageNet. Experimental results show that our SV-based ReLU budgel can yield the best accuracy-efficienc trade-off. We only need up to 2.5x fewer ReLUs to achieve similar accuracy and gain up to 1813 accuracy improvement with a similar ReLU budget compared to prior arts and can be applied to activation functions and models.
2 Preliminaries
2.1 Cryptographic primitives
We briefly describe the relevant cryptographic primitives in this section.
2

---

Published in Transactions on Machine Learning Research (08/2025)
Table 1: Comparison between existing approaches in yielding efficient models to perform PI. Only our method has fino-grained relu budget allocation for higher model performance and efficiency. PA denotes polynomial approximation on ReLU layer. NAS denotes network architecture search.
Method used
Granularity
Fine- sine-grained?
Sphyux Cho et al. (2022a)
NAS
layer
CryptoNAS Ghodsi
2020
NAS NAS +PA
layer
DELPHI Mishra et al.
20
layer
SAFENet Lou et al. 202T
NAS +PA
channel
DeepReDuce Jaa et al. (2021
Manual 4 l-normalization
layer
SNL Cho et al. (2022b
pixel, channel
SENct Kundu ct al. 2023
layer-ina lyverimpprtance based
channcl, pixel
Ours
Shapley value based
channel, pixel
Homomorphic Encryption: HE is a public key eneryption scheme that allows computations to be per- formed ciphertexts, enabling the results to be decrypted without needing to access the plaintexts. The encryption function E produces ciphertext / from plaintext message m using a public key pk, ex- pressed as & = pk). In Pl, the results of linear operations can be obtained homomorphically through mi om2 = D(t1 sk), where o represents a linear operation, is its corresponding homomorphic operation, and 4 and l2 are the ciphertexts of m, and m2, respectively.
Garbled Circuits: GC facilitate secure two-party computation of a Boolean function f without revealing Boolean circuit T.e process begins with the garbler, who generates encoded circuit C and corresponding input labels using the Garble(C) to send Ć and the labels to the other party who acts as evaluator. The evaluator evaluates the garbled circuit with the provided labels via Eval(C). Finally, the garbler decrypts the labels to get the plain results to share with the evaluator.
private inputs. The function is represented
Additive secret sharing. Given an original message m party P {0, one of the two Additive Secret Shares (ASS) is constructed by uniformly sampling randomness and setting (m) r, while the other share is formed (m) m-r. To reconstruct the message, one can simply add two shares (m) + (m))-p
2.2 Private Inference
Similar to previous works Mishra et al. 2020 Cho et al. (202bb); Kundu et al. (0223 method follows the two-party semi-honest threat model. Specifically, the client C' and the server S follow the protocol but attempt to infer each other's 's imput, namely the client's 's input data and the server's model parameters, during the inference process. To mitigate various threats, existing cryptographic- employ an online-offline topology. In this architecture, the client's 's data--ndepenndnt components are pre- computed during the oflline phase. For linear computations, such as matrix multiplication, HE commonly utilized in the offline stage, it allows operations like multiplication on ciphertexts without the need for decryption. To ensure the correctness of decryption, ciphertexts must be refreshod after limited number of operations, which can be accomplished through techniques such as boolstrapping Chillotti et al. (0016) or re-encryption Rathee et al. (2020). However, the computationally intensive nature of non-linear functions in GC results in high costs for operations like ReLU, even in the online phase. For SS-based customized privacy-preserves activation protocols, the frequent communication via Oblivious Transfer cryptographic primitives imposes prolonged inference times. This highlights a critical challenge in balancing efficiency and security in cryptographic computations.
inference frameworks often
2.3 Shapley Value
Shapley Value key concept in cooperative game thoory, which assumes a set of players collaborate achieve a total gain, and measures each player's contribution by averaging its marginal contribution across all possible coalitions. Shapley value has been widely applied to model explanation (Sundararajan & Najmi
3

---

Published in Transactions on Machine Learning Research (08/2025)
2020,, data evaluation Ghorbani & Zou (019]) and model compression Ghorbani & Zou 2020 it has
the following desirable properties:
Zero/Negative Contribution: The player with zero (or negative) contribution the model accuracy can be measured if VS C N/(i ( I U(S), which means adding the element to the subset of other layers does not change reduce the model performance metric (N denote the complete set, S denote a coalition of elements, and U denote the utility function). The proof is shown in appendix B.1 in Ghorbani & Zou 2020
The sum of the Shapley values all ReLU layers equals the value of the grand coalition, so that all the gain is distributed among the ReLU layers: ΣIEN SV SVN),, where SV(N) (N) denotes the value of grand coalition. The proof is shown in appendix B.1 and B.2 in Ghorbani & Zou 2020
Efficiency:
However, the calculation of exact SV requires exponential computational costs to enumerate all possible coalitions, and existing works use Monte Carlo sampling (MCs) Ghorbani & Zou (2019) or Multi-Armer
Bandit (MAB) Ghorbani & Zou (2020) to approximate it. However, such works still need an extra long time to converge via a large number of random samples.
2.4 ReLU linearization in Private Inference
Existing approaches to reduce ReLU operations in neural networks for private inference focus on either re- designing models or optimizing architectures. SAFENet Lou et al. (2021) achieves finer control by applying channel-wise substitutions and mixed- precissir activations, measuring activation importance via l normal- ization from preceding layers. CryptoNAS Ghodsi et al. (2020) redesigns networks using evolutionary NAS to minimize ReLU operations, while Sphyux Cho et al. (002aa) enhances this with differentiable NAS for more efficient redundant ReLU layers from standard models. Recent efforts Cho et al. (2022b) also apply I regularization to reduce ReLU operations, optimizing the trade-off between accuracy and non-linearity. However, these methods are sensitive hyperparameters, often leading to suboptinal solutions and failing to meet specific ReLU budget targets. Furthermore, those works overlook the potentially different importances of ReLU. operations across layers and only assign global ReLU budget for the entire model.
models. DeepReDuce Jha et al.
[021 takes a manual approach, pruming
SENet Kundu et al. 2023). on the other hand, designs a fine-grained, layerwise ReLU budget allocation strategy. Their method is based on a hypothesis that the importance of each activation layer is inversely correlated with the pruning sensitivity D of its corresponding layer with weight and bias parameters. They then assign the ReLU budget by measuring the pruning sensitivily. However, this hypothesis is only based on their empirical obsorvation a small experiment of training ResNet18 on CIFAR-100. It is not con- vincing that this is a generalizable rule in other datasets and model variants without further verification. Furthermore, they define the importance of ReLU layers the accuracy of retaining only one ReLU layer at time and removing all others, which overlooks the natural interactions between different ReLU layers in a model. Shapley value provides a compelling alternative to measure layerwise ReLU importance rigorously and directly.
3 Method
In this section, we introduce our PrivShap method for fine-gramularity network linearization method that includes two parts. First, Sec. 3.11 introduces our method uses SV as indicators and tree-trimming method to allocate a ReLU budget across layers. Second, Sec. 3.2 introduces gradient-projectinn pruning to remove activations at a finer granularity. To better formulate the interactions and cooperation of different ReLU layers, we treat them as cooperative players and assess their averaged marginal contribution to prescrving the inference accuracy. More formally, let N denote the set of all ReLU layers, S denote coalition of
I Pruning sensitivity denotes the accuracy reduction caused by pruming certain ratio parameters from it Ding 2019
4

---

Published in Transactions on Machine Learning Research (08/2025)
ReLU layers (i.e., subset of ReLU layers), and U denote the performance assessment function, specifically assessing the model accuracy of preserving only coalition of ReL.U layers and replacing all other ReLU The Shapley value of the i-th ReLU layer is then defined as:
layers as identity function 2
SV,
1 IN
Σ SCN(()
U(S UU(S)
(1)
( [S]
The Shapley value takes into account the interactions between the ReLU layers. As a simple example, suppose that two layers improve performance only if they are both present but harm performance if they are present individually, estimating their importance by preserving each of them a time (as Kundu et al.) (2023) did) might give misleading results. Shapley value, on the other hand, considers all possible coalitions.
3.1 Fine-grained ReLU importance Estimation
However, the exact SV mcasurement for the ReLU layers works for the model with a few layers. The computational complexity grows quickly as the model goes deeper, becoming prohibitively expensive even for ResNet-18. This is mainly due exponential number of coalitions S, and computing the marginal contribution for all of them is extremely time-consuming. Existing works have employed methods such
,1,2,4,53
acc:46.3%
acc4.4%%
acc:32.3%
2.3%%
V
1st tteraion
{1,,
81,455
4,,4
1,,
acc:l.3%
acc:l. 9%
a9c:6.1%
V accc1%
acc:5.
acc:5.8% 8% -
2nd iteration
{1,
E2,53
E1,5 N
E1,43
84,5
81
.... iteeatiinn
Figure The process of calculating the estimated shapley value of 3rd ReLU layer in a given example ReLU layer model, the dashed branches are trimmed as the remaining accuracy below pre-defined threshold (< 5%). Each iteration will generate one layer of child subsets with removing one element.
MCs and MAB algorithms to approximate the SV of data points or neurons once specific convergence criteria are met. Although MCs and MAB algorithms can provide approximate solutions, they do not eliminate the inherent computational burden associated with effectively estimating SV.
To avoid an exponential subset search space, the MCs and MAB first sample subsets of ReLU layers based on the given model architecture. After the candidate model architecture is determined, the model is trained via hundreds of epochs to converge on the given datasots. Then the layer-wise SVs are updated with the new model accuracy data point. Such a sampling process is repeated until the calculated SVs are converged within given small threshold, shown in Fig. 2 left. Therefore, the sampling trials and heavy training process within each trials are inevitably leading
unacceptably long training process.
To reduce heavy computational cost, we propose a tree-trimmin tree-trimming method to estimate the SV of cach ReLU layer, as shown in Fig. 2 right. Our method is based on an observation that all the ReLU layers have positive contributions to the total accuracy and there are few 'free-riders' layers. An intuitive explanation for this is that even if ReLU operations at some certain pixels might be harmful to the model accuracy Li al. 2022
2 All the layers with weight and biases preserved
5

---

Published in Transactions on Machine Learning Research (08/2025)
~200 epochs
train candidate model
inference trials
random sample ReLU layers
model accuracy
trimmed ReLU subsets
model accuracy
limited trials
SV update
end when converge
SV upddtte
Figure 2: The SV estimation process via Monte-Cane Carlo sampling (left). The process involves unbounded sampling trials and heavy training for each candidate model, while PrivShap (right) only requires inference with limited trials with bounded error (see in Fig. and Sec. 3.1.
an entire layer (i.e., a set of pixel-wise ReLU operations), can be less likely to have a negative contribution. This is also aligned with our empirical observation: the accuracy can plummet to a very low number even if only removing few ReLU layers from a trained model. This also makes such low accuracy model irreversible to high accuracy model by just adding one important activation layer. We start with model trained in standard, non-ReLU pruning manner. After the model is trained, calculate the (approximated) SV for its i-th ReLU layer by building a tree, where each node denotes inference. The root node represents all ReLU layers excluding the i-th one (i.e., N \ {i)), and each node represents coalition of ReLU layers containing one fewer ReLU layer than its father node. At each node with the corresponding coalition S, we calculate the marginal contribution of the i-th layer to inference accuracy, denoted as (iSU(i))- Importantly, we trim all the child nodes of S (S) is below a certain threshold, as the marginal contributions in these coalitions (i.e,, child nodes of S) are even lower, resulting a negligible perturbation of the SV. Figure [ illustrates this trimming process, where the dashed nodes denote those trimmed. Since cach RcLU layer makes similar contribution and those layers are minority in mainstream models, the valid nodes are bounded within very few iterations. Thus, we gain time efficiency by eliminating the heavy training and unbounded sampling trials, which makes our method suited for solving network lincarization issuc. The detailed runtime comparison is shown in Sec. 4.3
Preliminary Theoretical Analysis To demonstrate the accuracy of our approximation, we provide a preliminary theoretical analysis. We aim to show that given a coalition of ReLU layers S where its utility USS) is below a certain pruning threshold a, pruning S and all its subsets will only have a negligible perturbation to the overall SV. We start with a natural observation that 4S' S and USS) S a, then the marginal gain satisfies USS Ui) UUS) < 3, where < a. The perturbation caused by pruning S and all its subsets can be formulated as:
UIS U(S"
ASV
Σ
N
(1STT))
N (NI-1]
S'Cs
||S
(2)
[9]
B
Σ
(5
Σ
1= 3
(1S) + 1)
N
(MI-I)
N
[N|
OSA
1S
05R
Even when pruning at the nodes close to higher level of the tree, this perturbation remains negligible, as B is typically two orders of magnitude smaller than the pruning threshold

---

Published in Transactions on Machine Learning Research (08/2025)
Algorithm Layer-wise ReLU importance Estination
Input: onc-layer subsets N, the set of RcLU layers S, model M. Output: ReLU importance (SV) for each relu layer st.
1: su =
2: queue -- init with
subsets + generating all subsets of N
in elements descending order
quene <- subsets
while queue is not 0 do
6: mask pop the mask combination from queue
7:
ssst - -et accuracy of masked model
8:
all [l__ass+ +- add the candidate layer
9.
U_si + +get accuracy of all all_ma model
10:
if U si threshold then
11:
for child in find subsets(mask) do if child not in queue then
one element less subsets
12:
guewe.push(chilke end if end for 16: end if
13:
14:
15:
17: 80 s8 + (U_si- si -U__iiU_)) -U{))/((()))
18: end while
19: su = sv/
3.2 Prajeett-rooectioon Pruning Strategy
After obtaining the approximated SV for each ReLU layer, normalize these values and multiply them by the total ReLU budget to get the layer-wise ReLU budget. To avoid conflicts, we ensure that the allocated budget does not exceed the number of pixel-wise ReLU operations that each layer can provide. We fine- tune the pre-trained standard model (i.e., all ReLU persevered) and prune ReLU layers during fine-tuning, Practically, we consider x; E Rxxmxn denote the feature map before the i-th ReLU layer, where d, m, n denole the number of channel, width and length, respectively. We altach binary mask 6; with the shape 2, to the i-th ReLU layer. The output feature map of the masked ReLU layer can be formulated
x- -1 c.)+ x, + c ReLU(xx).
(3)
The masks al every layer are initialized as all-ones as we start with a standard model with all ReLU operation preserved.
Previous works, such SNL, use the norm to increase the sparsity of G, given L minw.c (((w,c(X),v) + A(Ei [|a]]]), where W denotes the weights and C= ( b denotes the total number of ReLU layers for a given model. However, it experience a significant drop in accuracy when targeting a low ReLU budget since the optimization goal has been shifted from improving accuracy to increasing sparsity. This causes model performance degradation. To address this, we propose gradient- projection strategy, which remove the I norm from the objective function and attach a pruning process after every gradient descent step. To be more specific, we calculate the gradient for G, which yields ranking of importance for the d x indicates higher importance of the ReLU operation at the corresponding location. Next, retain the top-k important ReLU operations, where & is the layer-wisc ReLU budget determined by the approximated Shapley value. The mask & is updated accordingly, where pruned ReLU operations are set to 0 and others to 1. This strategy ensures that the layerwise ReLU budget is always met during the entire fine-tuning process. Morcover, the objective function the sum of cross-entropy loss (to fit ground truth labels) and Lullback-Leibler (KL) divergence loss (to approximate the original model), which ensures the optimization is only focused improving model performance. The overall pipeline is depicted in Fig. 3
pixel-wise ReLU operations on that layer, where larger gradient magnitude
7

---

Published in Transactions on Machine Learning Research (08/2025)
AR
PR
Finergrained loyerwise
ReLU allocation
Bradient-projectin mask pruning
AR
PR
f1,3,4,53
layerwise iepor (tree bosed)
[2.31, 10.9, 10.1, 7.2, 1.2]
2) normalized SV For allecating ReLU budget [74, 333, 324, 230, 38] Total RoLu budgeti
Top-k ReLU wask gradient magnitude
HHH
KD with wask frozen
Figure The overview of PrivShap, AR denotes the all-ReLU model and PR denotes the partial-ReLU model. We calculate the approximated SV for each non-linear layer (ReLU) based the tree search algorithm. The normalized SV of each ReLU layer is used to allocated the layerwise ReLU budget, with given ReLU budget. Then we train the PR model with AR model by keeping top-k ReLU mask gradient magnitude and narrowing the similarity between PR and AR model. Such process applies the generated fino-grained layerwise budget allocation.
4 Evaluation
4.1
Experiment Setup
To evaluate our method and align with SOTA evaluation setting for clear comparison, we apply PrivShap models such ResNet18, 18, ResNet34 with SILU activation function, and WedeResNet-22 in three benchmark datasots: CIFAR-10, CIFAR-100 and ImageNm. CIFAR-10 and CIFAR-100 both have images with resolution of 32x32. CIFAR-10 contains 10 classes with 5,000 training images and 1,000 test images per class, whereas CIFAR-100 includes 100 classes, with 500 training images and 100 test images per class, Tiny-ImageNet has a larger image resolution of 64x64 and consists of 200 classes, each with 500 training and 50 validation images. The experiments are conduct on two servers with an AMD EPYC 7413 24-core Processor 64GB RAM, under WAN (1 GBps, ms latency) and LAN (300MBps, ms latency).
We first train the all-ReLU (AR) models on CIFAR-10, CIFAR-100, and Tmag-Im datascts using the SGD optimizer with an initial learning rate of 0.1, momentum of 0.9, and weight decay of 0.0005. The learning rate decays by factor of 0.1 at 80 and 120 epochs for CTFAR-10/100, using a batch size of 256. For Tiny-ImageNet, we train for 200 epochs with the same hyperparameters. After this, we apply gradient- projection pruming for partial-ReLU (PR) models with trimming threshold 5% and an initial learning rate of 0.001 using the Adam optimizer. Finally, the ReLU masks are frozen and the model parameters are fino-tuned following the distillation process of Kundu et al. 2023).
We follow the latency measurement method of Cho et al. (022b) and break down the sources of latency into two categories: ciphertext linear operations and GC-based ReLU operations. As the SNL follows the DELPHI framework, the ciphertext lincar computation is moved to offline and use BFV scheme for HE with default. 4096-slots eryplographic parameter of SEAL library. We set 15 bit-length share for model and 31 bit share for ReLU protocol. For ReLU latency, the wall-clock time for 1000 ReLU operations is measured as 0.021 seconds per 1000 ReLUs. We compare our method with multiple basclines including SNL Cho et al.) (2022b), DeepReDuce Jha al. (2021), Sphyux Cho
(022a)), SAFENet Lou et al. (2021), CryptoNAS Ghodsi et al. 2020),, SE.eet Kundu et al. (0033) and Auto FHE Ao & Boddeti (024)).
8

---

Published in Transactions on Machine Learning Research (08/2025)
The performance of PrivShap and other methods on model architectures with CIFAR-100. 'r' denotes the given ReLU budget, Comm. Saving denotes the ratio of communication costs associated with an AR model to that of the corresponding PR model with reduced ReLUs.
Table 2:
baseline Acc%
Test Acc%
Comm. Saving
min<r <max
model
#Re.UU(k)
Method
CIFAR-100
VGG11
76.84
90.8
75.4
7.2x
ResNet18
78.3
75.2
Ours
74.9
9.4x
0<r<100k
(esNet34(SiiLU)
78.23
69.6
71.2
11.2x
VGG11
76.84
91.5
SNL
70.2
5.3x
ResNet18
78.3
86.3
DeepReDuce
68.5
10.3x
VGG16
78.7
110
78.2
3.4x
Ours
ResNet50
78.3
160
77.8
3.6x
(eSL)
76.92
280
Sphynx
71.4
2.8x
100k<r<400k
WRN22-8
81.2
265
SAFENet
78.5
4.1x
VGG11
76.84
150
CryptoNAS
70.3
2.7x
Res
75.15
167
AutoFHE
72.3
4.2x
RosNet18
78.3
135
SENet 74.2
4.5x
Table 3: The performance of PrivShap and other methods on model architectures with CIFAR-10. 'r' denotes
the given ReLU budget, Comm. Saving denotes the ratio of communication costs associated with an AR model to that of the corresponding PR model with reduced ReLUs.
baseline
Test Acc%
Comm. Saving
min<r <max model
RcLU(k)
Method
Acc%
CIFAR-10
VGG11
93.2
80.8
92.5
4.5x
(esL)t33(SiLLU)
95.8
55.2
Ours
90.4
10.4x
0<r<100k
ResNet101
95.9
49.6
89.2
21.7x
VGG11
93.2
85.5
SNL
85.8
17.1x
ResNet18
95.8
75.3
DeepReDuce
83.2
9.3x
VGG16
94.1
110
93.2
3.1x x
Ours
(eS.)))4(SIILU)
95.8
160
95.3
3.5x
ResNet18
95.9
180
Sphynx
91.2
5.6x
100k<r<400k
WRN22-8
94.6
165
SAFENet
90.4
2.8x
VGGI1
93.2
150
CryptoNAS
87.5
2.3x
ResNet34
94.3
168
AutoFHE
91.7
3.8x
Res.Net18
95.8
125
SENet
89.7
4.9x
4.2 PrivShap achieves better inference accuracy-efficency trade-off
To demonstrate method to model performance and inference acceleration pruned model, compare our method with gradient-based ReLU pruning (SNL), ReLU budget allocation (SENet), and other SOTA, as shown in Fig. 4 Our method achieves up to an 8.13% accuracy improvement over SENet on ResNet-34 in CIFAR-100 with a ReLU budget 150k and 5.5% accuracy boost on 7k lower ReLU budget. On Tiny- ImageNet, we observe an improvement accuracy of 3.1% to 4% in various ReLU budgets. For CIFAR-10, the accuracy gain is around 2%, likely due to the simpler 10-class classification task and high bascline accuracy.

---

Published in Transactions on Machine Learning Research (08/2025)
Timylmaggeet
62-
sShapp
32-
52
220
230
200
300
150
Tablil (rsppelcc
Palli busggette
Patu bastpet00
Figure 4: The accuracy performance of PrivShap v.s existing ReLU pruning works on various ReLU-budgett in ResNet-34. Our method achieves better Pareto frontier than existing works different datasets.
Table 4: The exact and estimated SVs and standard deviation per ReLU layer in ResNet-9 on CIFAR- 100 with 5 trials. p denotes the Pearson correlation.
ReLU Layer Index
2
3
4
6
8
Exact SV
10.98
11.12
11.55
11.03
10.85
10.87
6.47
0.13
Estimated
8.66
9.17
9.24
9.28
9.07
8.45
5.27
0.09
0.997
(V(our))
+.01
+.02
#.01
+.02
1.02
+.02
1.02
1.02
+.02
Estimated SV(MC)
8.82
9.02
8.92
9.11
9.32
8.61
6.44
0.11
0.987
+.05
+.08
+.02
#.03
+.01
+.05
5.02
#.04
+.04
Estimated
9.41
10.43
9.96
10.17
9.35
9.15
7.33
0.24
0.981
SV(MAB)
+.07
+.12
+.08
+.05
+.02
+.06
+.04
+.02
+
For low ReLU budgets (< 100k), our method reduces the number of ReLUs by 2.3x and results in a 11.2x communication savings in online latency for NesNet-34 in CIFAR-100, 100, as shown in Table 2 outperforming existing methods both accuracy and communication efficiency. Our method also demonstrates 1.3x ReLU budgets compared to the SOTA and up to 21.7x communication savings for ResNet-101 in CIFAR-10, shown in Table 3 Our method also maintains higher accuracy at higher ReLU budgets (> 100k) compared to other approaches, achieving around 3.2% more accuracy in ResNet-18 with CIFAR-100, shown in Table 2
Table 5: The wall-clock time and pruned model accuracy of different SV estimation methods
Pruned model accuracy
Model
Methods
Wall-clock time
C-100
C-10
T T-imagenet
Ours
5 min
76.2%
93.8%
65.4%
ResNet-9
MC
45 min
71.4%
89.3%
57.7%
MAB
> 52 min
71.9%
90.6%
59.6%
Ours
35 min
78.5%
94.3%
65.1%
ResNet-18
MC
70 min
73.9%
91.5%
59.6%
MAB
78 min
72.4%
93.1%
63.7%
Ours
45 min
77.3%
93.5%
67.1%
RosNet-34
MC
210 min
72.1%
90.1%
58.7%
MAB
> 196 min
73.6%
87.9%
63.2%
10

---

Published in Transactions on Machine Learning Research (08/2025)
4.3 Tree- tree-trimming algorithm outperforms other SV estimation methods
To evaluate the accuracy of our SV estimation using the tree-trimming algorithm, compare the estimated values with the exact SV. Given the high computational complexity of exact SV calculation, we performed our experiments on a lightweight model, ResNet-9, which consists of eight ReLU layers, on the CIFAR- 100 dataset. We use inference accuracy as the utility function and a threshold of 5% in tree-trimming estimation. Table 4 shows the estimated and exact SVs of all ReLU layers, which align with each other, achieving a Pearson corrclation of 0.997. We compare with other SV estimation methods, including MC Ghorbani & Zou (2019) and MAB Ghorbani & Zou (2020). They show slightly lower correlations.
Notably, both the exact and estimated SVs show fluctuating importance trend on ReLU importance, and the last two ReLU layers appear to be the least important compared to the former oncs. Previous works, such as SENet, SNL, and DeepReDuce, often assume a consistent Irend of layerwise importance. These results are counterexamples, implying that these assumptions might be oversimplified and not able to generalize to different model architectures and downstream tasks.
Moreover, we evaluate the computational efficiency and the accuracy of pruned models using different SV estimation methods across various model architectures and datasets (Table 5). For fair comparison, the MCs and MAB methods calculate the approximated SV for each ReLU pixel in a single run and keep the top-k budget of ReLU pixels based on normalized SV importance. Results show that MC and MAB take 2x to 4.7x more wall-clock time (which denotes the sum of time of calculating the SV for each ReLU layer) than our mro-trimmming method. Our method achieves 2.8% to 4.6% higher model accuracy, while only costing about half of the wall-clock time. All these results provide strong evidence that the tree-trimming algorithm we proposed outperforms other SV estimation methods in both accuracy and efficiency.
4.4 Trimming threshold sensitivity
As detailed in Sec. B our tree-trimming algorithm has hyperparameter, the threshold for troe-trimming. We now discuss the sensitivity of this threshold. We estimate the SV per ReLu layer with trimming thresholds of 5%, 10%, and 15%, different thresholds on various models and datasets (See Figure In general, the estimated SVs show consistent trends across different thresholds.
Furthermore, we evaluated how this threshold affects the ultimate inference accuracy. With a 150k ReLU budget, we plot inference accuracy as function of trimming threshold across different models and datasets in Fig. Not surprisingly, the inference accuracy drops with threshold increases, to inaccurate SV estimation (see details in Sec. 3.1 However, the gradient is reasonably small between 5% and 20%, which implies that the model's performance remains stable if the trimming threshold is chosen from this range.
higher threshold leads
5 Discussion and Conclusion
In this paper, we propose PrivShap, method for finer-grained network linearization to improve private inference. We show that previous works use over-simplefied layerwise ReLU importance assumptions, such as inverse correlation between ReLU and weight layers, which can not generalize to different models and tasks. We propose to use Shapley Values for fine-grained and accurate ReLU importance estimation and allocate the budget accordingly. As exact. Shapley value calculation extremely computationally expensive, we propose a treo-trimming algorithm for fast Shapley Value approximation, allowing for generalization to larger models with many layers. Additionally, we propose a gradient-projection pruning strategy to enforce the number of ReLU operations per layer below allocated budgets. Our experiments show that PrivShap reduces ReLUs by up to ~2.x with similar accuracy, and improves accuracy by up to ~8.13% with comparable ReLU budgets, when compared with the prior arts. Furthermore, when compared with other Shapley value estimation approaches, the proposed tree-trimming method outperforms in both approximation accuracy and efficiency.
Naturally, our method has some limitations. 1) Our work aims to provide new perception on measuring the importance of activation layers that can guide the pixel-wise activation pruning for fast private inference.
11

---

Published in Transactions on Machine Learning Research (08/2025)
CIFAR 10 T- 535 10 13%
020
50
015
CIRA 5% CIRAR T: 19%
ReaU layorindex
RoLU bayor ind
Figure The approximated SV each ReLU layer in ResNet-18 on CIFAR10.
Figure 6: The approximated SV of each ReLU layer in ResNet-16 18 on CIFAR100.
CIFAR 10 59
20-
CIFAR 195 T- 5% CIFAR 10%
10 17%
M M
M
2-
15 Ballll layer incles
30
20
25
30
PeeUU layer indle
Figure The approximated SV each ReLU layer in ResNet-34 on CIFAR10.
Figure 8: The approximated SV of each ReLU layer in ResNet-34 on CTFAR100.
CIFAR-100
C/AR-11
68
nodel SOCUSSC
80
80-
86
reseet 10
#5set:30 WidReeceet33
30
30
00
SCCL threshold
accuracc chreshold
ScouraCY dhreshold
Figure 9: The accuracy performance of PrivShap with different accuracy thresholds in various datasets.
While lack of further experiments on larger model and more complex dataset, our method can be generalized such scenarios as those models follow similar structure. 2) Our method can be sensitive to the threshold choice to make the optimized trade-off between estimation accuracy and estimation efficiency for specific models. The threshold choice and theoretical analysis remain to be explored in future study. 3) Our method
12

---

Published in Transactions on Machine Learning Research (08/2025)
can be limited to apply to linear layers pruning, since linear parameters increase exponentially in each layer and hard to mask out. We plan to explore such issue in future study.
6 Acknowledgment
This work supported in part by the NSF under Grant OAC-2320999, CNS-2120279, 11S-2236578 DGE- 2336109, CNS-2413009, and SaTC-2439013, CNS-2153358, DoD Center of Excellence in Al and Machine Learning (CoE~AIML) under Contract Number W911NF-200 20-22-0777 and the Commonwealth Cyber Initia- tive, the Air Force Research Lab under grant FA8750-19-3-1000, and NSA under grants HQ00342410026 and 1198230-11-1-0233.
References
Wei Ao and Vishnu Naresh Boddeti. AutoFHE: Automated adaption of CNNs for efficient evaluation over FHE. In USENIX Security 24, pp. 2173-2190. USENIX Association, 2024. ISBN 978-1-939133-444
Yulong Bai, Ying Wang, Xiangrui Xu, Yuhang Yang, Hina Batool, Zahid Iqbal, and Jiuyun Xu. Asyncde- fender: Dynamic trust adaptation and collaborative defense for byzantine-robust asynchronous federated lcarning. Computer Networks, 269:111430, 2025. ISSN 1389-1286. doi: /ttp///o...og/0010/6////
Mihir Bellare, Viet Tung Hoang, and Phillip Rogaway. Foundations of garbled circuits. In Proc. CCS, pp. 784-796, 2012.
Jung Hee Cheon, Andrey Kim, Miran Kim, and Yongsoo Song. Homomorphie encryption for arithmetic of approximate numbers. In Proc. ASIACRYPT, 2017.
Ilaria Chillotti, Nicolas Gama, Mariya Georgieva, and Malika Izabachene. Faster fully homomorphic encryp- tion: Bootstrapping in less than 0.1 seconds. In ASIACRYPT, pp. 3-33, 2016.
Minsu Cho, Zahra Ghodsi, Brandon Reagen, Siddharth Garg, and Chinmay Hegde. Sphynx: deep neural network design for private inference. IEEE Security & Privacy, 20(5):22-34, 2022a. doi: 10.1109/MSEC. 2022.3165475.
Minsu Cho, Ameya Joshi, Brandon Reagen, Siddharth Garg, and Chinmay Hegde. Selective network lin- earization for efficient private inference. In International Conference on Machine Learning, pp. 3997-39611 PMLR, 2022b.
Xiaohan Ding, Xiangxin Zhou, Yuchen Guo, Jungong Han, Ji Liu, et al. Global sparse momentun sgd for pruning very deep neural networks. Advances in Neural Information Processing Systems, 32, 2019.
Junfeng Fan and Frederik Vercauteren. Somewhat practical fully homomorphic encryption. Cryptology
ePrinl Archive, 2012.
Zahra Ghodsi, Akshaj Kumar Veldanda, Brandon Reagen, and Siddharth Garg. Cryptonas: Privale inference on a relu budget. Advances in Neural Information Processing Systems, 33:16961-16971. 2020.
Amirata Ghorbani and James Zou. Data shapley: Equitable valuation of data for machine learning. Tn International conference machine learning, pp. 2242 2251. PMLR, 2019.
Amirata Ghorbani and James Y Zou. Neuron shapley: Discovering the responsible neurons. Advances in neural information processing systems, 33:5922-5932, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual lcarning for image recognition. In Proc. IEEE CVPR, 2016.
Nandan Kumar Jha and Brandon Reagen. Deepreshape: Redesigning neural networks for efficient private inference. In Transactions Machine Learning Research (TMI.R), 2024.
13

---

Published in Transactions on Machine Learning Research (08/2025)
Nandan Kumar Jha, Zahra Ghodsi, Siddharth Garg, and Brandon Reagen. Deepreduce: Relu reduction for fast, private inference. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 4839 4849. PMLR, 18 24 Jul 2021.
Chiraag Juvekar, Vinod Vaikuntanathan, and Anantha Chandrakasan, {GAZELLE): A low latency frame- work for secure neural network inference. In 27th USENIX security symposium (USENIX security 18),
pp. 1651-1669, 2018.
Souvik Kundu, Shunlin Lu, Yuke Zhang, Jacqueline Tifsany Liu, and Peter Anthony Beerel. Learning to Linearize Deep Neural Networks for Secure and Efficient Private Inference. In The Eleventh International Conference on Learning Representations, 2023.
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. In International Conference on Learning Representations, 2022.
Qian Lou, Yilin Shen, Hongxia Jin, and Lei Jiang. Safenet: A secure, accurate and fast neural network inference. In International Conference on Learning Representations, 2021.
Pralyush Mishra, Ryan Lehmkuhl, Akshayaram Srinivasan, Wenting Zheng, and Raluca Ada Popa. Delphi: a cryptographic inference system for neural networks. In Proceedings of the 2020 Workshop on Privacy- Presercing Machine Learning in Practice, pp. 27 30, 2020.
Deevashwer Rathee, Mayank Rathee, Nishant Kumar, Nishanth Chandran, Divya Gupta, Aseem Rastogi, and Rahul Sharma. CrypTFFlo2:: Practical 2-Parly Secure Inference. In Proc. ACM CCS, 2020. ISBN 9781450370899, doi: (11145/3372297.344
Lloyd S Shapley. A value for n-person games. Contribution to the Theory of Games, 2, 1953.
Mukund Sundararajan and Amir Najmi. The many shapley values for model explanation. In International conference machine learning, pp. 9269-9278. PMLR, 2020.
Zhenzhen Wang, Cesar A. Santa-Maria, Aleksander S. Popel, and Jeremias Sulam. Bi-level graph learn- ing unveils prognosis-relovant tumor microenvironment patterns in breast multiplexed digital pathology. bioRriv, 2024. doi: 10.1101 10.1101/2024.04 /22///00....2.000111111
Zheng Xing and Weibing Zhao. Unsupervised action segmentation via fast learning of semantically consistent actoms. In Proceedings of the AAAT Conference on Artificial Intelligence, pp. 2270-6278, 2024a.
Zheng Xing and Weibing Zhao. Block-diagonal guided DBSCAN clustering. IEEE Trans. Knowl. Data Eng., 36(11):579--5722, 2024b.
Zheng Xing and Weibing Zhao. Segmentation and completion of human motion sequence via temporal learning of subspace variety model. IEEE Trans. Image Process., 1(1):1-1, 2024c.
Jiuyun Xu, Yingzhi Zhao, Xiaowen Li, Liang Zhou, Kongshang Zhu, Xiangrui Xu, Qiang Duan, and Ruru Zhang. Teg-di: Dynamic incentive model for federated learning based on tripartite evolutionary game. Newrocomputing, pp. 129259, 2024a.
Jiuyun Xu, Liang Zhou, Yingzhi Zhao, Xiaowen Li, Kongshang Zhu, Xiangrui Xu, Qiang Duan, and RuRu Zhang. A two-stage fodcrated learning method for personalization via selective collaboration. Computer Communications, pp. 108053, 2025.
Xiangrui Xu, Qiao Zhang, Rui Ning, Chunsheng Xin, and Hongyi Wu. Comet: A Commuiccation-effiien and Performant Approximation for Private Transformer Inference. arXiv preprint orXiv:2405. 17485, 2024b.
Xiangrui Xu, Qiao Zhang, Rui Ning, Chunsheng Xin, and Hongyi Wu. SPOT: Structure Patching and Overlap Tweaking for Effective Pipelining in Privacy-Preservvnn Preserving MLaaS with Tiny Clients. In 2024 IEEE 44th International Conference Distributed Computing Systems (ICDCS), pp. 1318 1329, 2024c.
14

---

Published in Transactions on Machine Learning Research (08/2025)
Sergey Zagoruyko. Wide residual networks. arXiv preprint orXiv:1605.071466 2016.
Kongshang Zhu, Jiuyun Xu, Liang Zhou, Xiaowen Li, Yingzhi Zhao, Xiangrui Xu, and Shibao Li. Dmaf: data-model anti-forgetting for federated incremental learning, Cluster Computing, 28(1):30, 2025.
7 Appendix
7.1 More comparisons
We evaluate our method against DhepReShape Jha & Reagen 2024 by comparing the pruned model accu- racy on the ResNet34 architecture under fixed ReLU budget. This comparison conducted on both the CIFAR-100 and Tiny ImageNet datasets. Fig 10 and Fig show that our method has around 4.3% more accuracy of pruned model accuracy on same given ReLU budget.
DeepiteShape
Deep Sthhppe
0.70
0.715
Accurecy
Accurecy
0.65
0.700
3.075
0.55
100
150
200
250
300
350
300
200
300
400
500
600
700
800
ReLukk) budget
of ReLUIko budget
Figure 10: The accuracy of ResNet34 on CIFAR- 100, HybReNet for DecpReShape for diffcrent given ReLU budgets.
Figure 11: The accuracy of ResNet34 on Tiny- Imagenet, HybReNet for DecpReShape for dif- ferent given ReLU budgets.
7.2 Evidence of Marginal Gain
To provide some evidence for Ui) - U(")) S 3, where < a in Section. 3.1 subset's the marginal utility gain U(S" U - US") under an ancestor node that satisfies U(S) < 5% for ResNet-9, 18, 34 models CIFAR 100. The Fig. in two order of magnitude of U(S).
we sample 100 descendant
13 15
and 17 show that the marginal utility can be smaller
63304
0680
0.0
Mest with Devehon
Lager ade
Figure 12: The 100 random samples of US' U i) (S') value distribution of each i-th ReLU layers in ResNet-9 model (CIFAR-100).
Figure 13: The zoom-out comparison with each ReLU layer marginal gain and threshold ResNet-9 (CIFAR-100).
15

---

Published in Transactions on Machine Learning Research (08/2025)
0.000
6304
6.304
0.0
04932
1000
3:500
31
35
13
Figure 14: The 100 random samples of US' U i) (S') value distribution of each i-th ReLU layers in ResNet-18 18 model (CIFAR-100).
Figure 15: The zoom-out comparison with each ReLU layer marginal gain and threshold in ResNet-18 (CIFAR-100).
6.306
000
04812
Denaban
0.990
049M
Figure 16: The 100 random samples of USS' U i) (S') value distribution of each i-th ReLU layers in ResNet-34 model (CIFAR-100).
Figure 17: The zoom-out comparison with each ReLU layer marginal gain and threshold in ResNet-34 (CIFAR-100).
16


Reviews:
[{"review_id": "cICbxOP78X", "reviewer": "Reviewer_TeZD", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n\nThe paper introduces a novel method for ReLU pruning and demonstrates its effectiveness through experiments, showing that it outperforms state-of-the-art approaches. The authors also provide a theoretical analysis to support the validity of their tree-trimming algorithm.\n\nWeaknesses:\n\nMostly smaller weaknesses that I list here:\n1. The writing could be improved, particularly in the methods section. For example, Section 3.2 feels somewhat rushed and would benefit from clearer explanations or additional detail. Please check the requested changes section for more details.\n2. In the theoretical analysis section, there is a strong statement without any evidence or proof: \"We start with a natural observation that $\\forall S' \\subseteq S$ and $U(S) \\leq \\alpha$, , then the marginal gain satisfies  $U(S' \\cup \\{i\\}) - U(S') \\leq \\beta$, where $\\beta \\ll \\alpha$.\" And the whole proof is based on this statement.\n3. In some of the experiments shown in Tables 2 and 3, it is hard to compare different methods since they have different architectures. I wonder why the authors decided to compare their method that way.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Due to the high computational overhead of ReLU activations in private inference, this paper introduces a ReLU pruning method to reduce their cost. The authors propose a tree-trimming algorithm based on Shapley values to estimate the importance of each ReLU layer. Using these importance values, they allocate a computational budget to each layer and apply gradient-projection pruning to selectively remove ReLUs at finer granularities (e.g., pixel and channel levels).\n\nThe authors, in their experiments on three datasets and different model architectures, show that their method results in fewer ReLU operations and overall has a better performance compared to the SOTA methods with a similar ReLU budget.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. This one is more of a question than a request since I'm not super familiar with HE methods. I wonder if instead of removing ReLU activations, we can replace them with other non-linear activations. Would that have the same computational overhead issue? Or are there other reasons that authors and other papers decided to completely remove ReLU activations?\n2. Table 1 is not referenced anywhere and needs to be explained in the main text.\n3. The theoretical analysis shows that if $U(S' \\cup \\{i\\}) - U(S') \\leq \\beta$ for all $S' \\subseteq S$, then $\\Delta \\text{SV}_i \\leq \\beta \\frac{|S| + 1}{|N|}$ and the authors concluded we can prune and nodes closer to the root and avoid the exponential computations with negligible purterbation but it is all based on having  $\\beta \\ll \\alpha$ which is not shown anywhere in the paper why it is true. Can you provide some evidence for that?\n4. In Figure 1, the number for one of the dashed arrows is missing.\n5. I recommend that the authors provide a bit more explanation at the beginning of Section 3, before introducing Shapley value details, to give a clearer overview of how their method works. Specifically, it would help to clarify that the approach involves two steps: first, using Shapley values and tree-trimming to allocate a ReLU budget across layers, and second, applying gradient-projection pruning to remove activations at a finer granularity. This structure is not immediately clear from the current text, and I had to reread the section several times to fully understand it.\n6. Section 3.2 would benefit from additional explanation, detail, and mathematical formulation. For instance, the dimensions d, m, and n in $x_i \\in \\mathbb{R}^{d \\times m \\times n}$ are not defined—what do they represent? It’s also unclear from which objective the $l_1$-norm is being removed. Additionally, the gradient descent step is not sufficiently explained. Even if these components are based on prior work, since they are integral to your method, you should provide the necessary details to ensure clarity and self-containment for the reader.\n7. In Table 2 and 3, model architectures are different and some of the methods are not comparable. Can you please explain why?\n8. In the first paragraph of page 10, you mention your method has more communication savings for ResNet-18 in CIFAR-10, but I do not see any ResNet-18 for your method in the table.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "2W6SX0kGzL", "reviewer": "Reviewer_R74T", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n- The maniscript uses Shapley values to quantify ReLU layer importance, moving beyond heuristic assumptions,  which provides a theoretically grounded framework for fine-grained budget allocation.\n\n- The maniscript provides bounds on the error introduced by tree trimming, ensuring that the approximation does not significantly perturb Shapley value estimates.\n\n\n**Weaknesses**\n- The method relies on manually setting a trimming threshold of 5% to prune subsets. While experiments show stability within varying values (5, 10, 15%), an adaptive thresholding mechanism could improve robustness across architectures.\n\n- While the tree-trimming algorithm improves efficiency, its applicability to large-scale models remains untested.\n- In Table 5, why do the wall-clock time and pruned model accuracy of ResNet-18 and ResNet-34 appear to be identical? Could the authors clarify whether this reflects experimental consistency?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This maniscript addresses the critical challenge of reducing communication overhead in private inference (PI) caused by non-linear operations like ReLU. The authors propose PrivShap , a novel method leveraging Shapley values from cooperative game theory to assess the layer-wise importance of ReLU layers and allocate budgets for pruning. To mitigate the computational complexity of exact Shapley value calculation, they introduce a tree-trimming algorithm for efficient estimation and a gradient-projection pruning strategy to enforce budget constraints. Extensive experiments on diverse models (ResNet, WideResNet, VGG) and datasets (CIFAR, Tiny-ImageNet) demonstrate superior accuracy-efficiency trade-offs compared to existing methods like SENet and SNL.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please refer to the Weaknesses.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Qw22c2Ulua", "reviewer": "Reviewer_DU4C", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n- The method is clearly illustrated in the paper.\n- The method is relatively simple, which is nice. And the results of the experiments, specifically Figure 4, are very convincing.\n\n## Weaknesses\n\nI do not find any critical flaw in this paper. But I'm quite new to this line of work so I'll wait to to see other reviewers' comments. Nonetheless, I have a couple of comments:\n\n- In the experiments, can the authors compare the proposed method with the recent private inference methods from Jah and Reagan (2024) and Sarkar, Kundu & Beereil (2024)?\n- In Table 4, why do you use Pearson correlation as the performance metric? Aren't distance based metric like $\\ell^2$ loss better for measuring the approximation quality?  \n  Also, the values of $\\rho$ is this table are indistinguishable without standard deviations from repeated trials.\n- For the proof in page 6, doesn't the conclusion immediately follow from the LHS of Eq. (2) since $\\binom{|S|}{|S'|} \\leq \\binom{|N|-1}{|S'|}$?\n\n\nReferences:\nJha, N. K., & Reagen, B. (2024). DeepReShape: Redesigning Neural Networks for Efficient Private Inference. Transactions on Machine Learning Research. Retrieved from https://openreview.net/forum?id=iwCBWULItx\nSarkar, S., Kundu, S., & Beereil, P. A. (2024). RLNet: Robust Linearized Networks for Efficient Private Inference. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 244–253. doi:10.1109/CVPRW63382.2024.00029", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper focuses on communication-efficient private inference of ReLU networks via layer pruning. To this end, the authors propose to calculate the ReLU only for the top-$k$ layers in terms of their Shapley values. To combat the high computational cost of computing the exact Shapley values, the authors propose to approximate them by greedily construct a tree with a network with one layer, say layer $i$, removed as its root. A tree is then constructed by iteratively removing a network's layer, and the branch to any node (any network) with low accuracy is removed. Shapley value for layer $i$ is then calculated using only the set of visible nodes in the tree, which is much smaller than the set of all sub-networks of the root network.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "In Section 2.3, please refer to the specific section in the cited paper that contains the proofs of Zero/Negative Contribution and Efficiency.", "broader_impact_concerns": "There is no ethical concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)