S++: A Fast and Deployable Secure-Computation Framework forPrivacy-Preserving Neural Network Training
Prashanthi Ramachandran 1 Shivam Agarwal 1 Arup Mondal 1
Aastha Shah 1 Debayan Gupta 1
1
Ashoka University
prashanthi.r@alumni.ashoka.edu.in, shivam.agarwal@alumni.ashoka.edu.in, arup.mondal phd19@ashoka.edu.in,aastha.shah@alumni.ashoka.edu.in, debayan.gupta@ashoka.edu.in

Abstract
We introduce S++, a simple, robust, and deployable framework for training a neural network (NN) using private datafrom multiple sources, using secret-shared secure function
evaluation. In short, consider a virtual third party to whom
every data-holder sends their inputs, and which computes theneural network: in our case, this virtual third party is actuallya set of servers which individually learn nothing, even with amalicious (but non-colluding) adversary.
Previous work in this area has been limited to just one specificactivation function: ReLU, rendering the approach impractical for many use-cases. For the first time, we provide fastand verifiable protocols for all common activation functions
and optimize them for running in a secret-shared manner. The
ability to quickly, verifiably, and robustly compute exponentiation, softmax, sigmoid, etc., allows us to use previously
written NNs without modification, vastly reducing developer
effort and complexity of code. In recent times, ReLU has beenfound to converge much faster and be more computationallyefficient as compared to non-linear functions like sigmoid ortanh. However, we argue that it would be remiss not to extend
the mechanism to non-linear functions such as the logistic
sigmoid, tanh, and softmax that are fundamental due to theirability to express outputs as probabilities and their universal
approximation property. Their contribution in RNNs and a
few recent advancements also makes them more relevant.

Introduction
Neural networks (NN) are used in areas ranging from image classification to machine translation, and there are often
multiple parties that contribute possibly private data to thetraining process. However, training data interaction and theresulting model may still leak a significant amount of private data. Thus, there arises a need for secure neural networks, which can help parties collaboratively train a neuralnetwork without revealing their private input data.
Several approaches have been proposed to overcome possible data leakage, based on secure multi-party computation(MPC). MPC is a powerful way to protect sensitive training
data which uses a range of cryptographic primitives to allow
multiple parties to compute a function without revealing the
inputs of any individual party (beyond what is implied by
Copyright c 2021, Association for the Advancement of Artificial
Intelligence (www.aaai.org). All rights reserved.

the output of the function itself). However, previously proposed MPC-based schemes (Ohrimenko et al. 2016; Huntet al. 2018; Mohassel and Zhang 2017; Wagh, Gupta, andChandran 2018; Patra and Suresh 2020) do not support models that use exponentiation-based activation functions suchas logistic sigmoid, softmax, and tanh. These protocols arelargely restricted to ReLU and its variants, which might restrict applicability in some specific cases (as described in the
motivation section).
In this paper, we propose S++, a three-party securecomputation framework for secure exponentiation, andexponentiation-based activation functions such as logistic
sigmoid, tanh, and softmax. This enables us to constructthree-party secure protocols for training and inference ofseveral NN architectures such that no single party learns
any information about the data. S++ is an efficient MPCbased privacy-preserving neural network training framework based on (Wagh, Gupta, and Chandran 2018) for 3PC
with the activation functions logistic sigmoid, tanh, theirderivatives, and softmax. In our setting, there are D (where
D can be arbitrary) data owners who wish to jointly train amodel over their data with the help of 3-servers. In the setup
phase, these data owners create ”additive secret-shares” oftheir input data and send one share each to 2-servers. In thecomputation phase, the 3-servers (the 2 primary servers andthe helper server) collectively run an interactive protocol
to train a neural network on the data owners’ data without
learning any information beyond the trained model.

Contributions
We summarize our key contributions as follows:• We first propose a secure protocol for exponentiation in
the three-party setting. This protocol is based on SCALE
MAMBA’s (Aly et al. 2020) protocol for base 2 exponentiation. It also uses primitives from (Catrina and Saxena2010). We modify these ideas for our three-party setting
and additively secret-shared data.
• We describe novel secure protocols for the logistic sigmoid, softmax, and tanh—popular activation functions
that are significantly more complex than ReLU (describedby (Wagh, Gupta, and Chandran 2018))—along with theirderivatives with the help of the above-mentioned exponentiation protocol. The inclusion of these protocols in a

secure and private setting vastly increases the practicality of the framework and enables people to convert their
protocols into secure ones without having to redesign theactual internal structure of their NNs.

Organization of the paper
The remainder of this paper is organized as follows: wefirst go over our motivation for extending secure protocols
to functions such as the logistic sigmoid, tanh and softmax. Then, we look at recent work in this area, largely inthe realm of MPC. After that, we describe the notations
used in our protocols and explain some cryptographic primitives. In the Protocols section, we delve into our architecture
and the supporting protocols, specifically the supporting 3server protocols that serve as building blocks for our main
protocols. We describe our main protocols: exponentiation,
logistic sigmoid, tanh, their derivatives, and softmax. After
that, we describe the theoretical efficiency of our protocols
and provide an evaluation of our experiments. Lastly, we describe the future plans and directions for this work.

Motivation
Logistic sigmoid and variants The logistic sigmoid is
a common activation function to introduce non-linearity,
1
where sigmoid(x) =. It essentially squashes a real
1 + e−xvalue in [0, 1] and has been used widely over the years for
its simplicity, especially in binary classification tasks witha maximum likelihood loss function. It also has importantvariants like the tanh and softmax:• Tanh as a rescaled logistic sigmoid: The tanh is a rescaling and stretching of the logistic sigmoid that maps to [-1,1] i.e., tanh(x) = 2×sigmoid(2x) − 1. Extending the protocol to the tanh facilitates applications in recurrent neuralnetworks where it is widely used.
• Logistic sigmoid as a special case of the softmax:
The softmax is often used for classification tasks, whereezi. The logistic sigmoid happens
softmax(zi ) = Pkzi
i=1 e
to be a special case of this function, often used for binary classification. Enabling softmax thus also extends
the framework to multi-class classification problems.
Why is ReLU not enough? Despite ReLU being morecomputationally efficient than logistic sigmoid or tanh
(Krizhevsky, Sutskever, and Hinton 2017), we argue that it isworth extending the secure computation protocol to popularfunctions particularly involving exponentiation such as logistic sigmoid, tanh, and softmax for the following reasons(with more details in the appendix):• For classification tasks, output layers usually use softmax
or logistic sigmoid because they make for more interpretable values in [0, 1] as opposed to ReLU and variations that are unbounded. In fact, though previously missing in the literature, (Asadi and Jiang 2020) provide theoretical proofs on the universal approximation of softmax
used in the output layer for multiclass classification tasks
as well.

• ReLU’s unbounded nature limits its applicability in
RNNs. Capturing long-term dependencies becomes
harder with an unbounded function due to exploding gradients, as explored in (Pascanu, Mikolov, and Bengio2013). Because of the temporal correlations captured by
RNNs (by forming connections between hidden layers),
long-term components add up and the norm of the gradient of the hidden layers explode and ReLU’s unboundednature exacerbates this problem. There have been a few
notable improvements to assuage this, but even in LSTMs,
the tanh often gives more consistent results (Kent andSalem 2019) and ReLU tends to require greater finetuning.
• ’Leaky’ ReLU has been used to overcome ReLU’s vanishing gradient problem. (Xu, Huang, and Li 2016) have also
revised logistic sigmoid to a scaled version of it and tanhto a ‘leaky’ tanh. The ‘leaky’ version penalizes the negative part of the domain. They found it to be much fasterthan the tanh, it gave better results than ReLU, and is almost identical to leaky ReLU. Such potential improvements highlight the need for more secure protocols forsuch functions.
Though ReLU has become widely adapted in the past few
years, there are still some tasks wherein sigmoids are preferred over ReLU, and it is worthwhile to extend such secure
protocols for these as well.

Recent Work
There have been many privacy-preserving machine learning(PPML) frameworks for various situations, such as Decision Trees (Agrawal and Srikant 2000; Lindell and Pinkas
2000), Linear Regression (Du and Atallah 2001; Sanil et al.
2004), k-means clustering (Jagannathan and Wright 2005;
Bunn and Ostrovsky 2007), SVM classification (Yu, Vaidya,
and Jiang 2006; Vaidya, Yu, and Jiang 2008), and logistic
regression (Slavkovic, Nardi, and Tibbits 2007). However,these cannot generalize to a number of (complex and highaccuracy) standard machine learning applications.
To overcome this, SecureML (Mohassel and Zhang 2017)provided secure protocols for linear regression, logistic regression, and neural networks with linear activations, using
a combination of arithmetic and garbled circuits for a single
semi-honest adversary in both the 2 and 3-server models.
It also provided secure protocols for logistic sigmoid andsoftmax in the 2-server setting (although very briefly) anda method for truncating decimal numbers. MiniONN (Liuet al. 2017) optimized the protocols of SecureML by reducing the offline cost of matrix multiplications and increasing the online cost for the 2-server model in the semihonest adversary setting. Chameleon (Riazi et al. 2018) andGazelle (Juvekar, Vaikuntanathan, and Chandrakasan 2018)provided secure inference protocols which are computationally secure against one semi-honest adversary in the 3-serverand 2-server models. Chameleon removes expensive oblivious transfer protocols by using the third party as a dealer,while Gazelle focuses on making the linear layers morecommunication efficient by providing specialized packing
schemes for additively homomorphic encryption schemes.

SecureNN (Wagh, Gupta, and Chandran 2018) provides
secure neural network training protocols for non-linear activation functions in the 3-party setting with up to one malicious corruption and shows that the honest-majority can improve the performance by several orders of magnitude. For
linear regression, logistic regression and neural networks,
the problem is even more challenging as the training procedure computes many instances of non-linear activation functions such as logistic sigmoid, tanh, and softmax that are expensive to compute inside a 2 and 3-server.
SecureNN provides efficient protocols for computing the
ReLU activation, maxpool and their derivatives. It suggeststhat the general idea can extended to other variants andpiecewise linear approximations. However, (Mohassel and
Zhang 2017) show experimentally that low-degree polynomial approximations, like piecewise linear approximations,
are ineffective in terms of accuracy. It can be proven that
hard sigmoid, a piecewise linear approximation of the logistic sigmoid, performs worse than logistic sigmoid, especiallyin the case of linear regression (Maksutov 2018). SecureML
also provides higher-order approximations of logistic sigmoid and softmax using a combination of ReLU functions
in place of exponentiation. However, they do not provide a
detailed algorithm and find the running time in the offline
phase to be high.
To overcome this research gap, we propose S++, basedon (Wagh, Gupta, and Chandran 2018), for the 3-server setting with efficient protocols for logistic sigmoid and tanh
and their derivatives, as well as the softmax. This can be extended to the derivative of the softmax as well.

two primary servers hold additive secret shares of the data.
Let P0 , P1 , P2 be the three servers and C be a set of n participants {C1 , C2 , . . . Cn }, where the ith party Ci holds its
own private dataset, Di . Mnn is the neural network model
to be trained on the parties’ private data. The participants{C1 , C2 , . . . Cn } split their data into 2-out-of-2 additive secret shares and give one share to each of the two servers P0
and P1 . The third server, P2 , similar to (Wagh, Gupta, andChandran 2018), is crucial to the protocols, but does not holdany data. We use the same integer representation for fixed
point numbers as (Wagh, Gupta, and Chandran 2018). However, to tackle any overflow that can render the exponentiation protocol impractical for use, we also consider extending
the integer ring to Z128 . S++ provides security against one
semi-honest adversary and up to one malicious corruption.

Cryptographic Primitives
Notation
In our work, we use 2-out-of-2 additive sharing over the evenring ZL where L = 2l . For our purposes, we use the samering as (Wagh, Gupta, and Chandran 2018), i.e., Z264 . The
L
2-out-of-2 additive shares are represented by hxiL
0 and hxi1 .
where L represents the ring ZL over which the value x has
been shared.
We use the notation haij to denote Party Pj ’s additive secret share of the value a, and bxc to denote the floor of thevalue x. The notation (p0 , p1 , · · · , pn−1 ) is used to denotethe bits of a n-bit value, p.
Further, we use the following primitives from (Wagh,
Gupta, and Chandran 2018) in our work:
1. Matrix Multiplication of two secret shared matrices represented by FM atM ul ({P0 , P1 }P2 ).
2. Division of a two secret shared values represented by
FDivision ({P0 , P1 }P2 ).

Figure 1: Function Dependencies in S++.

Supporting Protocols
Matrix Multiplication (FM atM ul ({P0 , P1 }P2 )): In S++,
this function from (Wagh, Gupta, and Chandran 2018) has
been used for multiplying fixed point values (represented as
unsigned integers in the ZL ring) by considering the values
as 1 × 1 matrices.
Two parties, P0 and P1 are required to hold shares of X ∈ZLm×n and Y ∈ ZLn×v . After running this interactive secureLprotocol, P0 gets hX.Y iL
0 and P1 gets hX.Y i1 .

Architecture

Division (FDivision ({P0 , P1 }P2 )): This function, also
from (Wagh, Gupta, and Chandran 2018), is used to perform secure division on secret shared values. In this protocol, two parties, P0 and P1 are required to hold shares ofvalues xj and yj respectively, for jin{0, 1}. After runningLthe protocol, they obtain hx/yiL
0 and hx/yi1 respectively.

S++ works with the 3-server setting similar to SecureNN (Wagh, Gupta, and Chandran 2018). In this setting,there are two primary servers and one helper server. The

Taylor Series Expansion (FtaylorExp ({P0 , P1 }P2 ); see algorithm 1): In this protocol, we compute ex , where x is asecret fractional value. We do so by computing the value of

These protocols are described in the following section.

Protocols of S++

the Taylor expansion up to four terms. In the end of the protocol, parties P0 and P1 obtain shares of ex .

Algorithm 2 Exponentiation FExp ({P0 , P1 }, P2 )L
Input: P0 and P1 hold hxiL
0 and hxi1 (shares of a valuex).
Algorithm 1 Taylor Expansion, FtaylorExp ({P0 , P1 }, P2 )L
Input: P0 , P1 hold {hxiL0 } and {hxi1 } respectively
where x < 1.
x L
Output: P0 and P1 obtain hyiLj = he ij .
Common Randomness: P0 and P1 hold random sharesof zero - u0 and u1 respectively.
1: Parties P0 and P1 compute hciL
j = j.
L
2: Parties P0 and P1 compute hnumeratoriL
j = hxij anddenominator = 1.
3: Now, Parties P0 and P1 compute hciL= hciLjj +Lhnumeratorij .
4: for i = 2, . . . , 4 do
5:Parties P0 , P1 and P2 invoke FM atM ul ({P0 , P1 }P2 )Lwith inputs hnumeratoriL
j and hxij to obtainLhnumeratorij for j ∈ {0, 1}.
6:
denominator = denomiator × i
7:Parties P0 and P1 compute hciL= hciLjj +hnumeratoriLjdenominator

8: end for
9: Pj for j ∈ {0, 1} output hciL
j + uj .

Exponentiation (FExp ({P0 , P1 }, P2 )); see algorithm 2:
In this protocol, we compute ex , where x is secret. This protocol is based on SCALE MAMBA’s (Aly et al. 2020) base2 exponentiation protocol. It uses primitives like FT runc ,FBitDecomp , FP reM ult from (Catrina and Saxena 2010).
While (Aly et al. 2020) uses the polynomial P1045 (X)from (Hart 1978) to compute the exponentiation for the fractional part of a number, we use a secure protocol to computethe output using the Taylor series expansion of ex . We introduce function FtaylorExp defined above for this purpose.
This protocol first uses FT runc to split the input a into its
integer (aint ) and fractional (afrac ) parts. After that, it evaluates eaint using primitives FBitDecomp and FP reM ult described above. It further evaluates eafrac using FtaylorExp . It
then multiplies these two values to obtain eaint +afrac = ea .

Main Protocols

x L
Output: P0 and P1 obtain hyiLj = he ij .
1: For j ∈ {0, 1}, party Pj calls FT runc ({P0 , P1 }) to obth
tain haiLj , which is the j share of bxc.
2: For j ∈ {0, 1}, party Pj gets the fractional part of x by
locally computing:LLhbiL
j = hxij − haij
3: For j ∈ {0, 1} party Pj invokes FBitDecomp ({P0 , P1 })L
L
to obtain (hc0 iLj , hc1 ij , · · · , hcm−1 ij ): shares of(x0 , x1 , · · · , xm−1 ), where x is a m-bit number.
4: for i = 0, 1, . . . , m − 1 do
2iL
5:
Pj for j ∈ {0, 1} computes hvi iL
j = e .(hci ij ) +L
j − (hci ij ).
6: end for
7: Pj for j ∈ {0, 1} invokes FP reM ult ({P0 , P1 }) to gethmiLj.
8: Pj for j ∈ {0, 1} invokes FT aylor ({P0 , P1 }) to get
hniLj
9: Finally,
Pj for j∈{0, 1} invokes
LFM atM ul ({P0 , P1 }P2 ) with inputs hmiLj and hnij toLobtain share hmij of:
y = m × n.

Algorithm 3 Sigmoid, FSigmoid ({P0 , P1 }, P2 )L
Input: P0 , P1 hold {hxiL0 } and {hxi1 } respectively.
L
Output: P0 , P1 get {hσ(x)i0 } and {hσ(x)iL1 } respecex
tively where σ(x) = 1+ex.
Common Randomness: P0 and P1 hold random sharesof zero - u0 and u1 respectively.
1: Parties P0 , P1 and P2 invoke FExp ({P0 , P1 }, P2 ) withLL
x L
inputs hxiL
0 and hxi1 and obtain haij = he ij .
L
2: Now, parties P0 and P1 compute hbij = haiLj + j.
3: P0 , P1 and P2 invoke FDivision ({P0 , P1 }, P2 ) with inL
Lputs haiLj and hbij to obtain hcij for j ∈ {0, 1}.
L
4: Pj for j ∈ {0, 1} output hcij + uj .

Our main protocols include logistic sigmoid (FSigmoid ; described in algorithm 3), tanh (FT anh ; described in algorithm 5), and their derivatives; see algorithms 4 and 6, as
well as the softmax function (FSof tmax ; described in algorithm 7).
Sigmoid, FSigmoid ({P0 , P1 }, P2 ); see algorithm 3):
This protocol can be used to securely compute logistic sigmoid. Parties P0 , P1 and P2 invoke FExp with shares of the
input x to obtain output shares of ex . They then jointly compute shares of 1+ex . They use (Wagh, Gupta, and Chandranex2018)’s secure division protocol to obtain σ(x) = 1+ex.

Derivative of Sigmoid, F D Sigmoid ({P0 , P1 }, P2 ); see algorithm 4): This protocol can be used to securely compute the derivative of the logistic sigmoid. We use the idea0that dS(x)d(x) = σ (x) = σ(x) · (1 − σ(x)). In the end of the
protocol, parties P0 and P1 obtain the shares of σ 0 (x).

Algorithm 4 Derivative of Sigmoid hF D sigmoid (x)iL
Input: P0 , P1 have inputs hxiL
j for Pj such that j ∈{0, 1}.
0L
0
Output: P0 , P1 get hσ 0 (c)iL
0 and hσ (c)i1 where σ (c) =σ(x)(1 − σ(x))
Common Randomness: P0 and P1 hold random sharesof zero - u0 and u1 respectively.
1: P0 , P1 and P2 invoke FSigmoid ({P0 , P1 }, P2 ) with Pj
Lfor j ∈ {0, 1} having input hxiL
j to learn haij =Lhσ(x)ij .
L
2: Pj computes haiLj = −1 × hxij + j for j ∈ {0, 1}
LL
3: Pj computes hbij = haij for j ∈ {0, 1}
4: P0 , P1 and P2 invoke FM atM ul ({P0 , P1 }, P2 ) with Pj
Lfor j ∈ {0, 1} having inputs haiLj and hbij to learn
LL
hciL
j = haij × hbij .
5: Pj for j ∈ {0, 1} output:
LhciL
j + huij

Algorithm 6 Derivative of Tanh hF D T anh (x)iL
Input: P0 , P1 have inputs hxiL
j for Pj such that j ∈{0, 1}.
0L
Output: P0 and P1 get htanh0 (c)iL
0 and htanh (c)i1 respectively.
Common Randomness: P0 and P1 hold random sharesof zero - u0 and u1 respectively.
L
1: P0 and P1 compute haiLj = 2 × hxij .
2: P0 , P1 , P2 invoke F D sigmoid with haiLj to obtain shareshbiL.
j
3: P0 and P1 output:
L
4.hbiLj + huijTanh, FT anh ({P0 , P1 }, P2 ); see algorithm 5): This protocol can be used to securely compute the tanh function. For
this protocol, we use the idea that tanh(x) is related to σ(x)as tanh(x) = 2σ(2x) − 1.
Algorithm 5 Tanh hFT anh (x)iL
Input: P0 , P1 have inputs hxiL
j for Pj such that j ∈{0, 1}.
L
Output: P0 , P1 get htanh(c)iL
0 and htanh(c)i1 wheretanh(c) = 1+e2−2x − 1
Common Randomness: P0 and P1 hold random sharesof zero - u0 and u1 respectively.
L
1: Pj computes haiLj = 2 × hxij for j ∈ {0, 1}
2: P0 , P1 andP2 invoke FSigmoid ({P0 , P1 }, P2 ) with Pj
Lfor j ∈ {0, 1} having input haiLj to learn hpij =hσ(a)iLj.
3: Pj for j ∈ {0, 1} output:
L
2 × hP iLj − j + huij
Derivative of Tanh, F D T anh ({P0 , P1 }, P2 ); see algorithm 6): This protocol can be used to securely computethe derivative of the tanh function. For this protocol, we use
the idea that tanh0 (x) is related to σ 0 (x) as tanh0 (x) =4σ 0 (2x).
Softmax, FSof tmax ({P0 , P1 }, P2 ); see algorithm 6):
This protocol can be used to securely compute the softmax
function. The two parties compute the numerator ezi and thePkdenominator i=1 ezi and invoke the FDivision function to
perform secure division.

Empirical Evaluation
Implementation and Experimental Setup
Our system is implemented in C++ using standard libraries.
To be able to execute machine learning protocols, we use

Algorithm 7 Softmax, FSof tmax ({P0 , P1 }, P2 )L
Input: P0 , P1 hold {hzi iL0 }i∈[k] and {hzi i1 }i∈[k] respectively.
Output: P0 , P1 get {hsmax (zi )iL
and0 }i∈[k]L{hsmax (zi )i1 }i∈[k] respectively where smax (zi )
=
zi
Pke
.
zi
i=1 e
Common Randomness: P0 and P1 hold random sharesof zero - u0 and u1 respectively.
1: for i = 1, 2, . . . , k do
2:
Parties P0 , P1 and P2 invoke FExp ({P0 , P1 }, P2 )Lwith inputs {hzi iL0 }i∈[k] and {hzi i1 }i∈[k] and P0 , P1
LLziobtain shares hci i0 , hci i1 resp. of cLi =e .
3: end for
Pk
4: for j ∈ {0, 1}, Pj calculates Sj = i=1 hci iLj.
5: for i = 1, 2, . . . , k do
6:
Parties
P0 ,
P1
and
P2
invokeFDivision ({P0 , P1 }, P2 ) with inputs hci iL
andjLhSij .
7:
Parties Pj for j ∈ {0, 1} output {hmax (zi )iLj }i∈[k] +uj .
8: end forfixed-point representation in the integer ring Z64 . This allows us to compute precise values using our machine learning protocols. The fixed-arithmetic used in our implementation is the same as (Wagh, Gupta, and Chandran 2018) which
borrowed the idea from (Mohassel and Zhang 2017).
Exponentiation is complicated and may overflow the integer ring. So, we consider the ring Z128 as proposed by (Alyet al. 2020). So far, we have implemented the Taylor series
exponentiation (refer to step 7 of Algorithm-1) which works
for values less than 1. We provide benchmarks for our logistic sigmoid, tanh and softmax functions, their derivatives andthe Taylor series exponentiation in the secure setting with input vectors of varying sizes. These benchmarks include theexecution time and the total communication (in MB) of these
protocols.
We run our protocols in a LAN setting on an Intel i5 7th
Gen with 8GB RAM. The average bandwidth measured was
40Mb/s for download and 9.2Mb/s for upload and the average ping was 12ms.

Experimental Results
We show the average time taken by the protocols in Table 1.
ProtocolFExp
FSigmoid
Ftanh
FSof tmax
F D sigmoid
F D tanh
FtaylorExp

Dimension
64x16
128x128576x20
64x16
128x128576x20
64x16
128x128576x20
64x16
128x128576x20
64x16
128x128576x20
64x16
128x128576x20
64x16
128x128576x20Time(s)
0.08
2.134
0.882
0.252
5.631
2.615
0.275
5.32
2.613
0.324
5.438
2.617
0.464
8.033
4.121
0.383
4.465
2.84
0.032
0.092
0.427

Comm.(mb)
0.025
0.393
0.276
2.58
41.28829.03
2.58
41.28829.03
2.58
41.288
29.03
2.597
41.55
29.2142.58
41.288
29.03
0.005
0.079
0.055

Table 1: Benchmarks in LAN setting on i5 7th Gen

Conclusion and Future Work
Previous work in secure computation of neural networks
has been able to achieve reasonable efficiency for many
real world applications. However, the lack of options in the
choice of protocols that these implementations can run canlimit the capacity of activities the user is able to perform inthe secure setting.
Through the addition of the exponentiation function, it becomes possible to add a large number of secure activation

functions that were previously impossible to compute. We
have shown logistic sigmoid, tanh, softmax and their derivatives as examples of such functions. We implement thesefunctions with similar overheads as the previously proposed
activation functions.
In its current form, the exponentiation function overflows
for small values making it impractical for usage in regularneural network. This also makes us unable to use the functions that call the exponentiation function, as part of their
execution, in practical settings.
Exponentiation protocols in a fixed-point MPC setting
have been provided by (Aly and Smart 2019) and implemented by (Aly et al. 2019). Ideas from these protocols canbe used to avoid using a workaround that would involve increasing the size of shares to accommodate the overflow.
The future work on S++ will be focused on dealing withthe overflow such that the exponentiation function can handle the magnitude of the values that it would encounter in a
regular neural network. Once this is solved, the other functions that we have proposed, that use exponentiation, willautomatically become suitable for practical usage.

References
Agrawal, R.; and Srikant, R. 2000. Privacy-preserving data
mining. In Proceedings of the 2000 ACM SIGMOD international conference on Management of data, 439–450.
Aly, A.; Cong, K.; Cozzo, D.; Keller, M.; Orsini, E.; Rotaru,
D.; Scherer, O.; Scholl, P.; Smart, N.; Tanguy, T.; et al. 2020.
SCALE–MAMBA v1. 10: Documentation .
Aly, A.; Orsini, E.; Rotaru, D.; Smart, N. P.; and Wood, T.
2019. Zaphod: Efficiently combining lsss and garbled circuits in scale. In Proceedings of the 7th ACM Workshop on
Encrypted Computing & Applied Homomorphic Cryptography, 33–44.
Aly, A.; and Smart, N. P. 2019. Benchmarking privacy preserving scientific operations. In International Conference
on Applied Cryptography and Network Security, 509–529.
Springer.
Asadi, B.; and Jiang, H. 2020. On Approximation Capabilities of ReLU Activation and Softmax Output Layer in Neural
Networks. arXiv preprint arXiv:2002.04060 .
Bunn, P.; and Ostrovsky, R. 2007. Secure two-party k-meansclustering. In Proceedings of the 14th ACM conference on
Computer and communications security, 486–497.
Catrina, O.; and Saxena, A. 2010. Secure Computation with
Fixed-Point Numbers. In International Conference on Financial Cryptography and Data Security. Springer.
Du, W.; and Atallah, M. J. 2001. Privacy-preserving cooperative scientific computations. In csfw, 0273. Citeseer.
Hart, J. F. 1978. Computer Approximations. In Krieger Publishing Co., Inc. Springer.
Hunt, T.; Song, C.; Shokri, R.; Shmatikov, V.; and Witchel,E. 2018. Chiron: Privacy-preserving machine learning as aservice. arXiv preprint arXiv:1803.05961 .
Jagannathan, G.; and Wright, R. N. 2005. Privacy-preserving
distributed k-means clustering over arbitrarily partitioned
data. In Proceedings of the eleventh ACM SIGKDD international conference on Knowledge discovery in data mining,
593–599.

Juvekar, C.; Vaikuntanathan, V.; and Chandrakasan, A. 2018.
{GAZELLE}: A low latency framework for secure neuralnetwork inference. In 27th {USENIX} Security Symposium({USENIX} Security 18), 1651–1669.
Kent, D.; and Salem, F. 2019. Performance of three slim
variants of the long short-term memory (LSTM) layer. In
2019 IEEE 62nd International Midwest Symposium on Circuits and Systems (MWSCAS), 307–310. IEEE.
Krizhevsky, A.; Sutskever, I.; and Hinton, G. E. 2017. Imagenet classification with deep convolutional neural networks.
Communications of the ACM 60(6): 84–90.
Le, Q. V.; Jaitly, N.; and Hinton, G. E. 2015. A simple way
to initialize recurrent networks of rectified linear units. arXiv
preprint arXiv:1504.00941 .
Lindell, Y.; and Pinkas, B. 2000. Privacy preserving data mining. In Annual International Cryptology Conference, 36–54.
Springer.
Liu, J.; Juuti, M.; Lu, Y.; and Asokan, N. 2017. Oblivious
neural network predictions via minionn transformations. In
Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, 619–631.
Maksutov, R. 2018.
Deep study of a not very deepneural network. Part 2: Activation functions.
https://towardsdatascience.com/deep-study-of-a-not-very-deepneural-network-part-2-activation-functions-fd9bd8d406fc.
Accessed: 2020-11-06.
Mohassel, P.; and Zhang, Y. 2017. SecureML: A system forscalable privacy-preserving machine learning. In 2017 IEEE
Symposium on Security and Privacy (SP), 19–38. IEEE.
Ohrimenko, O.; Schuster, F.; Fournet, C.; Mehta, A.;
Nowozin, S.; Vaswani, K.; and Costa, M. 2016. Oblivious multi-party machine learning on trusted processors. In
25th {USENIX} Security Symposium ({USENIX} Security
16), 619–636.
Pascanu, R.; Mikolov, T.; and Bengio, Y. 2013. On the difficulty of training recurrent neural networks. In International
conference on machine learning, 1310–1318.
Patra, A.; and Suresh, A. 2020. BLAZE: Blazing Fast
Privacy-Preserving Machine Learning.
arXiv preprint
arXiv:2005.09042 .
Riazi, M. S.; Weinert, C.; Tkachenko, O.; Songhori, E. M.;
Schneider, T.; and Koushanfar, F. 2018. Chameleon: A hybrid secure computation framework for machine learning applications. In Proceedings of the 2018 on Asia Conference on
Computer and Communications Security, 707–721.
Sanil, A. P.; Karr, A. F.; Lin, X.; and Reiter, J. P. 2004. Privacy
preserving regression modelling via distributed computation.
In Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining, 677–682.
Slavkovic, A. B.; Nardi, Y.; and Tibbits, M. M. 2007. ”
Secure” Logistic Regression of Horizontally and VerticallyPartitioned Distributed Databases. In Seventh IEEE International Conference on Data Mining Workshops (ICDMW
2007), 723–728. IEEE.
Vaidya, J.; Yu, H.; and Jiang, X. 2008. Privacy-preserving
SVM classification. Knowledge and Information Systems
14(2): 161–178.
Wagh, S.; Gupta, D.; and Chandran, N. 2018. SecureNN:
Efficient and Private Neural Network Training. IACR Cryptology ePrint Archive 2018: 442.

Xu, B.; Huang, R.; and Li, M. 2016. Revise saturated activation functions. arXiv preprint arXiv:1602.05980 .
Yu, H.; Vaidya, J.; and Jiang, X. 2006. Privacy-preserving
svm classification on vertically partitioned data. In Pacificasia conference on knowledge discovery and data mining,
647–656. Springer.

Appendix
Universal approximation of softmax: (Asadi and Jiang
2020) provide theoretical proofs on the universal approximation of ReLU used in hidden layers of deep networks andsoftmax used in the output layer for multiclass classification
tasks. They find that such a proof for the softmax is missingin the literature despite it being widely used for these tasks,and provide their own to prove the power of the softmax
used in output layers to approximate indicator functions.
ReLU and gradients: (Pascanu, Mikolov, and Bengio2013) delve into the exploding and vanishing gradients ofactivation functions that tend to cause problems in recurrent
neural networks because of the temporal correlations theycapture (by forming connections between hidden layers).
Long-term components add up and the norm of the gradientof the hidden layers explode. ReLU’s unbounded nature
exacerbates this problem. There have been a few notable
improvements to assuage this, namely gradient clipping,identity initialization (Le, Jaitly, and Hinton 2015), andgated RNNs like LSTMs and GRUs. However, even in
LSTMs, the tanh often gives more consistent results ((Kentand Salem 2019)) and ReLU tends to require greater
fine-tuning.
The Leaky Tanh: (Xu, Huang, and Li 2016) propose a‘leaky’ tanh that penalizes the negative part i.e., instead of
the regular tanh function, the penalized tanh behaves likethis:
tanh(x)x>0(1)
penalized tanh(x) =a.tanh(x) otherwise
where a ∈ (0, 1). They also found this variation to be twotimes faster than the tanh.