# Privacy-Preserving Outsourced Artificial Neural Network Training for Secure Image Classification
## 4 The Framework and Construction for Privacy-Preserving Image Classifier
### Description of Proposed Scheme
#### 4.4.3 Privacy-Preserving Training  
We now show the procedure of lossless ANN training for image classification by FEIP scheme and mask matrix support without interacting with users. The server \(\mathcal{S}_{\mathcal{NN}}\) initiates the protocol with a uniform \(W^{(h)}\) and \(W^{(o)}\) and sets the gradient update step \(\lambda\). Through executing the _IPPNN_ protocol, the server \(\mathcal{S}_{\mathcal{NN}}\) can update its weight matrices iteratively on the encrypted data with the help of the server \(\mathcal{S}_{\mathcal{AI}}\), and finally obtain a well-trained model. We describe training process of per iteration in the _IPPNN_ protocol below. **The Secure Computation of \(\partial E_{\mathbf{S}}\partial\mathbf{w}_{i,j}^{(\mathbf{o})}\). ** The update rule (3) with a random sample \(\mathbf{x}\) can be described as  
\[\frac{\partial E_{\{\mathbf{x}\}}}{\partial w_{i,j}^{(o)}}=\Big{(}g\Big{(}\sum_{k =1}^{b}w_{ik}^{(o)}f\Big{(}w_{k}^{(h)}\cdot\mathbf{x}\Big{)}\Big{)}-y_{i}\Big{)}f( \mathbf{w}_{j}^{(h)}\cdot\mathbf{x}) \tag{5}\]  
where \(i=1,\ldots,c\), \(j=1,\ldots,b\). From (5), we find that the update rule requires \(\mathcal{S}_{\mathcal{NN}}\) to compute the gradients of the loss function (2) which involve inner-product \(\mathbf{w}_{k}^{(h)}\cdot\mathbf{x}\), sigmoid function \(f(\cdot)\), and softmax function \(g(\cdot)\) and cannot be completed using only homomorphic additions and multiplications. We first deal with the formidable obstacle \(f\big{(}\mathbf{w}_{k}^{(h)}\cdot\mathbf{x}\big{)}\). Recall the server \(\mathcal{S}_{\mathcal{NN}}\) owns the model parameters \(w_{i,k}^{(o)}\), with the inner product \(f(\mathbf{w}_{k}^{(h)}\cdot\mathbf{x})\) and provided label \(y_{i}\), the \(\mathcal{S}_{\mathcal{NN}}\) can obtain the output of hidden layer neurons. Then the computation of \((o_{i}-y_{i})h_{l}\) takes place in plaintext. We provide details of how \(f(\mathbf{w}_{k}^{(h)}\cdot\mathbf{x})\) are computed by mask matrix structure. The \(\mathcal{S}_{\mathcal{NN}}\) first computes \[\widetilde{W}^{(h)} =(\widetilde{\mathbf{w}}_{1}^{(h)},\ldots,\widetilde{\mathbf{w}}_{b}^{(h)})\] \[=W^{(h)}A^{-1}\] \[=(\mathbf{w}_{1}^{(h)}A^{-1},\ldots,\mathbf{w}_{b}^{(h)}A^{-1}).\] After that, the \(\mathcal{S}_{\mathcal{NN}}\) sends \(\widetilde{W}^{(h)}\) to the \(\mathcal{S}_{\mathcal{AH}}\). 2. Then the \(\mathcal{S}_{\mathcal{AH}}\) takes \(\widetilde{W}^{(h)}\) and \(\widetilde{\mathbf{x}}\) as input, and return the inner-products \(\widetilde{\mathbf{w}}_{1}^{(h)}\)\(\cdot\widetilde{\mathbf{x}},\ldots,\widetilde{\mathbf{w}}_{b}^{(h)}\cdot \widetilde{\mathbf{x}}\), that is \(\mathbf{w}_{1}^{(h)}\cdot\mathbf{x},\ldots,\mathbf{w}_{b}^{(h)}\cdot\mathbf{x}\). This is because that \[\widetilde{\mathbf{w}}_{j}^{(h)}\cdot\widetilde{\mathbf{x}}=\mathbf{w}_{j}^{(h)}A^{-1}A \mathbf{x}=\mathbf{w}_{j}^{(h)}\cdot\mathbf{x}.\]  
After that, an execution process of activation function takes place. This results in the \(\mathcal{S}_{\mathcal{AH}}\) having all outputs of hidden layer neurons  
\[\mathbf{h}=(h_{1},\ldots,h_{b})=(f(\mathbf{w}_{1}^{(h)}\cdot\mathbf{x}),\ldots f(\mathbf{w}_{ b}^{(h)}\cdot\mathbf{x})) \tag{6}\]  
Then, the \(\mathcal{S}_{\mathcal{AH}}\) sends \(\mathbf{h}\) to the \(\mathcal{S}_{\mathcal{NN}}\). Once the server \(\mathcal{S}_{\mathcal{NN}}\) receives the outputs of hidden layer neurons, it prepares \(w_{ik}^{(o)}\) and \(y_{i}\) to perform the computation of \((g(\sum_{k=1}^{b}w_{ik}^{(o)}h_{k})-y_{i})h_{j}\) and obtains the gradient \(\partial E_{\{\mathbf{x}\}}/\partial w_{i,j}^{(o)}\). The gradient \(\partial E_{S}/\partial w_{i,j}^{(o)}\) in a mini-batch setting can be computed easily from a group of values \(\partial E_{\{\mathbf{x}^{(l)}\}}/\partial w_{i,j}^{(o)},l=1,...,|S|\). **The secure computation of \(\partial E_{S}/\partial w_{j,k}^{(h)}\). ** The server \(\mathcal{S}_{\mathcal{AH}}\) follows the FEIP in the process of obtaining \(\partial E_{S}/\partial w_{i,j}^{(o)}\).