# Privacy-Preserving Outsourced Artificial Neural Network Training for Secure Image Classification
## 3 Preliminaries  
### Artificial Neural Network  
Artificial neural network [35] provides a general method for learning non-linear functions and constructing prediction model from training samples. The most common architecture of ANN is composed of the input layer, multiple hidden layers, and the output layer. ANN models have proven successful in many practical problems, such as learning to recognize handwritten characters. Figure 1 shows an ANN model for image recognition.  
**Notation.** Here, we suppose that the input data \(\mathbf{x}=(x_{1};\ldots;x_{a})\in\mathbb{R}^{a\times 1}\) has \(a\) different features and each sample belongs to one of the \(c\) classes denoted by a label \(\mathbf{y}=(y_{1},\ldots,y_{c})\), the hidden layer has \(b\) neurons.  
We represent the hidden layer weight matrix as \(W^{(h)}\in\mathbb{R}^{b\times a}\). Each row of \(W^{(h)}\) is a list of real numbers, which identifies a vector \(\mathbf{w}_{j}^{(h)}=(w_{j1}^{(h)},\ldots,w_{ja}^{(h)})\in\mathbb{R}^{1\times a}, j=1,\ldots,b\). Similarly, we denote the matrix \(W^{(o)}\in\mathbb{R}^{c\times b}\) as an output layer weight matrix. Each row of \(W^{(o)}\) identifies a vector \(\mathbf{w}_{i}^{(o)}=(w_{i1}^{(o)},\ldots,w_{ib}^{(o)})\in\mathbb{R}^{1\times b}, i=1,\ldots,c\).  
We call the last layer as the output layer  
\[\mathbf{o}=(o_{1};\ldots;o_{c})=g\big{(}W^{(o)}\mathbf{h}\big{)}\in\mathbb{R}^{c\times 1},\]  
where \(\mathbf{h}=(h_{1};\ldots;h_{b})=f(W^{(h)}\mathbf{x})\in\mathbb{R}^{b\times 1}\) consists of all outputs of hidden layer. In specific, \(f(\mathbf{w}^{(h)}\mathbf{x})\in\mathbb{R}\) is a sigmoid function \(f(z)=\frac{1}{1+e^{-z}}\), where \(\mathbf{w}^{(h)}\) is a hidden layer weight vector. The function \(g\) in ANN can be a softmax function: \(g(z_{k})=e^{z_{k}}/\sum_{i}e^{z_{i}}\). Note that in our privacy-preserving image classifier construction scheme based on ANN any activation function can substitute sigmoid function \(f\) or softmax function \(g\).  
**Gradient descent algorithm.** Considering cross entropy as the loss function,  
\[E(w)=-\sum_{i=1}^{c}y_{i}\ln(o_{i}), \tag{1}\]  
where \(y_{i}\) is the class label, \(o_{i}\) is the output of prediction function, \(w\) denotes the model parameter.  
Gradient descent search aims to determine a weight \(w^{*}\) that minimizes \(E(w)\) by starting with an arbitrary initial weight. Then, the weight is altered in the direction that produces the steepest descent. The training rule for gradient descent is  
\[w_{t+1}\gets w_{t}-\eta\
abla E(w_{t}) \tag{2}\]  
where \(\eta\) is the learning rate, and \(\
abla E(w_{t})\) is the gradient computed at the \(t\)th iteration. This process continues until the global minimum is reached.  
**Derivation of the gradient descent rule.** By selecting a mini-batch sample set \(S\) randomly from the whole database \(\mathcal{DB}=\{(\mathbf{x}^{(1)},\mathbf{y}^{(1)}),\ldots,(\mathbf{x}^{(m)},\mathbf{y}^{(m)})\}\), the gradient with respect to the output layer weight \(w_{i,j}^{(o)}\) in a 3-layer-perception becomes  
\[\frac{\partial E_{S}}{\partial w_{i,j}^{(o)}}=\frac{1}{|S|}\sum_{l\in S}{(o_{i }^{(l)}-y_{i}^{(l)})h_{j}^{(l)}}, \tag{3}\]  
where \(i=1,\ldots,c,j=1,\ldots,b\).  
Figure 1: Image recognition by ANN model.  
Similarly, we update the hidden layer weight \(w_{j,k}^{(h)}\) via the formula as follows  
\[\frac{\partial E_{S}}{\partial w_{j,k}^{(h)}}=\frac{1}{|S|}\sum_{l\in S }-h_{j}^{(l)}(1-h_{j}^{(l)})\sum_{i=1}^{c}\left[(y_{i}^{(l)}-o_{i}^{(l)})w_{i,j} ^{(o)}\right]x_{k}^{(l)}, \tag{4}\]  
where \(j=1,\ldots,b,k=1,\ldots,a\).