## 2 Proposed Ensemble Model for 8-Class BC Image Classification  
### Working Principles of Deep Neural Networks Learning Algorithms  
There are many types of deep learning, different kinds of auto-encoder, that vary in architectures and training algorithms. However, one basic element of deep learning is the additional new activation functions such as rectified linear unit (ReLU), SoftMax, and Swish. These are identified to be useful to train deeper networks [43]. Compared to many classical bounded activation functions such as \(tanh(x)\) and \(\delta(x)\), many of the new activation functions are convex with a large area of non-zeros derivatives [44]. Figure 1 illustrates attributes of new and classical activation functions. ReLU was proposed to replace \(tanh(x)\) and \(\delta(x)\), which are known to be standard ways to model a _neuron's_ output \(f\) as a function of its input \(x\). ReLU is non-saturating nonlinearity:  
\[h(x)=max(0,\ x) \tag{1}\]  
ReLU is a preferred choice of activation function for hidden layers, as it is faster and more efficient for deep convolutional neural networks compared to their equivalents with \(tanh(x)\)units [44]. Leak ReLU was proposed as an improvement of the ReLU activation function. ReLU sometimes destroys some neurons in each iteration, a condition known as the dying ReLU condition:  
\[h(x)= \begin{cases}x,&\text{if }x>0\\ 0.01x,&\text{otherwise}\end{cases}  \tag{2}\]  
Figure 1: New and classic activation functions. A more generalized variant of ReLU is parameterized ReLU:  
\[h(x)= \begin{cases}x,&if\ x>0\\ ax,&otherwise\end{cases}  \tag{3}\]  
where \(a\) is a trainable parameter with the value of \(a=0.01\); parameterized ReLU acts as Leaky ReLU. SoftMax is a generalization of logistic regression. It is usually used in a final output layer to handle a case of a multi-class problem. Recall that the logistic regression learning model is a sigmoid function:  
\[h_{\theta}(x)=\delta(x)=\frac{1}{1+e^{-\theta^{T}}X} \tag{4}\]  
In logistic regression the output \(y^{i}\) for an input instance \(i\) is assumed to belong to binary set, i.e., \(y^{i}\in\{0,\ 1\}\). With SoftMax \(y^{i}\) belong to multivariate set, i.e., \(y^{i}\in\{1,\dots K\}\) where \(K\) is the number of target classes. Recall that the cost function for the logistic regression model is:  
\[J(\theta)=-\frac{1}{m}[\sum_{i=1}^{m}y^{i}log(h_{\theta}(x^{i}))\ +(1-y^{i}) log(1-h_{\theta}(x^{i}))]+\frac{\lambda}{2m}\sum_{j=1}^{n}\theta_{j}^{2} \tag{5}\]  
In SoftMax, the interest is on multi-class classification rather than binary classification as in the case of logistic regression. Thus, given a test input \(x\), the learning model estimates the probability of \(p(y=k/x)\) for each value of \(k=1,\dots,K\). Hence, the learning model will output a K- dimensional vector for which its element is summed up to 1. The SoftMax function for multi-class prediction is of the form:  
\[h_{\theta}(x)=\delta(x)=\frac{1}{1+e^{-(\theta^{\prime})^{T}}X} \tag{6}\]  
Equation (6) predicts the probability of one of the classes, and \((1-\frac{1}{1+e^{-(\theta^{\prime})^{T}}X})\) for the other class; where \(\theta^{\prime}\) is a single parameter vector. Equation (7) below describes the cost function for SoftMax regression:  
\[J(\theta)=-\left[\sum_{i=1}^{m}\sum_{k=1}^{K}\ 1\{y^{i}\ =k\}log\frac{exp( \theta^{(k)T}x^{i})}{\sum\limits_{j=1}^{K}exp(\theta^{(j)T}x^{i})}\right] \tag{7}\]  
where \(1\{.\}\) is an indicator function. i.e., \(1\{\textit{a}\textit{t}\textit{r}\textit  
This natural approach of aggregating a statistical summary of these features at the various location is referred to as pooling and sometimes means \"pooling\" or max \"pooling\" pending the pooling operation used. Formally, the pooling operation involves dividing the array of convolved features into disjoint \(m\times n\) regions and applying to mean (or maximum) feature activation over these regions to generate pooled convolved features. Finally, the pooled convolved features with many lower dimensions are used for classification.