# Identification of Intrinsically Disordered Protein Regions Based on Deep Neural Network-VGG16
## 2 Materials and Methods
### Designing and Training Our Deep Neural Network  
The overall architecture of our deep neural network is comprised of ten identical deep neural structures, with each of them being derived from the original VGG16 model and a MLP network. Each of these ten identical deep neural architectures shown in Figure 3 only utilizes the convolutional blocks of the original VGG16 model, i.e., erasing the fully-connected (FC) layer of VGG16, but preserving the remaining VGG16 model as well as its associated weights and bias. The outputs of these ten identical deep neural structures are then used as the inputs of our constructed MLP network for classification. The MLP network that we construct is a fully-connected MLP network with two layers, which employs the loss function defined as  
\[Loss=\frac{1}{m}\sum_{i=1}^{m}[-(\alpha_{1}y^{(i)}log(\hat{g}^{(i)})+\alpha_{2}(1 -y^{(i)})log(1-\hat{g}^{(i)}))], \tag{4}\]  
where \(m\) represents the batch size, i.e., the number of amino-acids used in each iteration during the training. \(y^{(i)}\) is a binary indicator (0 or 1) with 1, which suggests that the \(i\)-th amino-acid is disordered and 0 indicating that it is ordered. \(\hat{g}^{(i)}\) is the predicted probability of the output \(y^{(i)}=1\). Similarly, \(1-\hat{g}^{(i)}\) denotes the predicted probability of \(y^{(i)}=0\). Parameter \(\alpha_{1}\) and \(\alpha_{2}\) are the weights computed through  
\[\alpha_{1}=\frac{N_{ord}}{N_{ord}+N_{dis}}, \tag{5}\]  
\[\alpha_{2}=\frac{N_{dis}}{N_{ord}+N_{dis}}, \tag{6}\]  
where \(N_{ord}\) and \(N_{dis}\) correspond to the number of the ordered and disordered amino-acids in the training dataset, respectively. Because we obtain a \(1\times 1\times 512\) matrix from each our VGG16 output, ten \(1\times 1\times 512\) matrices are obtained from our deep neural network. Each of these \(1\times 1\times 512\) matrices are first flattened into a \(1\times 512\) vector using the flattening layer. Subsequently, the concatenation function from Keras framework [35] is utilized to concatenate these ten \(1\times 512\) vectors into a \(1\times 5120\) vector as the input to our MLP network. Each of two hidden layers of our MLP network contains 512 perceptrons and the activation functions for these two hidden layers are chosen to be the rectified linear unit (ReLU). Moreover, we employ the dropout algorithm [36], with a dropout percentage of 50% in each hidden layer of our MLP, to reduce overfitting. The output layer of our MLP network contains 2 perceptrons and it employs softmax as the activation function. The softmax activation function is defined as  
\[f(s_{p}^{(i)})=\frac{e^{s_{p}^{(i)}}}{e^{s_{1}^{(i)}}+e^{s_{2}^{(i)}}}\ \ \ \ \ (p=1,2), \tag{7}\]  
where \(s_{p}^{(i)}\) represents the \(p\)-th perceptron output value of the \(i\)-th amino-acid. The predicted probability of the output \(y^{(i)}=1\) is defined as  
\[f(s_{1}^{(i)})=\frac{e^{s_{1}^{(i)}}}{e^{s_{1}^{(i)}}+e^{s_{2}^{(i)}}}=\hat{g} ^{(i)}. \tag{8}\]  
Figure 2: The conversion flow. Similarly, the predicted probability of the output \(y^{(i)}=0\) is defined as  
\[f(\mathrm{s}_{2}^{(i)})=\frac{e^{\hat{s}_{2}^{(i)}}}{e^{\hat{s}_{1}^{(i)}}+e^{ \hat{s}_{2}^{(i)}}}=1-\hat{y}^{(i)}. \tag{9}\]  
We randomly subdivide the amino-acids of our training dataset DIS1450 into disjoint batches with each containing 128 amino-acids. The training process proceeds, as follows: for each amino-acid in a given batch, we use our deep neural network to compute the predicted probabilities that are defined in (8) and (9). Once we compute all of the predicted probabilities in this given batch, we then utilize Equation (4) to estimate the average loss of this batch. This computed average loss of the batch is employed to update the weights and bias of our MLP network through the Adam algorithm [37], where the initial learning rate of 0.0005 is chosen to perform the back propagation and updating. The above process is repeated for each batch until all of the batches have been exhausted, which is referred as an epoch.