# A Bi-LSTM Based Ensemble Algorithm for Prediction of Protein Secondary Structure
## 3 Features and Methods
### The Ensemble Algorithm Based on Bi-LSTM  
Compared to methods using a single model, ensemble methods have gradually been used for various tasks and achieve satisfactory performance in bioinformatics. For example, [PERSON] [25] proposed an ensemble method that combines four predictors and uses a random forest classifier for the prediction of human sub-cellular localization. Four predictors, based either on support vector machine [26] or naive Bayes [27], employed different features of protein to make predictions. Inspired by their work, we propose an ensemble algorithm for the sequence-based prediction of protein secondary structure. Five types of feature were extracted to represent a protein sequence, and these features describe protein sequences according to individual amino acid property, protein homology and mathematical encoding. Naturally, five sub-models were used to compose the ensemble model and one type of feature is put into one of the five sub-models. As shown in Figure 1, five sub-models (pssm_model, hmm_model, psssm_count_model, pps_model, wordem_model) compose the ensemble model. Each sub-model contains two bi-directional LSTM layers, followed by two more fully connected layers with 512 and 128 neurons, respectively. The fully-connected neurons in each network use rectified linear unit (ReLU) as activations. Note that since the sizes of these five types of features are different among the five sub-models, the numbers of neurons in bi-directional LSTM layers are different. There are 100, 128, 100, 64 and 128 neurons per direction in bi-directional LSTM layersfor the ppssm_model, hmm_model, psssm_count_model, pps_model and wordem_model, respectively. Besides, the embedding size of wordem_model was set as 50 [24]. These sub-models were joined by a bi-directional LSTM layer with 64 neurons per direction followed by two fully connected layers with 128 and 64 neurons, respectively, to form the ensemble model. In Figure 1, there are 7, 20, 20, 30 and 50 features fed as the input for the pps_model, psssm_model, psssm_count_model, hmm_model, and wordem_model. There were 640 (\(128\times 5\)) features finally fed the input for the final Bi-LSTM layer. Note that each number in Figure 1 denotes the neuron number for that layer. To reduce the network over-fitting, we utilize the dropout trick [28] with a dropout ratio of 70% in fully connected layers during the training. As the ensemble model has many neurons, we need a higher dropout ratio to prevent over-fitting. Adam optimization [29] with the initial learning rate of 0.0005 was used to train our model, which is an efficient stochastic optimization able to compute the adaptive learning rates during the training process. A 10-fold cross validation and early stopping was applied during the training to prevent from the over-fitting. Note that the early stopping was only applied to the ensemble model, so when the ensemble model stopped training, all sub-models stopped training too. The clip gradients trick [30] was also applied to prevent gradients from being exploded, which usually occurs in the LSTM-based method. Additionally, we used two Tesla K80 GPUs with 64G memory to speed up the training. The number of neurons of each sub-model was selected through the different experiments. When choosing these settings, we found that we could obtain satisfying prediction results by our ensemble method. As mentioned above, one type of feature is entered into one of the five sub-models and the outputs from the last fully connected layer in the network of each sub-model are combined by a bi-directional LSTM layer. A soft-max function is applied to the last fully connected layer of both sub-models and ensemble model to get normalized probabilities. Note that this ensemble model contains dual loss functions; specifically, one for the sub-models and another for the ensemble model.