## 5 Experiments and Evaluation
### Performance Improvements  
Table 5 presents the input of the classification layer and the initial values of the hidden and cell states in the proposed BiLSTM model. In model 1, initial hidden and cell states of BiLSTM were set to 0. For the input of the classification layer, the value obtained by concatenating the last hidden state of BiLSTM was used as in \(\left[\frac{}{Hidden(n)};\longleftrightarrow\frac{}{Hidden(n)}\right]\). Similar to model 1, the initial hidden and cell states were set to 0 in model 2. Since a significant performance improvement was observed in the model that only used a [CLS] token, the last hidden state of BiLSTM was concatenated with a [CLS] token as the input of the classification layer as in \(\left[\text{CLS};\ \xrightarrow{}\xrightarrow{}\xrightarrow{}\xrightarrow{} \xrightarrow{  
Compared to model 1, which is proposed in this paper, model 2 demonstrated approximately a 1.2% performance improvement in test 1-1 and test 3. Both models resulted in similar levels of accuracy in test 1-2 and test 2, for which transfer learning was not performed. When model 1 and model 2 are compared, concatenating a [CLS] token with the last hidden state of the BiLSTM can improve the performance. However, the speed was lower than the proposed model during training since the input size of the classification layer was enlarged by 1.5 times. Similar to model 2, model 3 also demonstrated approximately a 1.2% performance improvement in test 1-1 and test 3. A similar level of accuracy to model 2 was observed in test 1-2 and test 2, for which transfer learning was not performed. However, the training speed was faster than in model 2 since the input size of the classification layer was identical to that of model 1. Model 4 shows the accuracy when both the hidden and cell states are set to a [CLS], which is the essential idea of the BiLSTM. The accuracy was improved by 4% to 78.52% in test 3, which was the highest when compared to the proposed model, and the performance was improved by 8% compared to the linear model. A similar level of accuracy improvement was observed in test 1-1 as well. The accuracy was 2.8% higher than with model 3, which implies that using a [CLS] token as the initial cell state of BiLSTM can drastically improve the performance of the model. Conducting the above experiments demonstrates that using a [CLS] containing sentence information is highly beneficial for improving the performance since the cell of BiLSTM plays the role of a memory for recalling previous values. Additionally, we conducted an experiment to order to demonstrate the increasing performance in proportion to the size of the training datasets in Figure 7. The \(x\)-axis and the \(y\)-axis show the number of COVID-19 training datasets required for further learning after transfer learning and the accuracy of Model 4, respectively. In fact, this is very common in the field of machine learning, but in transfer learning with a small number of datasets, the increase takes effect without exception. \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Model** & **Test 1-1** & **Test 1-2** & **Test 2** & **Test 3** \\ \hline Model 1 & 60.54 & 50.78 & 52.21 & 74.51 \\ \hline Model 2 & 61.72 & 49.22 & 52.0 & 75.78 \\ \hline Model 3 & 61.7 & 53.52 & 53.12 & 75.61 \\ \hline
**Model 4** & **64.2** & **55.86** & **55.86** & **78.80** \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Performance of the proposed and comparative models.