# Research on Public Service Request Text Classification Based on BERT-BiLSTM-CNN Feature Fusion
## 4 Experiments and Results
### Experimental Comparison and Result Analysis
#### 4.3.2 Comparison of the Classification Effects of Different Parameters  
In the fusion model BBLC, the hyperparameter setting is essential. The different hyperparameter settings of each model result in different classification effects. For example, different Hidden_sizes in BiLSTM lead to different complexity of model learning sequence features, thus affecting the ability to model sequence data. The various sizes of convolutional kernels in CNN will impact the model's capacity to extract features from text segments of diverse lengths. To further enhance the performance of the BBLC, experiments involving the size of convolutional kernels and the dimensions of Hidden_sizes were conducted.  
Table 8 presents the scenario where the convolutional kernel size varies while other parameters remain constant. Table 8 shows that as the convolutional kernel size increases, the model's recall and F1 score also improve. When the convolutional kernel is too large, the model tends to overfit. Overfitting occurs when the model becomes overly complex and excessively adapts to the characteristics of the training data, leading to a loss of generalization ability. So, it is important to choose the appropriate convolution kernel size; Table 9 shows the dimensions of different hidden layers when other parameters are the same. Too small a hidden layer will lead to poor performance of the model on the test set, inadequate dimensionality of the hidden layer will result in subpar performance of the model on the test data set, and an excessively large hidden layer can lead to overfitting. Therefore, after a series of comparative experiments, Num_layers = 1, Kernel_sizes = [7,8,9], and Hidden_sizes = 768 were selected for hyperparameter training.  
\begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Model** & **Accuracy** & **Precision** & **Recall** & **F1-Score** \\ \hline Word2 Vec-CNN [5] & 0.9387 & 0.9410 & 0.9340 & 0.9353 \\ BERT-CNN & **0.9509** & **0.9354** & **0.9503** & **0.9395** \\ \hline Word2 Vec-Attention-CNN & 0.9335 & 0.9378 & 0.9237 & 0.9282 \\ BERT-Attention-CNN & **0.9446** & **0.9440** & **0.9287** & **0.9335** \\ \hline Word2 Vec-BiLSTM-Attention-CNN [25] & 0.9444 & 0.9356 & 0.9351 & 0.9325 \\ BERT-BiLSTM-Attention-CNN & **0.9505** & **0.9528** & **0.9457** & **0.9470** \\ \hline Word2 Vec-BiLSTM-CNN [17] & 0.9413 & 0.9407 & 0.9415 & 0.9389 \\
**BERT-BiLSTM-CNN** & **0.9536** & **0.9541** & **0.9519** & **0.9509** \\ \hline \hline \end{tabular}
\end{table}
Table 7: Comparison of models based on BERT and Word2 Vec.  
\begin{table}
\begin{tabular}{c c c c c c c} \hline \hline  & **Model** & \multicolumn{5}{c}{**BERT-BiLSTM-CNN**} \\ \hline
**Num\_Layers** & **Kernel\_Sizes** & **Hidden\_Sizes** & **Accuracy** & **Precision** & **Recall** & **F1-Score** \\ \hline
1 & [2,3,4] & 768 & 0.9461 & 0.9450 & 0.9454 & 0.9431 \\
1 & [3,4,5] & 768 & 0.9491 & 0.9510 & 0.9414 & 0.9442 \\
1 & [4,5,6] & 768 & 0.9467 & 0.9360 & 0.9461 & 0.9378 \\
1 & [5,6,7] & 768 & 0.9377 & 0.9377 & 0.9485 & 0.9392 \\
1 & [6,7,8] & 768 & 0.9529 & 0.9521 & 0.9495 & 0.9490 \\
1 & [7,8,9] & 768 & **0.9536** & **0.9541** & 0.9519 & **0.9509** \\
1 & [8,9,10] & 768 & 0.9464 & 0.9459 & 0.9474 & 0.9437 \\
1 & [9,10,11] & 768 & 0.9527 & 0.9532 & 0.9495 & 0.9491 \\
1 & [10,11,12] & 768 & 0.9519 & 0.9509 & **0.9563** & 0.9487 \\ \hline \hline \end{tabular}
\end{table}
Table 8: Experimental comparison of convolution kernel sizes.  
\begin{table}
\begin{tabular}{c c c c c c} \hline \hline  & **Model** & \multicolumn{5}{c}{**BERT-BiLSTM-CNN**} \\ \hline
**Num\_Layers** & **Kernel\_Sizes** & **Hidden\_Sizes** & **Accuracy** & **Precision** & **Recall** & **F1-Score** \\ \hline
1 & [7,8,9] & 128 & 0.9379 & 0.9440 & 0.9337 & 0.9353 \\
1 & [7,8,9] & 256 & 0.9483 & 0.9530 & 0.9434 & 0.9459 \\
1 & [7,8,9] & 512 & 0.9521 & 0.9528 & 0.9469 & 0.9480 \\
1 & [7,8,9] & 768 & **0.9536** & **0.9541** & **0.9519** & **0.9509** \\ \hline \hline \end{tabular}
\end{table}
Table 9: Experimental comparison of Hidden sizes.