## 4 Experimental Results
### Results  
#### 4.2.1 Effectiveness  
This work is compared with BSL1 and BSL2 with regard to the accuracy and efficiency of the abstraction model. (1) Accuracy of the abstraction model. The accuracy is the proportion of the number of samples on the training set, and they are the samples of which the RNN outputs approximate to its PFA. They can be calculated by \(acc=\frac{\sum_{x\in\mathcal{D}_{T}}sign(\mathcal{A}(x)-R(x))}{|\mathcal{D}_{T}|}\), where \(\mathcal{D}_{T}\) represents the samples in training datasets and \(|\mathcal{D}_{T}|\) is its total number of samples. \(\mathcal{A}\) is the learned model, \(R\) is the target RNN, and \(T\) is the testing set of sentences. \(sign(\mathcal{A}(x)-R(x))\) indicates whether the RNN and PFA perform approximating behavior on sample \(x\). There are three datasets in this work, among which the accuracy of Tomita syntax was taken as the mean of the three grammar rules. Classifiers are trained for different tasks on two types of RNNs; so, we create six models for these datasets. In order to scientifically and systematically compare with BSL1 and BSL2, the \(k\)-means algorithm is used in all experiments for state partitioning. The number of clusters is fixed to 18. Figure 8 shows a comparison of accuracy for six models under three abstracting techniques. It can be observed that the models learned by PSO4 RNN are significantly more accurate than those generated by BSL1 and BLS2. PSO4 RNN usually maintains an accuracy of 80% to nearly 95%, while the accuracy of BSL1 ranges from 50% to slightly above 90%. The accuracy error between the model learned by PSO4 RNN and the original RNN model fluctuates within the range of 10%. This indicates that PSO4 RNN approaches the behavior of RNNs at a satisfying accuracy. As is shown in Figure 8, the accuracy of PSO4 RNN is also slightly higher than that of BSL2. This is partly because the PD-PSO searches for a proper generalization score of PFA instead of the golden section search in BSL2. (2) Restoration degree of the abstraction model. If the behavior in PFA is consistent with the RNN, it indicates that they share similarities in testing datasets. Due to the inevitable loss of information details in abstraction, it is impossible to 100% replicate the behavior of the RNN. Therefore, the similarity must be examined after model abstraction. The corresponding metric of similarity is the model restoration degree: \(Re=\frac{\sum_{x\in T}sign(\mathcal{A}(x)-R(x))}{|\mathcal{T}|}\), where \(\mathcal{A}\) is the learned model, \(R\) is the target RNN, and \(T\) is the testing set of sentences. \(sign(\mathcal{A}(x)-R(x))\) indicates whether the RNN and PFA perform approximating behavior on sample \(x\). The restoration degree obtained from different models is shown in Table 2.  
Figure 8: Accuracy of different abstraction models on the training sets. (3) The abstraction error curve: The abstraction error is defined as the difference between the predicted results on DFA/PFA and the RNN on the training samples. During the experiment, three model abstraction techniques were trained twenty times on the DLRV dataset, and the average error was calculated for all iterations. The curve of their abstraction error is shown in Figure 9. It is an error reduction curve, as it has been normalized. According to the error curve, it can be seen that when a specific low error value is reached, PSO4 RNN has the least number of iterations, and the actual training time is also relatively short. The time consumption is shown in Table 3, which is calculated by averaging eight repeated experiments. All methods are set to 10 clusters in the state clustering process, and the time consumption is how long it takes to learn an automaton at an abstraction error of less than 0.4. From the statistical results, it can be seen that after ensuring sufficient accuracy for the level of abstraction, PSO4 RNN has the lowest time consumption in abstract learning compared to BSL1 and BSL2. This is because the PSO algorithm improves the search efficiency in the learning process of PFA.