# Hybridization of Intelligent Solutions Architecture for Text Understanding and Text Generation
## 6 Evaluation  
Analysis of the proposed approach was carried out using a dataset of legal documents including joint-stock company decision record sheets and protocols. The experiment task was to extract the pre-defined list of attributes from these documents, which includes the following:  
Proper names (full name),  
Names of companies,  
Addresses,  
Dates. The test dataset contains 200 documents. The text was preliminarily extracted from the scanned images using the Tesseract OCR program (version 5.0.0-alpha.20200328) [26] and then split into sentences using the Segmenter module from the Natasha library [27]. The experiments were carried out at a stand with the following characteristics: Intel core i9-10900, 32 GB DDR4, 512 GB SSD, NVIDIA GeForce RTX 2070 SUPER. The following versions of libraries were used to extract the attributes:  
* Natasha 1.4.0 [27],
* Spacey 3.0.5 [28],
* SDK Pullenti 3.23 [29],
* Stanza 1.2 [30],
* DeepPavlov 0.14.0 [31]. Using the library of extracting routines, one control expounder and five specialized expounders were created according to the approach introduced above. Each expounder starts the execution of experiment tasks according to incoming requests. The process starts from the control expounder. If it cannot solve the task itself, then it delegates it to other expounders or creates a new one. The decision of task delegation is made on the basis of the expounder's evaluation, and several parameters can be used to select the best one. For the considered experiment, the results of expounders' evaluation are presented in Figure 6. The expounders are evaluated and compared by two parameters: performance and accuracy. The results of the experiment showed that the fastest is the solution based on the Spacey library (80.94 sentences per second with an accuracy of 0.46), and the most accurate is the solution based on the Natasha library (12.17 sentences per second with an accuracy of 0.53). Such a low accuracy of each solution is associated with the presence of typos in the text, resulting from the work of the Tesseract OCR. The results of solving the task by the best expounder were improved using the method of aggregating multiple expounders. It was proposed to choose the best individual solution for each extracted attribute. The results of the experiment showed that document processing performance decreases and the accuracy increases with an expansion of aggregated solutions (see Figure 7). Therefore, when aggregating solutions obtained as a result of launching five expounders, the document processing performance is 2.43 sentences per second, and the accuracy is 0.85. Implementing the auctioning approach is illustrated by the following cases of expounders' hybridization, which were tested under the framework of the described experiment. Case 1: An expounder based on the Natasha library is installed as a priority expounder. If any type of attribute processing fails, the control expounder calls the expounder that has the highest priority for this type of attribute. The result of using combination 1 is shown in Figure 8. The efficiency of the solution when choosing the best expounder in terms of accuracy is 0.67, and the processing performance is 6.34 sentences per second. These characteristics are equal for both using the precision and score to prioritize the expounders. Figure 6: Comparative evaluation of the expounders: (**a**) performance, (**b**) accuracy. Figure 7: Accuracy versus performance dependency from the number of expounders used. Case 2: In addition to case 1, the auctioning algorithm is extended in the following way. Other expounders are additionally called with a frequency equal to their accuracy for each attribute type. The results are presented in Figure 9. For expounders' prioritization by precision, the resulting accuracy was 0.71, and the performance was 5.09 sentences per second. When choosing an expounder by score, the accuracy of the solution is 0.67, and the processing performance is 6.3 sentences per second.