# A Modular Framework for Domain-Specific Conversational Systems Powered by Never-Ending Learning
## 4 Experimental Results
### Throughput Evaluation  
A simulation process was carried out to test the scalability of the PoC of the conversational system using JMT (Java Modeling Tool version 1.2.5) software [26]. The objective was to determine the number of servers (in this case, containers) of each module (running in parallel) necessary to meet a given message arrival rate. The closed simulation model was used (see Figure 8); it has no input or output for external requests or responses [27]. Requests (messages) are circulated internally. The total number of requests remains constant. A closed system can be considered one in which the \"Out\" is connected back to the \"In\", whereby requests that leave the system immediately enter the system after a delay (\"think time\") that can be configured. The flow of \"Out-to-In\" requests defines the throughput of the closed system. When analyzing a closed system, the number of requests (messages) is assumed to be provided. The purpose of the analysis is to determine the throughput of completing the requests. In the solution adopted by the project, all messages enter and leave the Kafka component when passing from one container to another. In the simulation model considered, the Kafka component is represented by two queues: Kafka_1 and Kafka_2. Kafka_1 receives all the messages, and Kafka_2 sends the messages. However, for Kafka_2 to know how to route messages to the correct destination, a simulator feature was used that allows for identifying the messages leaving a queue destined for the next queue. This feature consists of identifying messages through classes. The following classes were defined:  
* C-I (blue): The class of messages arriving from clients;
* C-NLU (red): The class of messages that go to the NLU queue;
* C-Belief (green): The class of messages that go to the Belief Tracker + Policy + KM queue;
* C-NLG (pink): The class of messages that go to the NLG + KM queue;
* C-S (orange): The class of messages that go to the Session queue. At the exit of each queue, the message is converted to the class of the next queue before being forwarded to the [PERSON]_1 queue. Message classes are represented by different colors in the simulation results. The simulation was carried out using the closed model for the case of 1,000,000 msg/day to determine how many users are served simultaneously. The execution times of the modules used in the simulation were based on the average times obtained during the execution in eight virtual machines with 6 GB of memory and two virtual CPUs each. The obtained execution times used for throughput evaluation are presented in Table 3.  
The following scenario was used in this simulation:  
* The number of containers in the PO_Km module = 5;
* The number of NLG module containers = 11;
* The execution time of the modules based on Table 1;  
Figure 8: Java modeling tool queue model. The clients go from A letter to B letter in the icons. “Clientes” (from Portuguese) means “Clients” in English, and “Sessao” (from Portuguese) means “Session” in English. * The delay in generating a new message by the client after receiving a response from the previous message = 7 s;
* The boundary condition, where the use of modules is less than or equal to 0.7. In order to determine how many users were served simultaneously, corresponding to a flow of 11.57 messages per second, the number of users varied (50-150) when observing the flow corresponding to each number of users. The simulation results can be seen in the graph of the flow by number of users, illustrated in Figure 9.  
An approximate flow rate of 11.5 messages per second is observed to correspond to 95 users with a \"think time\" of 7 s.  
The graphs in Figures 10 and 11, respectively, illustrate the processor usage by the PO_Km and NLG modules with variations in the number of users.