# Enhancing Conversational Agent Development Through a Semi-Automatization Development Proposal
## 5 Application Example
### Evaluating the Chatbot  
This section describes the evaluation of the PCIBot conversational agent, which mainly consisted of analyzing the quality of the conversational agent's answers and the GPT-3 recommendations. In addition, we investigated how users interacted with the bot when it was unable to solve their query. Figure 9: Snippet of chatbot source code generated. In order to achieve the above objectives, the evaluation first focused on collecting data from users through a survey. The conversational agent, PCIBot, was created to provide information from the Information Sciences Graduate Program of the Universidad Autonoma de Sinaloa. It was implemented on the world's most widely used instant messaging platform, WhatsApp. To initiate the case study, the PCIBot WhatsApp phone number was shared with 100 users who belong to or graduated from four universities: Universidad Autonoma de Sinaloa (UAS), Universidad Politecnica de Sinaloa (UPSIN), Universidad Autonoma de Occidente (UAdeO), and Instituto Tecnologico de Mazatlan (ITMAZ). These institutions have in common the study of computer science and computer systems at the undergraduate level. Figure 10 shows the operation of the chatbot interacting with a user through natural language in the WhatsApp application installed on their mobile phone. The PCIBot is able to respond to the query through text and images. Then a satisfaction survey was shared with the conversational agent users. The survey was distributed through the Google Forms tool [41]; once conducted, by analyzing the questions related to the conversations between the user and the bot, the objectives of this case study were evaluated. The survey data and the associated questionnaire are available in FigShare via the link provided in [42]. The analysis of the results of the survey administered to PCIBot users is presented below. Regarding overall user satisfaction, while PCIBot received generally positive feedback (66% satisfaction rate), instances of user dissatisfaction were observed. Analysis of recorded user-PCIBot interactions revealed that dissatisfied users commonly sought functionalities not yet implemented within the agent. This led to requests beyond the bot's current capabilities, underscoring the necessity of a post-deployment maintenance phase for chatbot development, enabling iterative updates based on user needs. Regarding perceived ease of use, a significant majority (71%) of users reported ease in formulating requests and expressed overall satisfaction with the usability and utility  
Figure 10: PCIBot in action. of the agent. Analysis of interaction logs indicated that users frequently resolved their queries using recommendations provided by the bot. However, instances where the agent struggled to interpret user intent were also identified, highlighting a key area for future development. Improved training sequences are proposed to mitigate these occurrences and enhance the effectiveness of interaction. The integration of PCIBot within the WhatsApp platform was met with strong user acceptance (99%). User feedback emphasized the convenience and accessibility of 24/7 availability from their mobile devices within a frequently used application. Participants further noted the utility of a conversational agent for accessing PIS information, learning more about the graduate program's field of study, and gaining rapid access to information outside of regular business hours. In particular, a substantial majority (96%) preferred WhatsApp over other instant messaging platforms such as Facebook, demonstrating the value of tools that facilitate WhatsApp integration, such as the one presented in this investigation. Assessment of the efficacy of the GPT-3 autoregressive language model recommendation revealed general user satisfaction. However, a discrepancy emerged between reported satisfaction and actual feature usage, indicating that the DialogFlow natural language processing service was not required, as the bot successfully classified all queries.