# Integrating Chatbot Media Automations in Professional Journalism: An Evaluation Framework
## 4 Results and Discussion
### Back-End Workshop and Evaluation Results  
As described in Section 3.2.3, a two-part workshop was used as an evaluation method for each of the chatbot creation interfaces. This endeavor aimed to directly compare and contrast the usability of all the features present in each of the platforms and to examine how each one of them could realistically be utilized both by professional journalists as well as students in the field of media. A hands-on approach was adopted, where each participant was asked to create their own chatbot by the end of the procedure. The first part of the workshop, pertaining to the opinions of the students, took place during the first days of March 2022 and was approximately two hours long. The second part, which studied the perceptions of professional journalists, took place at the beginning of September 2022 and had a similar length. For the purposes of this workshop procedure, it was important to procure participants that were all adjacent to the field of journalism but with varying degrees of familiarity with the profession. When it comes to the first part of this two-part workshop, the participants (\(n=8\), 4 male and 4 female) were between the ages of 18 and 30, with no prior experience or specialization in the field of ICT or chatbot creation. They were all students in the School of Journalism of the Aristotle University of Thessaloniki, enrolled in the course on Human-Computer Interaction, thus making them qualified to accurately judge the design process through a journalistic approach. As for the second part of the workshop, the participants (\(n=7\), 4 male and 3 female) were all professional journalists with working experience in the field but without any specialization in ICT or prior knowledge when it comes to chatbot creation. Their age group ranged from 31 to 60 years old. After a detailed presentation of the creation process for each one of the platforms, the attendees were asked to allocate some time to create a sample chatbot for each one of the selected tools. Instructions were given for the specifications of the sample chatbot, but no extra help was provided to them to more accurately gauge their level of understanding for each of the creation interfaces. To ensure the validity of the results, the subjects were not allowed to communicate with each other during this stage of the study. After this process was concluded, participants were asked to fill out a questionnaire pertaining to their experience. The evaluation survey results are shown in Figure 5, and they are measured on a seven-point Likert scale where 1 represents a negative connotation, and 7 represents a positive one. As can be seen from the above, the results of the workshop show that all three platforms were generally well-received. Quriobot, however, was consistently ranked higher than its peers in all categories. This lead ranges from small to relatively significant, and even though it spans all metrics, it seems to stand out, particularlyin terms of ease of use and straightforwardness. Specifically, Quriobot received an average rating of 5.91 (SD = 1) in the ease-of-use category, compared to the other two platforms, which ranged from 4.25 (SD = 1.3) for SnatchBot to 4.58 (SD = 1.5) for Chatfuel. Similarly, the process of correcting user mistakes was deemed to be noticeably easier on Quriobot with a mean score of 5.75 (SD = 0.8) as opposed to SnatchBot (M = 4, SD = 1.3) and Chatfuel (M = 4.33 SD = 1.6). Statistics such as these swayed user opinions into ranking the chatbot creation interface of Quriobot higher than the other two alternatives, as participants concluded that it is more suited for all levels of users (M = 4.75, SD = 1.2) compared to the alternatives (SnatchBot: M = 2.91, SD = 0.9; Chatfuel: M = 3.41, SD = 1.5). When it comes to specific comments left by the participants during the discussion portion of the workshop, the declared opinions seem to substantiate the recorded results.