## 3 Materials and Methods  
### Study Design  
This study's main goal was to compare how users' perceived usability, ease of use, UX, mental effort, and trust between non-AI and AI-based tools differ when conducting qualitative data analysis. Simultaneously, we aimed to understand if user perceptions varied according to the kind of chatbot tool utilized. The basic conceptual model of the main variables that we aimed to investigate, evaluate, and compare with our research is shown in Figure 1.  
This study used the Taguette tool as an example of a tool that does not use AI for qualitative data analysis. Taguette is a free QA tool that can be used to analyze qualitative data such as interview transcripts, survey responses, and open-ended survey questions by encoding different text data segments and allowing multiple tagging in text segments, making it easier to identify patterns or themes [20]. As examples of two AI-based tools in our study, we chose ChatGPT (version 3.5) and Gemini, which also proved to be tools with high accuracy, comprehensiveness, and self-correction capabilities [74]. The study's main objective was to evaluate user perception of the observed AI-based chatbots and non-AI-based tools in the context of qualitative analysis. We focused primarily on the users' perceptions of using a specific tool for qualitative data analysis. The goal was also to establish measurement instruments to capture quantitative data and compare users' perceptions of different tools. Participants in this research were set in the role of UX researchers and tasked with a qualitative evaluation using individual tools. Since the goal was not to compare tools directly, we provided users with different sets of qualitative data obtained in preliminary research about the content, UX, and structure of the Faculty of Electrical Engineering, Computer Science and Informatics, University of Maribor website. The role of the participants in this study was to analyze the obtained qualitative data and to present both positive and negative aspects of the faculty's website content, UX, and structure quality. Participants were presented with three separate text documents (content.txt, ux.txt, and structure.txt), each containing 100 user responses. These responses were collected through an online survey designed to evaluate the faculty's website [[https://feri.um.si/en/](https://feri.um.si/en/)]([https://feri.um.si/en/](https://feri.um.si/en/)) (accessed on 15 December 2024). In the survey, 100 users, including students and faculty members (professors, teaching assistants, and others), answered three open-ended questions addressing weaknesses in the website's content, user experience, and structure. The data gathered as part of a prior user study, served as the basis for the qualitative analysis conducted in our experiments. Figure 1: Conceptual model of research. Responses were mostly short and presented in separate lines. Examples of the responses included _\"Too much content at once\"_, _\"I find the content OK, as it is related to the study or fields of study\"_, _\"A lot of irrelevant content. The front page should not contain news/announcements\"_, _\"Too few colours, page looks monochrome\"_, _\"Maybe someone can't find what they're looking for because they can't find the relevant subcontent\"_, _\"I didn't notice any shortcomings\"_, and so on. Data was organized into three separate documents to prevent participants from becoming overly familiar with a single dataset, which could have influenced their perception across the tools. By introducing a fresh dataset for each tool, we ensured that any differences in users' perceptions were more reflective of the tool than prior knowledge of the data. The first file, content.txt, contained answers to questions concerning the quality of the content provided on the faculty website. The second file, ux.txt, included a set of answers to the open-coded question concerning the UX quality of the faculty's website.