## 3 Methodology
### Measures and Metrics
#### 3.4.3. Chatbot Efficiency and Effectiveness  
Efficiency and effectiveness are metrics used to evaluate chatbot performance across different interaction modes and levels of question complexity. Unlike the CUQ, which captures user satisfaction across multiple dimensions, these metrics provide operational performance indicators. Efficiency, measured by response time, reflects how quickly the chatbot processes and responds to user queries. Shorter response times indicate higher efficiency, directly influencing users' perceptions of the system's reliability and speed, thus maintaining engagement.  
The chatbot's internal system recorded response times, measuring the interval between a user's inquiry submission and the chatbot's response. Tasks were categorized based on query complexity:  
* Simple tasks, e.g., \"What are the library hours?\"
* Intermediate tasks, e.g., \"What are all the available courses this semester?\"
* Complex tasks, e.g., \"What options are available for academically struggling students, and how can GPA be improved while considering college policies?\"  
\begin{table}
\begin{tabular}{p{56.9 pt} p{113.8 pt} p{113.8 pt} p{113.8 pt}} \hline \hline
**Items** & **Dimension** & **Definition** & **Calculation** \\ \hline FA-S1 & & Measures how deeply users are engrossed and absorbed during their interaction, indicating a state of immersion and time distortion. & (FA-S1 + FA-S2 + FA-S3)/3 \\ \hline PU-S1 & & Evaluates how intuitive and easy the chatbot is to use, with reversed scoring for negative statements (PU-S1, PU-S2, PU-S3) to ensure that higher scores reflect better usability. & ((6-PU-S1) + (6-PU-S2) + (6-PU-S3))/3 \\ \hline AE-S1 & & Assesses the visual and sensory appeal of the chatbot, reflecting how design and aesthetics enhance the user experience. & (AE-S1 + AE-S2 + AE-S3)/3 \\ \hline RW-S1 & & Measures the sense of value, interest, and satisfaction that users derive from their interaction, reflecting motivation and engagement. & (RW-S1 + RW-S2 + RW-S3)/3 \\ \hline \multirow{4}{*}{Overall engagement} & \multirow{4}{*}{Overall engagement} & \multirow{4}{*}{Represents the overall engagement level, encompassing all four dimensions to provide a comprehensive view of the user experience.} & The sum of all 12 items with PU-S1, PU-S2, PU-S3 \\ \cline{1-1} \cline{3-3}  & & & \\ \cline{1-1} \cline{3-3}  & & & \\ \hline \hline \end{tabular}
\end{table}
Table 3: The UES-SF.  
Response times for each interaction mode (text-based, menu-based, and hybrid) were evaluated across these task categories, offering insights into the chatbot's efficiency in handling varying query complexities.  
Effectiveness, on the other hand, is assessed through error scaling, which indicates the chatbot's ability to accurately interpret and respond to user inputs. A lower error score signifies higher effectiveness, demonstrating the chatbot's capacity to handle user queries without misunderstandings or incorrect answers.  
To scale error scores to a range of 0 to 100, the following formula was applied, as shown in Equation (2):  
\[\text{Scaled Effectiveness}\ =\ (1-\frac{\text{{Observed Error Score}}}{\text{Maximum Error Score}})\times 100 \tag{2}\]  
This approach normalizes effectiveness scores by considering the proportion of errors relative to the maximum possible errors for a given task or query. Scaling ensures consistency across different question complexities and allows for meaningful comparisons of chatbot performance.  
For effectiveness, participants (Group B) were asked to rate the chatbot's answers based on the guidelines presented in Table 4.