## 3 Methodology
### Composite Metric Development  
PCA was employed to derive weights for the metrics based on their variance contributions, following a normalization process to ensure comparability across all metrics (CUQ, UES-SF, efficiency, and effectiveness). This statistically robust approach prioritizes metrics with higher variability, reflecting their relative influence on the composite user experience (UX) score. The PCA-based composite metric is calculated using Equation (3):  
\[S=\sum_{i=1}^{n}w_{i}X_{i} \tag{3}\]  
where \(S\) represents the composite UX score, \(w_{i}\) is the weight assigned to each metric \(X_{i}\) based on its variance contribution derived from PCA, and \(X_{i}\) is the value of the individual metric (e.g., CUQ, UES-SF, response time, or error scores). \begin{table}
\begin{tabular}{p{56.9 pt} p{113.8 pt} p{113.8 pt}} \hline \hline
**Score** & **Description** & **Guideline** \\ \hline
1—Very Poor & All or most responses are incorrect. & The user made errors in all three scenarios or only had one correct answer across all scenarios (i.e., two or more errors per scenario). \\ \hline
2—Poor & Several responses are incorrect, with only partial success. & The user made errors in two scenarios but managed to correct one scenario (e.g., two correct, one wrong, but the wrong ones have multiple errors). \\ \hline
3—Fair & There were mixed results, with some correct and some incorrect responses. & The user made one significant or several minor errors across the three scenarios (e.g., one correct, two partially correct with 1 or 2 errors each). \\ \hline
4—Good & Most responses are correct, with minor errors. & The user performed well in two scenarios (with minor errors) and has one perfect scenario (i.e., one or no errors across scenarios). \\ \hline
5—Excellent & All responses are correct, with no errors observed. & The user achieved perfect execution across all three scenarios (e.g., 0 errors in any scenario). \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Guidelines to assign error scores. This approach provides a balanced and statistically grounded representation of UX, integrating diverse metrics into a unified score. To ensure consistency, all metrics were standardized using z-scores prior to applying PCA. The PCA process followed these steps:  
1. Standardizing Metrics: Metrics (CUQ, UES-SF, effectiveness, and efficiency) were converted to z-scores to eliminate scale discrepancies. 2. Extracting Component Loadings and Variance Contributions: PCA identified the proportion of variance explained by each component, and component loadings _Wij_ were used to determine the weight of each metric. 3. Calculating Variance-Weighted Contributions: Each metric's weight was calculated as the product of its loading and the variance explained by the corresponding component. 4. Normalizing Weights: Weights were normalized to ensure that their total equaled 1, enabling fair comparisons across metrics. 5. Constructing the Final Composite Metric: The normalized weights were used to calculate the composite UX metric \(S\) by aggregating the weighted standardized metrics. This process produced weights of 0.696 for effectiveness, 0.267 for CUQ, 0.073 for UES-SF, and \(-\)0.036 for efficiency. These weights highlight the relative importance of each dimension, capturing interactions between UX elements, and offering a comprehensive understanding of user experience beyond individual metrics. The PCA-based metric emphasizes the holistic nature of UX by reflecting how dimensions such as usability, engagement, effectiveness, and efficiency collectively influence user satisfaction. This approach provides actionable insights for developers, enabling the identification of strengths and areas for improvement in chatbot design. By integrating these dimensions into a single weighted measure, the framework transcends traditional composite UX metrics, offering a nuanced and scalable approach to chatbot evaluation. Steps for Composite Metric Calculation:  
1.